Sharp tool SPL for post SQL calculation

In modern application development , Usually only SQL Implement simple data access actions , The main calculation process and business logic are directly implemented in the application , The main reason is ：

• Too complicated SQL It is difficult to debug 、 To write 、 read 、 modify .
• SQL With dialect characteristics , A lot of use SQL after , It will make the program difficult to transplant .
• In terms of architecture, business logic is required to be implemented in the application , Instead of relying on the database , Otherwise, the coupling is too high .
• Some calculations SQL Not good at , Including complex set computation 、 Ordered computing 、 Associated calculation 、 Multi step calculation , It often needs to be moved out of the database .
• When implementing process control , Because it is more difficult to transplant 、 Higher coupling 、 Affect data security , Inconvenient to use stored procedures .

Besides , There are also scenarios involving multiple databases and non databases , It can't be used SQL Complete the calculation task , Can only be done externally .

such , It is necessary to implement in the application SQL Post calculation task .

SQL The returned data is generally structured data , So good SQL Post computing technology should also have convenient structured data objects , It can further calculate and process the returned data ; Provide rich library functions , Possession is no less than SQL Computing power ; It is better to support loop and judgment syntax for process control . Specially ,SQL Post - computing technology should be used in applications , Be easy to integrate .

Java Is an important development language , but JDK The method provided is too basic , Although it can be achieved SQL Post calculation , But the development efficiency is very low .

ORM yes Java It is used to realize SQL Common solutions for post calculation . But several more popular ORM Lack of professional structured data objects , Dynamic data structures are not supported . Although we can use Java Achieve process control , But it is difficult to calculate flexibly . these ORM The computing power of technology is far less than SQL, The calculation functions provided are very limited , use Java Hard writing is still very common .

Stream The chain programming of ORM Of HQL More object oriented , And there are Lambda Grammatical addition , It is also often used for SQL Post calculation , There are some ORM It can also generate Stream object . but Stream There is also no specialized structured data object , Dynamic data structures are not supported . Besides ,Stream Their computing power is also poor , Not even as good as ORM, Even sorting 、 Group summary 、 Associate such basic calculations , It should also be supplemented by a large number of codes .

Kotlin be based on JVM, And in chain programming and Lambda Grammatically correct Stream A series of improvements have been made , It can also be used for SQL Post calculation . But because the bottom of compiled languages ,Kotlin Only right Stream Small fine adjustment , There is no shortage of major shortcomings .

Python Pandas Strong structured data processing capability , Sometimes it can also be used for SQL Post calculation , But because of the lack of an easy-to-use interface , It's hard to be Java Integrate , Rarely seen in formal projects .

esProc SPL It's better SQL Post computing technology .

Professional structured data objects

SPL yes JVM Open source structured data computing engine under , Built in professional structured data object sequence table , And database tables / Records are easily interchangeable , Support dynamic data structure , Provides flexible and easy-to-use access methods 、 Maintenance methods 、 Calculation function . The sequence table is highly professional , It provides a strong underlying support for data calculation and process control , It can be easily realized SQL Various business logic in post calculation .

Direct database exchange method , It can be found in the database table （SQL Result set ） and SPL The sequence tables are converted to each other .

such as , Use query Function execution SQL, Generate a single record sequence table ：

	A	B
1	=connect("orcl")	// Connect to database
2	.query("select * from sales where orderid=?",201)=r=A1	// Query single record
3	=db.close()	// Close database connection

If SQL Return multiple records , Multiple record sequence tables are automatically generated ：

=T=A1.query("select * from salesR where SellerID=?",10)

The reverse is also simple , use update Function can batch persist the order table records to the database . such as , The original sequence is T, After the addition, deletion and modification of multi condition records, the sequence table is NT, Write the change results of both to the database ：

=A1.update(NT:T,sales;ORDERID)

Flexible sequential table access method , You can freely access the sequence table by field name or record number .

The... Of the order table 3 Bar record ：T(3)

After taking 3 Bar record :T.m([-1,-2,-3])

Take the field value of the record ：T(3).Amount*0.05

Take a column , Back to the assembly ：T.(Amount)

Take a few columns , Returns the set of sets ：T.([CLIENT,AMOUNT])

First by field name and then by record serial number ：T.(AMOUNT)(3)

First get by record serial number and then by field name ：T(3).AMOUNT

Easy to use sequence table maintenance method , You can add, delete, and modify a single or multiple records .

Additional records ：T.insert(200,"APPL",10,2400.4,date("2010-10-10"))

Modify the record ：T(3).Amount = T(3). Amount*1.05

Delete record ：T.delete(T.select(ORDERID\==209 || ORDERID==208))

Rich order table calculation functions , Complete SQL Formula calculation .

Filter ：T.select(Amount>1000 && Amount<=3000 && like(Client,"*bro*"))

Sort ：T.sort(-Client,Amount)

duplicate removal ：T.id(Client)

Summary ：T.max(Amount)

Filter after grouping and summarizing ： T.groups(year(OrderDate),Client; avg(Amount):amt).select(amt>2000)

relation ：join(Orders:o,SellerId ; Employees:e,EId).groups(e.Dept; sum(o.Amount))

intersection ：T1.id(Client) ^ T2.id(Client)

TopN：T.top(-3;Amount)

grouping topN：T.groups(Client;top(3,Amount))

Support dynamic data structure , The new data structure can be inferred from the calculation results in the previous step , And automatically generate a new order table , The new sequence table can be calculated directly . For example, group and summarize first , Re filtration , Last sort ：

T.groups(SellerId, Client; sum(Amount):amt, count(1):cnt).select(amt>10000 && amt<=30000 && like(Client,"\*bro*")).sort(amt)

Use an order table that supports dynamic data structures , Developers can focus more on computing itself , Instead of thinking about how to define the result set in advance . This coding style is not only short and easy to understand , And more in line with natural thinking , Development efficiency can be significantly improved . In multi-step complex business logic , The advantages of dynamic data structure are more obvious .

Powerful structured data computing capability

SPL Provides a variety of easy-to-use grammars , Built in a large number of powerful functions , It can simplify complex ordered operations 、 Set operations 、 Distribution calculation 、 Associated calculation . Many uses SQL And stored procedures are difficult to express , use SPL Can be easily realized .

Function options 、 Convenient syntax such as hierarchical parameters , Functions with similar functions can share a function name , Use only function options to distinguish the differences , Than SQL More flexible and convenient . such as select The basic function of the function is to filter , If only the qualified third party is filtered out 1 Bar record , Options available @1：

T.select@1(Amount>1000)

Parallel filtering , Suitable for large amount of data , Use options @m：

T.select@m(Amount>1000)

Sort by dichotomy , That is to use dichotomy to quickly filter the ordered data , Use @b：

T.select@b(Amount>1000)

Group in order , That is, the data in order of grouping fields , Group adjacent records with the same field values , Use @b：

T.groups@b(Client;sum(Amount))

Function options can also be combined , such as ：

Orders.select@1b(Amount>1000)

The parameters of structured operational functions are somewhat complex , such as SQL You need to use various keywords to separate the parameters of a statement into multiple groups , But this will use a lot of keywords , It also makes the sentence structure inconsistent .SPL The use of hierarchical parameters simplifies the expression of complex parameters , That is, through the semicolon 、 comma 、 The colon from high to low divides the parameters into three layers ：

join(Orders:o,SellerId ; Employees:e,EId)

Built in a large number of date functions and string functions , It is far more than other technologies in terms of quantity and function SQL, The same amount of operation code is shorter . such as ：

Time class functions , Date change ：elapse("2020-02-27",5) // return 2020-03-03

What day ：[email protected]("2020-02-27") // return 5, That is, week 4

N Date after working days ：workday(date("2022-01-01"),25) // return 2022-02-04

String class function , Judge whether it's all numbers ：isdigit("12345") // return true

Take the string before the substring ：[email protected]("abCDcdef","cd") // return abCD

Split into string array by vertical line ："aa|bb|cc".split("|") // return ["aa","bb","cc"]

SPL It also supports year increase or decrease 、 Ask for the day of the year 、 Quarter 、 Split string by regular expression 、 Dismantle SQL Of where or select part 、 Take out the words 、 Remove as marked HTML And so on .

Simplify complex ordered operations . Involving ordered operations across rows , There is usually some difficulty , For example, compared with the previous period and the same period .SPL Use " Field [ The relative position ]" Reference cross row data , Can significantly simplify the code , It can also automatically handle special cases such as array out of bounds , Regular ratio SQL More convenient . such as , Append a calculated column rate, Calculate the amount growth rate of each order ：

=T.derive(AMOUNT/AMOUNT[-1]-1: rate)

Using position expression and ordered function synthetically , quite a lot SQL Difficult to achieve orderly operation , Both can be used. SPL Easy solution . such as , According to the attendance sheet , Find continuity 4 Every day of the week, I am diligent 7 Hours of students ：

	A
1	=connect("mysql")
2	[email protected]("SELECT SID,ATTDATE,DURATION,null AS W FROM STUTEST WHERE DURATION>=7 ORDER BY SID,ATTDATE").run([email protected](ATTDATE))
3	[email protected](SID;[email protected](W;count(~):CNT).select(CNT==7)[email protected](W-W[-1]!=7).max(~.len()):weeks)
4	=A3.select(weeks>=4).(SID)

Simplify complex set operations ,SPL The aggregation of the order table is more thorough , With flexible syntax and powerful set functions , It can greatly simplify the complex set calculation . such as , Find employees in each department who are younger than the average age of the Department ：

	A
1	…// Omit the sequence table Employees Generation process of
2	=Employees.group(DEPT; (a=~.avg(age(BIRTHDAY)),~.select(age(BIRTHDAY)<a)):YOUNG)
3	=A2.conj(YOUNG)

Calculate the longest consecutive rise days of a stock ：

	A
1	…// Omit the sequence table AAPL Generation process of
2	=a=0,AAPL.max(a=if(price>price[-1],a+1,0))

SPL Sure It is convenient to realize step-by-step calculation , The aggregation of the order table is more thorough , You can easily express a set with variables , Suitable for multi-step calculation , quite a lot SQL Set operations that are difficult to express , use SPL Can be easily realized . such as , Find out the top half of the total sales n A big client , And sort by sales in descending order ：

	A	B
1	// Omit the process of fetching data
2	=A1.sort(amount:-1)	/ Sales are sorted in reverse order , Can be found in SQL Finish in
3	=A2.cumulate(amount)	/ Calculate the cumulative sequence
4	=A3.m(-1)/2	/ The final accumulation is the total amount
5	=A3.pselect(~>=A4)	/ More than half the position
6	=A2(to(A5))	/ Take value by location

Simplify complex correlation calculations . The professionalism of the preface is reflected in many aspects , One of them is to support the formal expression of association by object reference , Developers can intuitively access the associated table through the dot , So as to improve the development efficiency . quite a lot SQL Difficult to express correlation calculation , use SPL Can be easily realized . such as , Calculate the male employees of the female manager according to the employee table ：

=employees.select(gender:"male",dept.manager.gender:"female")

Flexible process control capability

SPL Provides flexible and easy-to-use branch judgment statements 、 Loop statement , Cooperate with professional structured data objects , It can easily implement various business logic .

Branch judgment statement ：

	A	B
2	…
3	if T.AMOUNT>10000	=T.BONUS=T.AMOUNT*0.05
4	else if T.AMOUNT>=5000 && T.AMOUNT<10000	=T.BONUS=T.AMOUNT*0.03
5	else if T.AMOUNT>=2000 && T.AMOUNT<5000	=T.BONUS=T.AMOUNT*0.02

Loop statement ：

	A	B
1	=db=connect("db")
2	[email protected]("select * from sales where SellerID=? order by OrderDate",9)
3	for T	=A3.BONUS=A3.BONUS+A3.AMOUNT*0.01
4		=A3.CLIENT=CONCAT(LEFT(A3.CLIENT,4), " co.,ltd.")
5		…

And Java The cycle is similar to ,SPL Also available break Keywords jump out of （ interrupt ） Current loop body , Or use next Keyword skip （ Ignore ） This cycle , Don't talk about it .

The flow control statement matches the sequence table , Business logic can be implemented in a unified syntax style , Including database reading and writing 、 Transaction processing 、 Process handling 、 Data calculation . such as , Calculate the bonus according to certain rules ：

	A	B	C
1	[email protected]("dbName")		/ Connect to database , Open transaction
2	[email protected]("select sum(Amount) from sales where sellerID=? and year(OrderDate)=? and month(OrderDate)=?", p_SellerID,year(now()),month(now()))		/ Query the current month's sales
3	=if(A2>=10000 :200, A2<10000 && A2>=2000 :100, 0)		/ Accumulated bonus this month
4	=p_Amount*0.05		/ This order has a fixed bonus
5	=BONUS=A3+A4		/ Total bonus
6	=create(ORDERID,CLIENT,SELLERID,AMOUNT,BONUS,ORDERDATE)		/ Create the data structure of the order
7	=A6.record([p_OrderID,p_Client,p_SellerID,p_Amount,BONUS, date(now())])		/ Generate an order record
8	>[email protected](A7,sales;ORDERID)		/ Try to write the stock in table
9	=db.error()		/ Warehousing results
10	if A9==0	>A1.commit()	/ success , Then commit the transaction
11	Else	>A1.rollback()	/ Failure , Then roll back the transaction
12	>db.close()		/ Close database connection
13	return A9		/ Return receipt result

Optimize the architecture

SPL Support JDBC Interface , The code can be external to Java, Less coupling .SPL It has the characteristics of interpretation and execution , Support out of Library computing and code migration , Support cross database and cross source calculation , stay SQL Good architecture can be provided in post computation .

SPL Provides Easy-to-use JDBC Interface , Can be Java Seamless code integration . such as , Ahead of SPL Save the code as a script file , And then Java The file name is called as a stored procedure in ：

Class.forName("com.esproc.jdbc.InternalDriver");
Connection connection =DriverManager.getConnection("jdbc:esproc:local://");
Statement statement = connection.createStatement();
ResultSet result = statement.executeQuery("call writeBonus()");

SPL The code is external to Java, Called by file name , It does not depend on the database , It doesn't depend on Java, Separation of business logic and front-end code , Low coupling .

Explain execution and hot switching . A large number of business logic , High complexity , Change is the norm . Good system architecture , Should be able to cope with changing business logic .ORM The essence is Java Code , It needs to be compiled before execution , It usually takes downtime to deploy , Dealing with changing business logic is cumbersome .SPL Is based on Java An interpretative language , Can execute without compiling , The script takes effect immediately after modification , Support hot switching without shutdown , Business logic suitable for dealing with change .

Facilitate code migration .SPL Get data from the database through the data source name , If a transplant is needed , Just change the data source configuration information in the configuration file , Without having to modify SPL Code .SPL Support dynamic data sources , You can switch between different databases through parameters or macros , For easier migration . To further enhance portability ,SPL It also provides standards that are independent of specific databases SQL grammar , Use sqltranslate The function converts the standard SQL Become a mainstream dialect SQL, Still pass query Function execution .

Convenient for management and operation . Because it supports calculation outside the library , Code can be managed by third-party tools , Facilitate teamwork ;SPL Scripts can be stored by file directory , Convenient and flexible , Low management cost ;SPL The permission requirements for the database are similar Java, Does not affect data security .

Cross database and cross source computing .SPL The sequence table can uniformly read various data sources （ contain RDB）, All kinds of data sources can be calculated with unified code , It can easily realize cross database calculation . such as , Yes MySQL and Oracle Internal correlation ：

	A
1	=mysql1.query("select SellerId, sum(Amount) subtotal from Orders group by SellerId")
2	=orcl.query("select EId,Name from employees")
3	=join(A1:O,SellerId; A2:E,EId).new(O.Name:name, O.Dept:dept, E.subtotal:amt)

SPL Support a variety of non RDB data source , Can be carried out RDB He Fei RDB Mixed calculation between . Include txt\csv\xls Wait for the documents ,MongoDB、Hadoop、redis、ElasticSearch、Kafka、Cassandra etc. NoSQL, as well as WebService XML、Restful Json And so on .

ORM as well as Stream/Kotlin Lack of professional structured data objects and computing capabilities ,Python Pandas Hard to be Java Integrate .SPL Built in professional structured data objects , It has powerful structured data computing ability and flexible process control ability , It can easily implement various business logic . Provides easy to use JDBC Interface , Can be Java Easily integrate . There are more structural advantages , Including low coupling 、 Explain execution and hot switching 、 Out of Library calculation , And cross database and cross source computing .

SPL Information

• SPL Official website
• SPL download
• SPL Source code