Standard SQL/JSON - the sobering part

自从jOOQ 3.14在2020年10月19日发布,支持SQL/JSON（和SQL/XML）以来,已经过去了将近1年.半年后,我们又发布了支持MULTISET 的jOOQ 3.15,It is based on these features,Provide type safety of nested set,这是每个ORDBMSShould be implemented the way.

在我们自己的SQL/JSON API的基础上进行构建（dogfooding）,To reveal the different suppliers of all kinds ofSQL/JSONImplementation of a lot of attention,坦率地说,It is a bit of a conscious experience.Although now have aISO/IEC TR 19075:6标准（This is mainly by theOracle推动的）,Many vendors have implemented some kind ofJSON支持,And it looks different in all dialects--So write the localSQLWrite the supplier has nothing to doSQL/JSON几乎不可能.你需要一个像jOOQ这样的APIOr other abstract concepts to regulate different dialects.

在这篇文章中,I want to share over the past year to meet some of the biggest considerations.还有很多,只要尝试用jOOQWill some standardSQL/JSONTranslated into various dialects就可以了.

JSON类型还是字符串？

JSONThe document can be seen as a simple string formatted content.当然,What I mean is that you can put the string parsed intoJSON文档,So why want to use the new typeSQLThe type system complication？

不幸的是,This proved to be most vendors a wrong decision.PostgreSQL通过提供JSON 和JSONB 类型来解决这个问题,The latter even in storage and indexJSONWhen the document is quite good.

首先,对于任何使用SQLA language other than,这并不奇怪.Type is a semantic.In series with the type of environment is not good.Or like the other words.

The poor quality of the bitter taste of the sweet forgotten after the price is low also exists for a long time--本杰明-富兰克林

"低价 "Means there is no formal kind of quick and dirty tandem type function add low prices.我们已经在BOOLEAN Type the last time and again happens,还有像MySQL这样的方言,Pretend to support this kind of situation is a good idea：

SELECT *
FROM t
WHERE 1 OR 0 -- TRUE OR FALSE
复制代码

Let's take a look at the example of what I mean.

MariaDB和MySQL

让我们先看一下MariaDB和MySQL的语法.下面是如何创建一个JSON数组：

select json_array(1, 2)
复制代码

生成

[1, 2]
复制代码

这很好!And it even up to the standardSQL语法.一个JSONArray can also be easily for nested：

select json_array(json_array(1, 2), 3)
复制代码

来产生：

[[1, 2], 3]
复制代码

现在,If the nested array from a derived table？

select json_array(nested, 3)
from (select json_array(1, 2) nested) as t
复制代码

它的结果是：

-- MySQL, MariaDB 10.6
[[1, 2], 3]

-- MariaDB 10.5
["[1, 2]", 3]
复制代码

遗憾的是.在MariaDB 10.5中,Nested array lost it "JSON类型注释",Back to its string type version.作为一个字符串,It needs to be quote.这似乎是一个错误jira.mariadb.org/browse/MDEV…,显然已经在MariaDB 10.6The repair that has been under a different question.但这并不是唯一的问题.BugThere are many similar problems on the tracker：https://jira.mariadb.org/browse/MDEV-13701.

MySQLSeems to be a little better now,Although there are some matters needing attention in polymerization（见下文）.

到目前为止,I find the only solution is for the above error非常费力的：

select json_array(json_merge_preserve('[]', nested), 3)
from (select json_array(1, 2) nested) as t
复制代码

想象一下,当你每次嵌套JSONIn this way.它是可行的,But it really need automation（例如通过jOOQ）.

甲骨文

OracleDefines the mostSQL标准,我非常喜欢它的SQL-idiomaticGrammar sense,就像SQL/XML.不幸的是,他们把新的JSONThe introduction of the type in theOracle 21c（Can't wait to use it）.因此,We must choose is toJSON表示为VARCHAR2 ,默认情况下（In some systems limited to4000字节,或最多32kb！）还是CLOB .If you are doing seriousJSON,You may can always be in your每一个 JSON函数调用中添加RETURNING CLOB 子句.有效地：

select json_array(1, 2 returning clob)
from dual
复制代码

Nested version looks like this：

select json_array(nested, 3 returning clob)
from (select json_array(1, 2 returning clob) nested from dual) t;
复制代码

这在OracleThe effect is much better,But there are still many edge is not solved.在Oracle 18cTry this one on：

select json_arrayagg(coalesce(json_object(), json_object()))
from dual
复制代码

It creates another string typeJSON实例：

["{}"]
复制代码

The solution is where all the addFORMAT JSON ,只是为了确定,例如：

select json_arrayagg(
  coalesce(json_object(), json_object()) format json
)
from dual
复制代码

现在,结果与预期一致：

[{}]
复制代码

为了安全起见,You may also want to在所有地方写上FORMAT JSON ,就像RETURNING CLOB

截断

Truncation may be useJSONThe most worrisome part.为什么我们在SQLThere is still a limit to the size of the data types in the？从应用的角度来看,It almost doesn't make any sense.然而,我们在这里.

甲骨文

在OracleRunning this query：

select json_arrayagg(owner) from all_objects;
复制代码

你会得到：

SQL错误 [40478] [99999].ORA-40478: The output value is too large（最大：4000）.

将最大的VARCHAR2 大小增加到32kb,Will only delay problems appear.This type of file is not "合理 "的大小限制,所以同样,你必须一直RETURNING CLOB .

select json_arrayagg(owner returning clob) from all_objects;
复制代码

Price is as usual.CLOB ,只是比VARCHAR2 ,无论是在OracleInternal or in the client application（例如基于JDBC的）中,More disgusting,Because when you just want a string,You have to deal with another resource.jOOQWill be there for you add clause,Almost no reason not to usejOOQ,因为获取CLOB 的值对jOOQ用户是完全透明的.

MySQL

直到最近,你还不能在MySQL中调用JSON_ARRAYAGG() ,而且MariaDBVersion will make server crash（https://jira.mariadb.org/browse/MDEV-21912）.当写这篇博客的时候,Neither of the two implementation supportsJSON_ARRAYAGG() 中的ORDER BY 子句,I think it is very important,So the alternative way is to useGROUP_CONCAT：

select concat('[', group_concat(id), ']')
from t_book
复制代码

当然,If the value is not a series digital,那就大错特错了,所以我们还需要使用JSON_QUOTE ,比如说：

select concat('[', group_concat(json_quote(title)), ']')
from t_book
复制代码

而且,如果你在其他JSONStructure embedded in these things,你必须使用JSON_MERGE（直到最近,But has now been abandoned）或JSON_MERGE_PRESERVE ,Now the real string intoJSON,例如：

select json_object(
  'titles', 
  json_merge_preserve(
    '[]', 
    concat('[', group_concat(json_quote(title)), ']')
  )
)
from t_book
复制代码

To produce such a file：

{"titles": ["1984", "Animal Farm", "O Alquimista", "Brida"]}
复制代码

如果没有这个JSON_MERGE_PRESERVE ,你就会得到：

{"titles": "[\"1984\",\"Animal Farm\",\"O Alquimista\",\"Brida\"]"}
复制代码

Absolutely is not something you can remember.

无论如何.This section is about the truncation!在MySQL中,对于大型的、聚合的JSONDocument what happens？试试这个：

select 
  concat('[', group_concat(json_quote(table_name)), ']')
from information_schema.tables
复制代码

它产生（在我的机器上）：

["innodb_table_stats","innodb_index_stats","CHARACTER_SETS","CHECK_CONSTRAINTS","COLLATIONS","COLLATION_CHARACTER_SET_APPLICABILITY","COLUMNS","COLUMNS_EXTENSIONS","COLUMN_STATISTICS","EVENTS","FILES","INNODB_DATAFILES","INNODB_FOREIGN","INNODB_FOREIGN_COLS","INNODB_FIELDS","INNODB_TABLESPACES_BRIEF","KEY_COLUMN_USAGE","KEYWORDS","PARAMETERS","PARTITIONS","REFERENTIAL_CONSTRAINTS","RESOURCE_GROUPS","ROUTINES","SCHEMATA","SCHEMATA_EXTENSIONS","ST_SPATIAL_REFERENCE_SYSTEMS","ST_UNITS_OF_MEASURE","ST_GEOMETRY_COLUMNS","STATISTICS","TABLE_CONSTRAINTS","TABLE_CONSTRAINTS_EXTENSIONS","TABLES","TABLES_EXTENSIONS","TABLESPACES_EXTENSIONS","TRIGGERS","VIEW_ROUTINE_USAGE","VIEW_TABLE_USAGE","VIEWS","COLUMN_PRIVILEGES","ENGINES","OPTIMIZER_TRACE","PLUGINS","PROCESSLIST","PROFILING","SCHEMA_PRIVILEGES","TABLESPACES","TABLE_PRIVILEGES","USER_PRIVILEGES","cond_instances","error_log","events_waits_current","events_waits_history","events_waits_history_long","events_waits_summary_by_host_by_event_name","events_waits_summary_by] 复制代码

等等,What is the end？

,"events_waits_summary_by]
复制代码

无效的JSON.因为GROUP_CONCAT The string is truncated.我们可以设置如下：

set @@group_concat_max_len = 4294967295;
复制代码

But now the output is correct,而且更长：

["innodb_table_stats",...,"t_identity_pk","t_triggers"]
复制代码

像jOOQ这样的APIFor you will be able to automatically set the session variable,But you may not want to for the localSQLHave been considering the question？

数据类型支持

JSONKnow that some of the data type.即.

字符串
数字
布尔型
空
对象
数组

这比SQL要少,But often good enough（After all, nothing can be encoded as a string）.但是当你在SQL中没有BOOLEAN 类型时（例如MySQL,Oracle）,You have a manual you have to manually coded asJSONBOOLEAN .

MySQL

MySQLLet you believe that this is feasible：

select json_array(true, false)
复制代码

Because the produce

[true, false]
复制代码

But it doesn't really work.This seems to be hard coded in the parser.只要你的true 和false Value is the expression,Rather than a literal meaning,例如,From a derived table：

select json_array(t, f)
from (select true as t, false as f) as t
复制代码

你就会得到：

[1, 0]
复制代码

There are different ways to simulate this.一种是：

select json_array(
  json_extract(case when t = 1 then 'true' when t = 0 then 'false' end, '$'),
  json_extract(case when f = 1 then 'true' when f = 0 then 'false' end, '$')
)
from (select true as t, false as f) as t：
复制代码

现在,我们又得到了：

[true, false]
复制代码

Oracle

不像MySQL / MariaDB,Oracle SQLDon't pretend that it has aBOOLEAN 类型.相反,It is encoded asNUMBER(1) 或CHAR(1) 或其他一些东西.Whatever the code,这就是解决方案.

select json_array(
  case when t = 1 then 'true' when t = 0 then 'false' end format json,
  case when f = 1 then 'true' when f = 0 then 'false' end format json
)
from (
  select 1 as t, 0 as f, null as n
  from dual
) t
复制代码

生成

[true,false]
复制代码

NULL处理

当用SQL/JSON使用NULL ,有各种注意事项.首先,SQLNULL 与JSONNULL 不是一回事.使用PostgreSQL.

select 
  a, 
  b, 
  a = b as equal, 
  a is null as a_is_null, 
  b is null as b_is_null from ( select null::jsonb as a, 'null'::jsonb as b ) as t 复制代码

产生了：

|a  |b   |equal|a_is_null|b_is_null|
|---|----|-----|---------|---------|
|   |null|     |true     |false    |

复制代码

The empty cell isSQLNULL 值,而null 值是JSONnull 值,不是 "SQLNULL".This is the only reasonable way,真的.All agree that？

MySQL

select 
  a, 
  b,
  a = b as equal, 
  a is null as a_is_null, 
  b is null as b_is_null from ( select null as a, json_extract('null', '$') as b ) as t 复制代码

产生的也是：

|a  |b   |equal|a_is_null|b_is_null|
|---|----|-----|---------|---------|
|   |null|     |1        |0        |

复制代码

所以,是的!

Oracle

让我们看看,This may be how do I create aJSONNULL 值：

select 
  a, 
  b, 
  case when a = b then 1 else 0 end as equal, 
  case when a is null then 1 else 0 end as a_is_null, 
  case when b is null then 1 else 0 end as b_is_null from ( select null as a, json_value('[null]', '$[0]') as b from dual ) t 复制代码

然而,众所周知,Oracle与NULL The string has a strange relationship.这就是结果：

|A  |B  |EQUAL|A_IS_NULL|B_IS_NULL|
|---|---|-----|---------|---------|
|   |   |0    |1        |1        |

复制代码

Doesn't seem to have a realJSONNULL 表示！I haven't found a solution to this problem.也许我会的.But this is very unfortunate,Caused many transition edge cases.

另一方面,In this paper introduces the dialect of,OracleIs the only for aggregation function is introduced into the very usefulNULL Deal with terms and conditions of the dialect.请看这个：

select
  json_arrayagg(a),
  json_arrayagg(a absent on null),
  json_arrayagg(a null on null)
from (
  select 1 as a from dual union all
  select null from dual
) t
复制代码

生成

|A  |B  |C       |
|---|---|--------|
|[1]|[1]|[2,null]|


复制代码

Pay attention to the above query inOracle 18cAlso can't produce the correct results,Because a parser/The optimizer error.在这个例子中,Use this method to solve this mistake：

select
  json_arrayagg(a) a,
  json_arrayagg(a + 0 absent on null) b,
  json_arrayagg(a - 0 null on null) c
from (
  select 1 as a from dual union all
  select null from dual
) t
复制代码

Other dialects to how toNULL Value is aggregated intoJSONThe document have a different opinion.在SQL中,Aggregation function tend to ignoreNULL 值,比如上面的Oracle,但是对于JSON,Usually must include the value,特别是在创建JSON_OBJECT ,Among them there is no key and the value of strictly are two different things.

Db2

Db2对SQL/JSONThe implementation of the very limited.It is very accord with a standard on grammar,But there is a serious mistake,比如这些.

This makes it is still can't use.当然,This will be improved in the near future.

SQL服务器

在这篇文章中,I also missSQL Server.SQL ServerSupport for a period of timeJSON和XML,但实现方式完全不同.You can't easily to form anyJSON对象或数组,But you can turn the result set in a straightforward way toJSON.

This results in flow processing provides a quick victory,But not very well for synthetic.例如,You can't use a scalar、The contents of the object to create aJSON_ARRAY ,Although it can be to create aJSON_OBJECT ：

select 1 as a, 2 as b
for json path, without_array_wrapper
复制代码

制作

{"a":1,"b":2}
复制代码

在SQL ServerCan also be simulated in a limited number ofJSON特性,And all does not support the actualJSONType of dialect,May need to be constantly escape/取消转义.

总结

SQL/JSONThe standardization of relatively late,主要是由Oracle制定的.This standard is very perfect.但遗憾的是,Many dialects in grammar and behavioral differences,Some of the dialects there is a big problem.

Of all the implementation,The sound isPostgreSQL,It introduces the appropriate data type,And provides a rich set of vendor specific function to direct inSQL中操作JSON.在不久的将来,PostgreSQL会像OracleThe same acceptance criteriaSQL/JSON,And strengthen their own implementation.What I don't think there will be a fundamental new things,Just a standard grammar will be more portable.

关于 "下一步是什么 "的预览,Please look at the speech.

对于像jOOQ这样的库来说,There is nothing insurmountable,It abstracts all the dialects,并使用统一的、Inspired by the standard ofAPI提供SQL/JSON（和SQL/XML）功能.Following the development of the dialect,jOOQWill use the new supplier、更好的语法,So you can achieve yourSQL/JSON查询的兼容性.

因此,像往常一样,使用jOOQTo solve the manufacturers of the subtle and boring grammar differences,And in today has been useSQL/JSON.Although here puts forward some matters needing attention,但SQL/JSON是现代SQLOne of the most exciting thing in

阅读更多：