Spark SQL generate JSON

background

Want to pass spark sql Inquire about hive The table then assembles the corresponding fields into sql, Be similar to json_object , But unfortunately spark 3.1.x did not json_object function , But also good spark sql Yes to_json function

Example ：

SELECT to_json(struct(bis_type,year,douban_rating))
from tv.test where date=20220620
limit 10

SELECT to_json(named_struct('bis_type',bis_type,'year',year,'douban_rating',douban_rating))
from tv.test where date=20220620
limit 10

result , You can keep the original data type . This is the most important point , assemble json, Keep the data type of the original field

{
    "bis_type":"xxx","year":2022,"douban_rating":8.3}

How to generate nested json

SELECT to_json(array(named_struct('ruleType', 'datetime', 'ruleDetail', array(map('columnName', 'a1','function','test'),map('columnName', 'a2','function','test')))));

result

[
  {
    
    "ruleType": "datetime",
    "ruleDetail": [
      {
    
        "columnName": "a1",
        "function": "test"
      },
      {
    
        "columnName": "a2",
        "function": "test"
      }
    ]
  }
]