当前位置：网站首页>Project - Smart City
Project - Smart City
2022-06-11 05:15:00 【Liuhaohao yyds】
project - Smart city
1、 Save the data source to the front-end database
1.1 Create a front-end database
stay mysql Created in pre database , Change the encoding format to utf-8
1.2 Create a table to import data
1.2.1 Commercial housing contract information table
-- stay mysql Create a table
create table t_fcj_nwrs_sellbargain(
id varchar(255) comment ' Id card number ',
r_fwzl varchar(255) comment ' Property address ',
htydjzmj varchar(255) comment ' The area of the House agreed in the contract ',
tntjzmj varchar(255) comment ' Building area in the house ',
ftmj varchar(255) comment ' Share the building area of the house ',
time_tjba varchar(255) comment ' Commercial housing filing time ',
htzj varchar(255) comment ' The total contract price '
);
-- Import data
load data local infile 'D:\\data\\T_FCJ_NWRS_SELLBARGAIN.csv' into table pre.t_fcj_nwrs_sellbargain fields terminated by ',';
1.2.2 List of industrial and commercial shareholders
-- stay mysql Create a table
CREATE TABLE t_gsj_reg_investor(
id VARCHAR(255) COMMENT ' ID No '
);
-- Import data
LOAD DATA LOCAL INFILE 'D:\\data\\T_GSJ_REG_INVESTOR.csv' INTO TABLE pre.t_gsj_reg_investor FIELDS TERMINATED BY ',';
1.2.3 Industrial and commercial legal representative information form
-- stay maysql Create a table
CREATE TABLE t_gsj_reg_legrepre(
id varchar(255) comment ' ID No ',
position varchar(255) comment ' position ',
tel varchar(255) comment ' contact number ',
appounit varchar(255) comment ' Appointment and removal institution ',
accdside varchar(255) comment ' Sending unit ',
posbrmode varchar(255) comment ' The way of production ',
offhfrom varchar(255) comment ' From the term of office ',
offhto varchar(255) comment ' The term of office is limited ',
stufftype varchar(255) comment ' Data type code '
);
-- Import data
LOAD DATA LOCAL INFILE 'D:\\data\\T_GSJ_REG_LEGREPRE.csv' INTO TABLE pre.t_gsj_reg_legrepre FIELDS TERMINATED BY ',';
1.2.4 Social security card information form
-- stay mysql Create a table
CREATE TABLE t_rs_zhcs_az01bsbkxx(
id varchar(255) comment ' ID number ',
citty_id varchar(255) comment ' City Code ',
ss_id varchar(255) comment ' Social security card number ',
fkrq varchar(255) comment ' Issuing date ',
yxqz varchar(255) comment ' Valid until ',
aaz502 varchar(255) comment ' Card status ',
aae008 varchar(255) comment ' Bank of deposit ',
aae008b varchar(255) comment ' Bank outlets ',
aae010 varchar(255) comment ' Financial card number ',
aae010a varchar(255) comment ' Financial accounts ',
aae010b varchar(255) comment ' Individual account '
);
-- Import data
LOAD DATA LOCAL INFILE 'D:\\data\\T_RS_ZHCS_AZ01BSBKXX.csv' INTO TABLE pre.t_rs_zhcs_az01bsbkxx FIELDS TERMINATED BY ',';
1.2.5 Employee deposit form
-- stay mysql Create a table
CREATE TABLE t_gjj_sspersons(
spcode varchar(255) comment ' Employee code ',
hjstatus varchar(255) comment ' Payment status ',
sncode varchar(255) comment ' Unit code ',
spname varchar(255) comment ' Personal name ',
id varchar(255) comment ' Id card ',
spcard varchar(255) comment ' Card number ',
sppassword varchar(255) comment ' password ',
zjfdm varchar(255) comment ' Mnemonic code ',
spkhrq varchar(255) comment ' Account opening date ',
spperm varchar(255) comment ' Starting date: ',
spgz decimal comment ' Wage base ',
spsingl decimal comment ' Employee payment rate ',
spjcbl decimal comment ' Unit payment rate ',
spmfact decimal comment ' Monthly payment ',
spmfactzg decimal comment ' Monthly payment of employees ',
spjym varchar(255) comment ' Last paid month and year ',
ncye decimal comment ' The balance at the beginning of the year ',
splast decimal comment ' Term balance ',
dwbfye decimal comment ' Balance unit part ',
grbfye decimal comment ' The balance of employees ',
spmend decimal comment ' balance ',
splastlx decimal comment ' Interest at the beginning of the year ',
spout decimal comment ' Total expenditure during the year ',
spin decimal comment ' Total income during the year ',
bnlx decimal comment ' Current interest product ',
nclx decimal comment ' Term interest product ',
dwhjny varchar(255) comment ' Paid by the company on: ',
zghjny varchar(255) comment ' The month and year of employee remittance ',
btyje decimal comment ' Monthly subsidy payment ',
btye decimal comment ' Balance subsidy ',
btbl decimal comment ' Subsidy payment ratio ',
bthjny varchar(255) comment ' Payment date of subsidy ',
spxh varchar(255) comment ' Sign of account cancellation ',
spzy varchar(255) comment ' Transfer flag ',
spxhrq varchar(255) comment ' Closing date ',
splr decimal comment ' Entry Clerk ',
spoldbankno varchar(255) comment ' Original account number ',
spdk varchar(255) comment ' Whether it's a loan or not ',
spdy varchar(255) comment ' Whether to mortgage ',
zhdj varchar(255) comment ' Account freeze ',
spnote varchar(255) comment ' remarks ',
modifytime varchar(255) comment ' Last modified ',
status varchar(255) comment ' Co branded card status ',
cbank varchar(255) comment ' Co branded card issuing bank ',
bcyje decimal comment ' Supplementary monthly payment ',
bcye decimal comment ' Balance replenishment ',
bcbl decimal comment ' Supplementary contribution ratio ',
bchjny varchar(255) comment ' Add the last remittance month and year ',
zjzl decimal comment ' Types of deposited funds '
);
-- Import data
LOAD DATA LOCAL INFILE 'D:\\data\\T_SSPERSONS_DELTA.csv' INTO TABLE pre.t_gjj_sspersons FIELDS TERMINATED BY ',';
2、 Use Datax Data collection
1 ods_t_fcj_nwrs_sellbargain
ods_t_fcj_nwrs_sellbargain.json
{
"job": {
"setting": {
"speed": {
"channel": 1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"id",
"r_fwzl",
"htydjzmj",
"tntjzmj",
"ftmj",
"time_tjba",
"htzj"
],
"connection": [
{
"table": [
"t_fcj_nwrs_sellbargain"
],
"jdbcUrl": [
"jdbc:mysql://master:3306/pre"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://master:9000",
"fileType": "text",
"path": "/daas/motl/ods/ods_t_fcj_nwrs_sellbargain/ds=20220531",
"fileName": "part",
"column": [
{
"name": "id",
"type": "STRING"
},
{
"name": "r_fwzl",
"type": "STRING"
},
{
"name": "htydjzmj",
"type": "STRING"
},
{
"name": "tntjzmj",
"type": "STRING"
},
{
"name": "ftmj",
"type": "STRING"
},
{
"name": "time_tjba",
"type": "STRING"
},
{
"name": "htzj",
"type":"STRING"
}
],
"writeMode": "append",
"fieldDelimiter": "\t"
}
}
}
]
}
}
Create output directory
hadoop dfs -mkdir -p /daas/motl/ods/ods_t_fcj_nwrs_sellbargain/ds=20220531
Execute the acquisition command
datax.py ods_t_fcj_nwrs_sellbargain.json
stay hive Create an external table
CREATE external TABLE IF NOT EXISTS ods.ods_t_fcj_nwrs_sellbargain(
id STRING comment ' Id card number ',
r_fwzl STRING comment ' Property address ',
htydjzmj STRING comment ' The area of the House agreed in the contract ',
tntjzmj STRING comment ' Building area in the house ',
ftmj STRING comment ' Share the building area of the house ',
time_tjba STRING comment ' Commercial housing filing time ',
htzj STRING comment ' The total contract price '
)PARTITIONED BY
(
ds STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS textfile
location '/daas/motl/ods/ods_t_fcj_nwrs_sellbargain/';
Add partitions to the table
alter table ods.ods_t_fcj_nwrs_sellbargain add partition (ds='20220531');
2 t_gjj_sspersons
ods_t_gjj_sspersons.json
{
"job": {
"setting": {
"speed": {
"channel": 3
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"spcode",
"hjstatus",
"sncode",
"spname",
"id",
"spcard",
"sppassword",
"zjfdm",
"spkhrq",
"spperm",
"spgz",
"spsingl",
"spjcbl",
"spmfact",
"spmfactzg",
"spjym",
"ncye",
"splast",
"dwbfye",
"grbfye",
"spmend",
"splastlx",
"spout",
"spin",
"bnlx",
"nclx",
"dwhjny",
"zghjny",
"btyje",
"btye",
"btbl",
"bthjny",
"spxh",
"spzy",
"spxhrq",
"splr",
"spoldbankno",
"spdk",
"spdy",
"zhdj",
"spnote",
"modifytime",
"status",
"cbank",
"bcyje",
"bcye",
"bcbl",
"bchjny",
"zjzl"
],
"splitPk": "id",
"connection": [
{
"table": [
"t_gjj_sspersons"
],
"jdbcUrl": [
"jdbc:mysql://master:3306/pre"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://master:9000",
"fileType": "text",
"path": "/daas/motl/ods/ods_t_gjj_sspersons/ds=20220531",
"fileName": "part",
"column": [
{
"name": "spcode",
"type": "STRING"
},
{
"name": "hjstatus",
"type": "STRING"
},
{
"name": "sncode",
"type": "STRING"
},
{
"name": "spname",
"type": "STRING"
},
{
"name": "id",
"type": "STRING"
},
{
"name": "spcard",
"type": "STRING"
},
{
"name": "sppassword",
"type": "STRING"
},
{
"name": "zjfdm",
"type": "STRING"
},
{
"name": "spkhrq",
"type": "STRING"
},
{
"name": "spperm",
"type": "STRING"
},
{
"name": "spgz",
"type": "STRING"
},
{
"name": "spsingl",
"type": "STRING"
},
{
"name": "spjcbl",
"type": "STRING"
},
{
"name": "spmfact",
"type": "STRING"
},
{
"name": "spmfactzg",
"type": "STRING"
},
{
"name": "spjym",
"type": "STRING"
},
{
"name": "ncye",
"type": "STRING"
},
{
"name": "splast",
"type": "STRING"
},
{
"name": "dwbfye",
"type": "STRING"
},
{
"name": "grbfye",
"type": "STRING"
},
{
"name": "spmend",
"type": "STRING"
},
{
"name": "splastlx",
"type": "STRING"
},
{
"name": "spout",
"type": "STRING"
},
{
"name": "spin",
"type": "STRING"
},
{
"name": "bnlx",
"type": "STRING"
},
{
"name": "nclx",
"type": "STRING"
},
{
"name": "dwhjny",
"type": "STRING"
},
{
"name": "zghjny",
"type": "STRING"
},
{
"name": "btyje",
"type": "STRING"
},
{
"name": "btye",
"type": "STRING"
},
{
"name": "btbl",
"type": "STRING"
},
{
"name": "bthjny",
"type": "STRING"
},
{
"name": "spxh",
"type": "STRING"
},
{
"name": "spzy",
"type": "STRING"
},
{
"name": "spxhrq",
"type": "STRING"
},
{
"name": "splr",
"type": "STRING"
},
{
"name": "spoldbankno",
"type": "STRING"
},
{
"name": "spdk",
"type": "STRING"
},
{
"name": "spdy",
"type": "STRING"
},
{
"name": "zhdj",
"type": "STRING"
},
{
"name": "spnote",
"type": "STRING"
},
{
"name": "modifytime",
"type": "STRING"
},
{
"name": "status",
"type": "STRING"
},
{
"name": "cbank",
"type": "STRING"
},
{
"name": "bcyje",
"type": "STRING"
},
{
"name": "bcye",
"type": "STRING"
},
{
"name": "bcbl",
"type": "STRING"
},
{
"name": "bchjny",
"type": "STRING"
},
{
"name": "zjzl",
"type": "STRING"
}
],
"writeMode": "append",
"fieldDelimiter": "\t"
}
}
}
]
}
}
Create output directory
hadoop dfs -mkdir -p /daas/motl/ods/ods_t_gjj_sspersons/ds=20220531
perform datax Collect data
datax.py ods_t_gjj_sspersons.json
stay hive Create a table
CREATE external TABLE IF NOT EXISTS ods.ods_t_gjj_sspersons(
spcode string comment ' Employee code ',
hjstatus string comment ' Payment status ',
sncode string comment ' Unit code ',
spname string comment ' Personal name ',
id string comment ' Id card ',
spcard string comment ' Card number ',
sppassword string comment ' password ',
zjfdm string comment ' Mnemonic code ',
spkhrq string comment ' Account opening date ',
spperm string comment ' Starting date: ',
spgz decimal comment ' Wage base ',
spsingl decimal comment ' Employee payment rate ',
spjcbl decimal comment ' Unit payment rate ',
spmfact decimal comment ' Monthly payment ',
spmfactzg decimal comment ' Monthly payment of employees ',
spjym string comment ' Last paid month and year ',
ncye decimal comment ' The balance at the beginning of the year ',
splast decimal comment ' Term balance ',
dwbfye decimal comment ' Balance unit part ',
grbfye decimal comment ' The balance of employees ',
spmend decimal comment ' balance ',
splastlx decimal comment ' Interest at the beginning of the year ',
spout decimal comment ' Total expenditure during the year ',
spin decimal comment ' Total income during the year ',
bnlx decimal comment ' Current interest product ',
nclx decimal comment ' Term interest product ',
dwhjny string comment ' Paid by the company on: ',
zghjny string comment ' The month and year of employee remittance ',
btyje decimal comment ' Monthly subsidy payment ',
btye decimal comment ' Balance subsidy ',
btbl decimal comment ' Subsidy payment ratio ',
bthjny string comment ' Payment date of subsidy ',
spxh string comment ' Sign of account cancellation ',
spzy string comment ' Transfer flag ',
spxhrq string comment ' Closing date ',
splr decimal comment ' Entry Clerk ',
spoldbankno string comment ' Original account number ',
spdk string comment ' Whether it's a loan or not ',
spdy string comment ' Whether to mortgage ',
zhdj string comment ' Account freeze ',
spnote string comment ' remarks ',
modifytime string comment ' Last modified ',
status string comment ' Co branded card status ',
cbank string comment ' Co branded card issuing bank ',
bcyje decimal comment ' Supplementary monthly payment ',
bcye decimal comment ' Balance replenishment ',
bcbl decimal comment ' Supplementary contribution ratio ',
bchjny string comment ' Add the last remittance month and year ',
zjzl decimal comment ' Types of deposited funds '
)PARTITIONED BY
(
ds STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS textfile
location '/daas/motl/ods/ods_t_gjj_sspersons/';
Add partitions
alter table ods.ods_t_gjj_sspersons add partition (ds='20220531');
3 t_gsj_reg_investor
ods_t_gsj_reg_investor.json
{
"job": {
"setting": {
"speed": {
"channel": 1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"id"
],
"connection": [
{
"table": [
"t_gsj_reg_investor"
],
"jdbcUrl": [
"jdbc:mysql://master:3306/pre"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://master:9000",
"fileType": "text",
"path": "/daas/motl/ods/ods_t_gsj_reg_investor/ds=20220531",
"fileName": "part",
"column": [
{
"name": "id",
"type": "STRING"
}
],
"writeMode": "append",
"fieldDelimiter": "\t"
}
}
}
]
}
}
Create output directory
hadoop dfs -mkdir -p /daas/motl/ods/ods_t_gsj_reg_investor/ds=20220531
perform datax
datax.py ods_t_gsj_reg_investor.json
stay hive Create a table
CREATE EXTERNAL TABLE IF NOT EXISTS ods.ods_t_gsj_reg_investor(
id STRING comment ' Id card number '
)PARTITIONED BY
(
ds STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS textfile
location '/daas/motl/ods/ods_t_gsj_reg_investor';
Add partitions
alter table ods.ods_t_gsj_reg_investor add partition (ds='20220531');
4 t_gsj_reg_legrepre
ods_t_gsj_reg_legrepre.json
{
"job": {
"setting": {
"speed": {
"channel": 1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"id",
"position",
"tel",
"appounit",
"accdside",
"posbrmode",
"offhfrom",
"offhto",
"stufftype"
],
"connection": [
{
"table": [
"t_gsj_reg_legrepre"
],
"jdbcUrl": [
"jdbc:mysql://master:3306/pre"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://master:9000",
"fileType": "text",
"path": "/daas/motl/ods/ods_t_gsj_reg_legrepre/ds=20220531",
"fileName": "part",
"column": [
{
"name": "id","type": "STRING"},
{
"name": "position","type": "STRING"},
{
"name": "tel","type": "STRING"},
{
"name": "appounit","type": "STRING"},
{
"name": "accdside","type": "STRING"},
{
"name": "posbrmode","type": "STRING"},
{
"name": "offhfrom","type": "STRING"},
{
"name": "offhto","type": "STRING"},
{
"name": "stufftype","type": "STRING"}
],
"writeMode": "append",
"fieldDelimiter": "\t"
}
}
}
]
}
}
Create output directory
hadoop dfs -mkdir -p /daas/motl/ods/ods_t_gsj_reg_legrepre/ds=20220531
perform datax
datax.py ods_t_gsj_reg_legrepre.json
stay hive CSCEC table
CREATE EXTERNAL TABLE IF NOT EXISTS ods.ods_t_gsj_reg_legrepre(
id STRING comment ' ID No ',
position STRING comment ' position ',
tel STRING comment ' contact number ',
appounit STRING comment ' Appointment and removal institution ',
accdside STRING comment ' Sending unit ',
posbrmode STRING comment ' The way of production ',
offhfrom STRING comment ' From the term of office ',
offhto STRING comment ' The term of office is limited ',
stufftype STRING comment ' Data type code '
)PARTITIONED BY
(
ds STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
LOCATION '/daas/motl/ods/ods_t_gsj_reg_legrepre/';
Add partitions
alter table ods.ods_t_gsj_reg_legrepre add partition (ds='20220531');
5、t_rs_zhcs_az01bsbkxx
ods_t_rs_zhcs_az01bsbkxx.json
{
"job": {
"setting": {
"speed": {
"channel": 1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"id",
"citty_id",
"ss_id",
"fkrq",
"yxqz",
"aaz502",
"aae008",
"aae008b",
"aae010",
"aae010a",
"aae010b"
],
"connection": [
{
"table": [
"t_rs_zhcs_az01bsbkxx"
],
"jdbcUrl": [
"jdbc:mysql://master:3306/pre"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://master:9000",
"fileType": "text",
"path": "/daas/motl/ods/ods_t_rs_zhcs_az01bsbkxx/ds=20220531",
"fileName": "part",
"column": [
{
"name": "id",
"type": "STRING"
},
{
"name": "citty_id",
"type": "STRING"
},
{
"name": "ss_id",
"type": "STRING"
},
{
"name": "fkrq",
"type": "STRING"
},
{
"name": "yxqz",
"type": "STRING"
},
{
"name": "aaz502",
"type": "STRING"
},
{
"name": "aae008",
"type": "STRING"
},
{
"name": "aae008b",
"type": "STRING"
},
{
"name": "aae010",
"type": "STRING"
},
{
"name": "aae010a",
"type": "STRING"
},
{
"name": "aae010b",
"type": "STRING"
}
],
"writeMode": "append",
"fieldDelimiter": "\t"
}
}
}
]
}
}
Create output directory
hadoop dfs -mkdir -p /daas/motl/ods/ods_t_rs_zhcs_az01bsbkxx/ds=20220531
perform datax
datax.py ods_t_rs_zhcs_az01bsbkxx.json
establish hive surface
CREATE external TABLE IF NOT EXISTS ods.ods_t_rs_zhcs_az01bsbkxx(
id string comment ' ID number ',
citty_id string comment ' City Code ',
ss_id string comment ' Social security card number ',
fkrq string comment ' Issuing date ',
yxqz string comment ' Valid until ',
aaz502 string comment ' Card status ',
aae008 string comment ' Bank of deposit ',
aae008b string comment ' Bank outlets ',
aae010 string comment ' Financial card number ',
aae010a string comment ' Financial accounts ',
aae010b string comment ' Individual account '
)PARTITIONED BY
(
ds STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS textfile
location '/daas/motl/ods/ods_t_rs_zhcs_az01bsbkxx/';
Add partitions
alter table ods.ods_t_rs_zhcs_az01bsbkxx add partition (ds='20220531');
3、 Incremental data
1 Save the data source to the front-end database
CREATE TABLE t_ga_hjxx_czrkjbxx(
GMSFHM varchar(200) comment' Citizenship number ',
QFJG varchar(200) comment' Issuing authority ',
YXQXQSRQ varchar(200) comment' Valid from ',
YXQXJZRQ varchar(200) comment' Expiry date ',
XM varchar(200) comment' full name ',
CYM varchar(200) comment' Name used before ',
XMPY varchar(200) comment' Name Pinyin ',
CYMPY varchar(200) comment' Former name Pinyin ',
XB varchar(200) comment' Gender ',
MZ varchar(200) comment' nation ',
CSRQ varchar(200) comment' Date of birth ',
CSSJ varchar(200) comment' time of birth ',
CSDGJDQ varchar(200) comment' Country of birth （ region ）',
CSDSSXQ varchar(200) comment' Place of birth: Province, city and county （ District ）',
CSDXZ varchar(200) comment' Detailed address of place of birth ',
DHHM varchar(200) comment' Phone number ',
JHRYXM varchar(200) comment' Name of the guardian ',
JHRYGMSFHM varchar(200) comment' Guardian - citizen ID number ',
JHRYJHGX varchar(200) comment' Guardian guardian relationship ',
JHREXM varchar(200) comment' Name of the second guardian ',
JHREGMSFHM varchar(200) comment' ID number of Guardian 2 ',
JHREJHGX varchar(200) comment' Guardian II guardianship relationship ',
FQXM varchar(200) comment' Father's name ',
FQGMSFHM varchar(200) comment' Father's citizenship number ',
MQXM varchar(200) comment' Mother's name ',
MQGMSFHM varchar(200) comment' Mother's citizenship number ',
POXM varchar(200) comment' Spouse's name ',
POGMSFHM varchar(200) comment' Spouse citizenship number ',
JGGJDQ varchar(200) comment' Country of origin （ region ）',
JGSSXQ varchar(200) comment' Native place: Province, city and county （ District ）',
ZJXY varchar(200) comment' Religious beliefs ',
WHCD varchar(200) comment' degree of education ',
HYZK varchar(200) comment' Marital status ',
BYZK varchar(200) comment' The state of military service ',
SG varchar(200) comment' height ',
XX varchar(200) comment' Blood type ',
ZY varchar(200) comment' occupation ',
ZYLB varchar(200) comment' Occupation category ',
FWCS varchar(200) comment' Service premises ',
XXJB varchar(200) comment' Information level ',
HSQL varchar(200) comment' When to move to ',
HYQL varchar(200) comment' Why did you come here ',
HGJDQQL varchar(200) comment' He Guoguo （ region ） Move in ',
HSSXQQL varchar(200) comment' What province, city and county （ District ） Move in ',
HXZQL varchar(200) comment' He detailed address moved here ',
HSLBZ varchar(200) comment' When did you come to this address ',
HYLBZ varchar(200) comment' Why did you come to this address ',
HGJDQLBZ varchar(200) comment' He Guoguo （ region ） Home address ',
HSSSQLBZ varchar(200) comment' What province, city and county （ District ） Home address ',
HXZLBZ varchar(200) comment' He detailed address comes to this address ',
SWRQ varchar(200) comment' Date of death ',
SWZXLB varchar(200) comment' Death cancellation category ',
SWZXRQ varchar(200) comment' Death cancellation date ',
QCRQ varchar(200) comment' Move out date ',
QCZXLB varchar(200) comment' Move out of the logout category ',
QWDGJDQ varchar(200) comment' The country to which you are moving （ region ）',
QWDSSXQ varchar(200) comment' Move to local provinces, cities and counties （ District ）',
QWDXZ varchar(200) comment' The detailed address of the relocation site ',
CSZMBH varchar(200) comment' Birth certificate number ',
CSZQFRQ varchar(200) comment' Date of issue of birth certificate ',
HYLB varchar(200) comment' Industry categories ',
QTSSXQ varchar(200) comment' Other provinces, cities and counties （ District ）',
QTZZ varchar(200) comment' Other addresses ',
RYLB varchar(200) comment' Personnel category ',
HB varchar(200) comment' Household type ',
YHZGX varchar(200) comment' The relationship with the head of household ',
RYZT varchar(200) comment' Personnel status ',
RYSDZT varchar(200) comment' Personnel locked status ',
LXDBID varchar(200) comment' offline DBID',
BZ varchar(200) comment' remarks ',
JLBZ varchar(200) comment' Record mark ',
YWNR varchar(200) comment' Business content ',
CJHJYWID varchar(200) comment' Create registered residence business ID',
CCHJYWID varchar(200) comment' Remove the registered residence business ID',
QYSJ varchar(200) comment' Starting time ',
JSSJ varchar(200) comment' End time ',
CXBZ varchar(200) comment' Reversal flag ',
JLX varchar(200) comment' Streets and alleys ',
MLPH varchar(200) comment' door （ floor ） Brand ',
MLXZ varchar(200) comment' door （ floor ） detailed address ',
PCS varchar(200) comment' Police Station ',
ZRQ varchar(200) comment' zone of responsibility ',
XZJD varchar(200) comment' Township （ The street ）',
JCWH varchar(200) comment' Residence （ The village ） The Committee ',
PXH varchar(200) comment' Sort number ',
MLPID varchar(200) comment' door （ floor ） card ID',
SSXQ varchar(200) comment' Provinces, cities and counties （ District ）',
HH varchar(200) comment' Account number ',
HLX varchar(200) comment' Type of household ',
HHID varchar(200) comment' Account number ID',
BDFW varchar(200) comment' Scope of change ',
XXQYSJ varchar(200) comment' Information activation time ',
DHHM2 varchar(200) comment' Phone number 2',
GXSJ varchar(200) comment' Update timestamp ',
ZXSJ varchar(200) comment' Death , Logout time of logout ',
CRYNBID varchar(200) comment' personnel nbidchar',
CGMSFHM varchar(200) comment' ID number 2',
GXSJD varchar(200) comment' Date format update time ',
ZJLB varchar(200) comment' Certificate type ',
last_mod TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);
-- Import data
LOAD DATA local INFILE 'D:\\data\\T_GA_HJXX_CZRKJBXX1.csv' INTO TABLE pre.t_ga_hjxx_czrkjbxx FIELDS TERMINATED BY ',' ;
LOAD DATA local INFILE 'C:\\data\\T_GA_HJXX_CZRKJBXX2.csv' INTO TABLE pre.t_ga_hjxx_czrkjbxx FIELDS TERMINATED BY ',' ;
2 use datax Collect data
{
"job": {
"setting": {
"speed": {
"channel": 1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"GMSFHM",
"QFJG",
"YXQXQSRQ",
"YXQXJZRQ",
"XM",
"CYM",
"XMPY",
"CYMPY",
"XB",
"MZ",
"CSRQ",
"CSSJ",
"CSDGJDQ",
"CSDSSXQ",
"CSDXZ",
"DHHM",
"JHRYXM",
"JHRYGMSFH",
"JHRYJHGX",
"JHREXM",
"JHREGMSFH",
"JHREJHGX",
"FQXM",
"FQGMSFHM",
"MQXM",
"MQGMSFHM",
"POXM",
"POGMSFHM",
"JGGJDQ",
"JGSSXQ",
"ZJXY",
"WHCD",
"HYZK",
"BYZK",
"SG",
"XX",
"ZY",
"ZYLB",
"FWCS",
"XXJB",
"HSQL",
"HYQL",
"HGJDQQL",
"HSSXQQL",
"HXZQL",
"HSLBZ",
"HYLBZ",
"HGJDQLBZ",
"HSSSQLBZ",
"HXZLBZ",
"SWRQ",
"SWZXLB",
"SWZXRQ",
"QCRQ",
"QCZXLB",
"QWDGJDQ",
"QWDSSXQ",
"QWDXZ",
"CSZMBH",
"CSZQFRQ",
"HYLB",
"QTSSXQ",
"QTZZ",
"RYLB",
"HB",
"YHZGX",
"RYZT",
"RYSDZT",
"LXDBID",
"BZ",
"JLBZ",
"YWNR",
"CJHJYWID",
"CCHJYWID",
"QYSJ",
"JSSJ",
"CXBZ",
"JLX",
"MLPH",
"MLXZ",
"PCS",
"ZRQ",
"XZJD",
"JCWH",
"PXH",
"MLPID",
"SSXQ",
"HH",
"HLX",
"HHID",
"BDFW",
"XXQYSJ",
"DHHM2",
"GXSJ",
"ZXSJ",
"CRYNBID",
"CGMSFHM",
"GXSJD",
"ZJLB",
"last_mod"
],
"connection": [
{
"table": [
"t_ga_hjxx_czrkjbxx"
],
"jdbcUrl": [
"jdbc:mysql://master:3306/pre?useUnicode=true&characterEncoding=utf-8"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://master:9000",
"fileType": "text",
"path": "/daas/motl/ods/ods_t_ga_hjxx_czrkjbxx",
"fileName": "part",
"column": [
{
"name":"GMSFHM",
"type":"STRING"
},
{
"name":"QFJG",
"type":"STRING"
},
{
"name":"YXQXQSRQ",
"type":"STRING"
},
{
"name":"YXQXJZRQ",
"type":"STRING"
},
{
"name":"XM",
"type":"STRING"
},
{
"name":"CYM",
"type":"STRING"
},
{
"name":"XMPY",
"type":"STRING"
},
{
"name":"CYMPY",
"type":"STRING"
},
{
"name":"XB",
"type":"STRING"
},
{
"name":"MZ",
"type":"STRING"
},
{
"name":"CSRQ",
"type":"STRING"
},
{
"name":"CSSJ",
"type":"STRING"
},
{
"name":"CSDGJDQ",
"type":"STRING"
},
{
"name":"CSDSSXQ",
"type":"STRING"
},
{
"name":"CSDXZ",
"type":"STRING"
},
{
"name":"DHHM",
"type":"STRING"
},
{
"name":"JHRYXM",
"type":"STRING"
},
{
"name":"JHRYGMSFH",
"type":"STRING"
},
{
"name":"JHRYJHGX",
"type":"STRING"
},
{
"name":"JHREXM",
"type":"STRING"
},
{
"name":"JHREGMSFH",
"type":"STRING"
},
{
"name":"JHREJHGX",
"type":"STRING"
},
{
"name":"FQXM",
"type":"STRING"
},
{
"name":"FQGMSFHM",
"type":"STRING"
},
{
"name":"MQXM",
"type":"STRING"
},
{
"name":"MQGMSFHM",
"type":"STRING"
},
{
"name":"POXM",
"type":"STRING"
},
{
"name":"POGMSFHM",
"type":"STRING"
},
{
"name":"JGGJDQ",
"type":"STRING"
},
{
"name":"JGSSXQ",
"type":"STRING"
},
{
"name":"ZJXY",
"type":"STRING"
},
{
"name":"WHCD",
"type":"STRING"
},
{
"name":"HYZK",
"type":"STRING"
},
{
"name":"BYZK",
"type":"STRING"
},
{
"name":"SG",
"type":"STRING"
},
{
"name":"XX",
"type":"STRING"
},
{
"name":"ZY",
"type":"STRING"
},
{
"name":"ZYLB",
"type":"STRING"
},
{
"name":"FWCS",
"type":"STRING"
},
{
"name":"XXJB",
"type":"STRING"
},
{
"name":"HSQL",
"type":"STRING"
},
{
"name":"HYQL",
"type":"STRING"
},
{
"name":"HGJDQQL",
"type":"STRING"
},
{
"name":"HSSXQQL",
"type":"STRING"
},
{
"name":"HXZQL",
"type":"STRING"
},
{
"name":"HSLBZ",
"type":"STRING"
},
{
"name":"HYLBZ",
"type":"STRING"
},
{
"name":"HGJDQLBZ",
"type":"STRING"
},
{
"name":"HSSSQLBZ",
"type":"STRING"
},
{
"name":"HXZLBZ",
"type":"STRING"
},
{
"name":"SWRQ",
"type":"STRING"
},
{
"name":"SWZXLB",
"type":"STRING"
},
{
"name":"SWZXRQ",
"type":"STRING"
},
{
"name":"QCRQ",
"type":"STRING"
},
{
"name":"QCZXLB",
"type":"STRING"
},
{
"name":"QWDGJDQ",
"type":"STRING"
},
{
"name":"QWDSSXQ",
"type":"STRING"
},
{
"name":"QWDXZ",
"type":"STRING"
},
{
"name":"CSZMBH",
"type":"STRING"
},
{
"name":"CSZQFRQ",
"type":"STRING"
},
{
"name":"HYLB",
"type":"STRING"
},
{
"name":"QTSSXQ",
"type":"STRING"
},
{
"name":"QTZZ",
"type":"STRING"
},
{
"name":"RYLB",
"type":"STRING"
},
{
"name":"HB",
"type":"STRING"
},
{
"name":"YHZGX",
"type":"STRING"
},
{
"name":"RYZT",
"type":"STRING"
},
{
"name":"RYSDZT",
"type":"STRING"
},
{
"name":"LXDBID",
"type":"STRING"
},
{
"name":"BZ",
"type":"STRING"
},
{
"name":"JLBZ",
"type":"STRING"
},
{
"name":"YWNR",
"type":"STRING"
},
{
"name":"CJHJYWID",
"type":"STRING"
},
{
"name":"CCHJYWID",
"type":"STRING"
},
{
"name":"QYSJ",
"type":"STRING"
},
{
"name":"JSSJ",
"type":"STRING"
},
{
"name":"CXBZ",
"type":"STRING"
},
{
"name":"JLX",
"type":"STRING"
},
{
"name":"MLPH",
"type":"STRING"
},
{
"name":"MLXZ",
"type":"STRING"
},
{
"name":"PCS",
"type":"STRING"
},
{
"name":"ZRQ",
"type":"STRING"
},
{
"name":"XZJD",
"type":"STRING"
},
{
"name":"JCWH",
"type":"STRING"
},
{
"name":"PXH",
"type":"STRING"
},
{
"name":"MLPID",
"type":"STRING"
},
{
"name":"SSXQ",
"type":"STRING"
},
{
"name":"HH",
"type":"STRING"
},
{
"name":"HLX",
"type":"STRING"
},
{
"name":"HHID",
"type":"STRING"
},
{
"name":"BDFW",
"type":"STRING"
},
{
"name":"XXQYSJ",
"type":"STRING"
},
{
"name":"DHHM2",
"type":"STRING"
},
{
"name":"GXSJ",
"type":"STRING"
},
{
"name":"ZXSJ",
"type":"STRING"
},
{
"name":"CRYNBID",
"type":"STRING"
},
{
"name":"CGMSFHM",
"type":"STRING"
},
{
"name":"GXSJD",
"type":"STRING"
},
{
"name":"ZJLB",
"type":"STRING"
},
{
"name":"last_mod",
"type":"STRING"
}
],
"writeMode": "append",
"fieldDelimiter": "\t",
}
}
}
]
}
}
CREATE external TABLE IF NOT EXISTS ods.ods_t_ga_hjxx_czrkjbxx_delta(
GMSFHM string comment' Citizenship number ',
QFJG string comment' Issuing authority ',
YXQXQSRQ string comment' Valid from ',
YXQXJZRQ string comment' Expiry date ',
XM string comment' full name ',
CYM string comment' Name used before ',
XMPY string comment' Name Pinyin ',
CYMPY string comment' Former name Pinyin ',
XB string comment' Gender ',
MZ string comment' nation ',
CSRQ string comment' Date of birth ',
CSSJ string comment' time of birth ',
CSDGJDQ string comment' Country of birth （ region ）',
CSDSSXQ string comment' Place of birth: Province, city and county （ District ）',
CSDXZ string comment' Detailed address of place of birth ',
DHHM string comment' Phone number ',
JHRYXM string comment' Name of the guardian ',
JHRYGMSFHM string comment' Guardian - citizen ID number ',
JHRYJHGX string comment' Guardian guardian relationship ',
JHREXM string comment' Name of the second guardian ',
JHREGMSFHM string comment' ID number of Guardian 2 ',
JHREJHGX string comment' Guardian II guardianship relationship ',
FQXM string comment' Father's name ',
FQGMSFHM string comment' Father's citizenship number ',
MQXM string comment' Mother's name ',
MQGMSFHM string comment' Mother's citizenship number ',
POXM string comment' Spouse's name ',
POGMSFHM string comment' Spouse citizenship number ',
JGGJDQ string comment' Country of origin （ region ）',
JGSSXQ string comment' Native place: Province, city and county （ District ）',
ZJXY string comment' Religious beliefs ',
WHCD string comment' degree of education ',
HYZK string comment' Marital status ',
BYZK string comment' The state of military service ',
SG string comment' height ',
XX string comment' Blood type ',
ZY string comment' occupation ',
ZYLB string comment' Occupation category ',
FWCS string comment' Service premises ',
XXJB string comment' Information level ',
HSQL string comment' When to move to ',
HYQL string comment' Why did you come here ',
HGJDQQL string comment' He Guoguo （ region ） Move in ',
HSSXQQL string comment' What province, city and county （ District ） Move in ',
HXZQL string comment' He detailed address moved here ',
HSLBZ string comment' When did you come to this address ',
HYLBZ string comment' Why did you come to this address ',
HGJDQLBZ string comment' He Guoguo （ region ） Home address ',
HSSSQLBZ string comment' What province, city and county （ District ） Home address ',
HXZLBZ string comment' He detailed address comes to this address ',
SWRQ string comment' Date of death ',
SWZXLB string comment' Death cancellation category ',
SWZXRQ string comment' Death cancellation date ',
QCRQ string comment' Move out date ',
QCZXLB string comment' Move out of the logout category ',
QWDGJDQ string comment' The country to which you are moving （ region ）',
QWDSSXQ string comment' Move to local provinces, cities and counties （ District ）',
QWDXZ string comment' The detailed address of the relocation site ',
CSZMBH string comment' Birth certificate number ',
CSZQFRQ string comment' Date of issue of birth certificate ',
HYLB string comment' Industry categories ',
QTSSXQ string comment' Other provinces, cities and counties （ District ）',
QTZZ string comment' Other addresses ',
RYLB string comment' Personnel category ',
HB string comment' Household type ',
YHZGX string comment' The relationship with the head of household ',
RYZT string comment' Personnel status ',
RYSDZT string comment' Personnel locked status ',
LXDBID string comment' offline DBID',
BZ string comment' remarks ',
JLBZ string comment' Record mark ',
YWNR string comment' Business content ',
CJHJYWID string comment' Create registered residence business ID',
CCHJYWID string comment' Remove the registered residence business ID',
QYSJ string comment' Starting time ',
JSSJ string comment' End time ',
CXBZ string comment' Reversal flag ',
JLX string comment' Streets and alleys ',
MLPH string comment' door （ floor ） Brand ',
MLXZ string comment' door （ floor ） detailed address ',
PCS string comment' Police Station ',
ZRQ string comment' zone of responsibility ',
XZJD string comment' Township （ The street ）',
JCWH string comment' Residence （ The village ） The Committee ',
PXH string comment' Sort number ',
MLPID string comment' door （ floor ） card ID',
SSXQ string comment' Provinces, cities and counties （ District ）',
HH string comment' Account number ',
HLX string comment' Type of household ',
HHID string comment' Account number ID',
BDFW string comment' Scope of change ',
XXQYSJ string comment' Information activation time ',
DHHM2 string comment' Phone number 2',
GXSJ string comment' Update timestamp ',
ZXSJ string comment' Death , Logout time of logout ',
CRYNBID string comment' personnel nbidchar',
CGMSFHM string comment' ID number 2',
GXSJD string comment' Date format update time ',
ZJLB string comment' Certificate type ',
last_mod string comment' Data update time '
)PARTITIONED BY
(
ds STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS textfile
location '/daas/motl/ods/ods_t_ga_hjxx_czrkjbxx_delta/';
4、ads Layer summary
1、 Statistics of the total number of births
select substr(CSRQ,1,4),-- year
sum(ZRS),-- The total number of
sum(case when XB==1 then ZRS else 0 end),-- Number of men by sex
sum(case when XB==2 then ZRS else 0 end)-- Number of women of gender
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,4)
-- Number of people
select
substr(CSRQ,1,4),-- year
"",
"",
sum(ZRS),-- The total number of
"S"
from dws.dws_rk_csxx_information_sum_d_f
where substr(CSRQ,1,4)=2019
group by substr(CSRQ,1,4)
union all
-- Gender indicators
select
substr(CSRQ,1,4),-- year
XB,
"",
sum(ZRS),-- The total number of
"X"
from dws.dws_rk_csxx_information_sum_d_f
where substr(CSRQ,1,4)=2019
group by substr(CSRQ,1,4),XB
union all
select
substr(CSRQ,1,4),-- year
"",
XZJD,
sum(ZRS),-- The total number of
"Z"
from dws.dws_rk_csxx_information_sum_d_f
where substr(CSRQ,1,4)=2019
group by substr(CSRQ,1,4),XZJD
union all
select
substr(CSRQ,1,4),-- year
"",
JCWH,
sum(ZRS),-- The total number of
"J"
from dws.dws_rk_csxx_information_sum_d_f
where substr(CSRQ,1,4)=2019
group by substr(CSRQ,1,4),JCWH
data format
sum= The total number of
xb= Number of persons by sex
XZJD= The number of townships
JCWH= neighborhood committee
2、 Calculate the year-on-year birth growth rate
Year on year birth growth rate
Grouped by year
-- Total birth index
insert overwrite table ads.ads_rk_csxx_information_screen partition(ds='20220602')
select
y,"Y",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,
sum(rs) over(order by y) as sum_rs -- The total number of births accumulated to that year
from(
select
substr(CSRQ,1,4) as y,-- year
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,4)
) as a
) as b
union all
-- Gender indicators
select
y,"Y",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,4) y,-- year
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,4),XB
) as a
) as b
union all
-- Township indicators
select
y,"Y",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,4) y,-- year
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,4),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"Y",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,4) y,-- year
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,4),JCWH
) as a
) as b
Group by month
select
y,"M",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,-- Number of births last month
sum(rs) over(order by y) as sum_rs -- Total number of births accumulated to the current month
from(
select
substr(CSRQ,1,6) as y,-- year
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,6)
) as a
) as b
union all
-- Gender indicators
select
y,"M",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,6) y,-- month
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,6),XB
) as a
) as b
union all
-- Township indicators
select
y,"M",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,6) y,-- month
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,6),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"M",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,6) y,-- month
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by substr(CSRQ,1,6),JCWH
) as a
) as b
quarter(from_unixtime(unix_timestamp('20160406','yyyyMMdd',),'yyyy-MM-dd'))
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd',),'yyyy-MM-dd')))
Group by quarter
select
y,"Q",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,-- Number of births last month
sum(rs) over(order by y) as sum_rs -- Total number of births accumulated to the current month
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) as y,-- quarter
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd')))
) as a
) as b
union all
-- Gender indicators
select
y,"Q",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) y,-- quarter
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))),XB
) as a
) as b
union all
-- Township indicators
select
y,"Q",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) y,-- quarter
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"Q",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) y,-- quarter
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))),JCWH
) as a
) as b
concat(substr('20220501',1,4),case when month(from_unixtime(unix_timestamp('20220501','yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end)
select
y,"H",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,-- Number of births last month
sum(rs) over(order by y) as sum_rs -- Total number of births accumulated to the current month
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) as y,-- Half a year
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end)
) as a
) as b
union all
-- Gender indicators
select
y,"H",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) y,-- Half a year
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end),XB
) as a
) as b
union all
-- Township indicators
select
y,"H",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) y,-- Half a year
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"H",d,score_type,rs,(rs-last_rs)/last_rs,rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) y,-- Half a year
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end),JCWH
) as a
) as b
3、dql Up dispatch （ Final sql）
dql function
-- Create view
drop view IF EXISTS dws.dws_rk_csxx_information_sum_d_f_temp;
create view dws.dws_rk_csxx_information_sum_d_f_temp as
select * from dws.dws_rk_csxx_information_sum_d_f
where ds='${ds}';
-- Cache the table
cache table dws.dws_rk_csxx_information_sum_d_f_temp;
insert overwrite table ads.ads_rk_csxx_information_screen partition(ds='${ds}')
-- Annual indicators
-- An indicator of total births
select
y,"Y",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,
sum(rs) over(order by y) as sum_rs -- The total number of births accumulated to that year
from(
select
substr(CSRQ,1,4) as y,-- year
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,4)
) as a
) as b
union all
-- Gender indicators
select
y,"Y",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,4) y,-- year
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,4),XB
) as a
) as b
union all
-- Township indicators
select
y,"Y",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,4) y,-- year
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,4),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"Y",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,4) y,-- year
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,4),JCWH
) as a
) as b
union all
select
y,"M",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,-- Number of births last month
sum(rs) over(order by y) as sum_rs -- Total number of births accumulated to the current month
from(
select
substr(CSRQ,1,6) as y,-- year
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,6)
) as a
) as b
union all
-- Gender indicators
select
y,"M",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,6) y,-- month
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,6),XB
) as a
) as b
union all
-- Township indicators
select
y,"M",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,6) y,-- month
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,6),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"M",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
substr(CSRQ,1,6) y,-- month
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by substr(CSRQ,1,6),JCWH
) as a
) as b
union all
select
y,"Q",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,-- Number of births last month
sum(rs) over(order by y) as sum_rs -- Total number of births accumulated to the current month
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) as y,-- quarter
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd')))
) as a
) as b
union all
-- Gender indicators
select
y,"Q",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) y,-- quarter
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))),XB
) as a
) as b
union all
-- Township indicators
select
y,"Q",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) y,-- quarter
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"Q",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))) y,-- quarter
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),quarter(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))),JCWH
) as a
) as b
union all
select
y,"H",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(order by y) as last_rs,-- Number of births last month
sum(rs) over(order by y) as sum_rs -- Total number of births accumulated to the current month
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) as y,-- Half a year
"" d,
sum(ZRS) rs,-- The total number of
"S" as score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end)
) as a
) as b
union all
-- Gender indicators
select
y,"H",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) y,-- Half a year
XB d,
sum(ZRS) rs,-- The total number of
"X" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end),XB
) as a
) as b
union all
-- Township indicators
select
y,"H",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) y,-- Half a year
XZJD d,
sum(ZRS) rs,-- The total number of
"Z" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end),XZJD
) as a
) as b
union all
-- Neighborhood committee indicators
select
y,"H",d,score_type,rs,coalesce((rs-last_rs)/last_rs,0.0),rs/sum_rs
from(
select
y,d,rs,score_type,
lag(rs,1,0) over(partition by d order by y) last_rs,
sum(rs) over(partition by d order by y) sum_rs
from(
select
concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end) y,-- Half a year
JCWH d,
sum(ZRS) rs,-- The total number of
"J" score_type
from dws.dws_rk_csxx_information_sum_d_f_temp where ds='${ds}'
group by concat(substr(CSRQ,1,4),case when month(from_unixtime(unix_timestamp(CSRQ,'yyyyMMdd'),'yyyy-MM-dd'))<=6 then 1 else 2 end),JCWH
) as a
) as b;
uncache table dws.dws_rk_csxx_information_sum_d_f_temp;
drop view IF EXISTS dws.dws_rk_csxx_information_sum_d_f_temp;
5、 Large screen display
边栏推荐
- Lianrui electronics made an appointment with you with SIFA to see two network cards in the industry's leading industrial automation field first
- Network security construction in 5g Era
- 推荐一款免费的内网穿透开源软件，可以在测试本地开发微信公众号使用
- Using keras to build the basic model yingtailing flower
- New product pre-sale: 25g optical network card based on Intel 800 series is coming
- Network adapter purchase guide
- Huawei equipment is configured to access the virtual private network through GRE tunnel
- 【入门级基础】Node基础知识总结
- KD-Tree and LSH
- Vins fusion GPS fusion part
猜你喜欢
PCB走线到底能承载多大电流
New library goes online | cnopendata immovable cultural relic data
KD-Tree and LSH
Cross modal retrieval | visual representation learning
Pychart displays pictures with imshow
MySQL regularly deletes expired data.
Simple knowledge distillation
Huawei equipment is configured with bgp/mpls IP virtual private network address space overlap
Click the icon is not sensitive how to adjust?
Lianrui: how to rationally see the independent R & D of domestic CPU and the development of domestic hardware
随机推荐
QT Road (2) -- HelloWorld
jvm调优六：GC日志分析和常量池详解
About custom comparison methods of classes and custom methods of sort functions
Apply the intelligent OCR identification technology of Shenzhen Yanchang technology to break through the bottleneck of medical bill identification at one stroke. Efficient claim settlement is not a dr
BP neural network derivation + Example
wxParse解析iframe播放视频
Network security construction in 5g Era
Yolov5 training personal data set summary
Sealem finance builds Web3 decentralized financial platform infrastructure
Opencv learning path (2-4) -- Deep parsing cvtcolor function
Using keras to build the basic model yingtailing flower
自定义View之基础篇
Pytoch machine learning GPU usage (conversion from CPU to GPU)
Poverty has nothing to do with suffering
[NIPS2021]MLP-Mixer: An all-MLP Architecture for Vision
How to purchase 25g optical network card
Huawei equipment is configured with bgp/mpls IP virtual private network address space overlap
go MPG
What is the difference between a wired network card and a wireless network card?
White Gaussian noise (WGN)