Use sqoop to export ads layer data to MySQL

background

• Use Sqoop hold ADS Export layer data to MySQL
• Use sqoop export To add when --columns, Avoid some strange mistakes
• Use Regular expressions Get field name

technological process

1. ADS Layers are not partitioned , Uncompressed , Bank deposit
2. ADS Floor construction table SQL Have separate documents , If the table is updated, the table creation statement of the file must be updated
3. Table name ：ADS Layer of HIVE Table has ads_ Prefix , Corresponding to MySQL Remove the prefix when creating the table
4. Field ：ADS Layer table and MySQL Tabular The field name and field order should be consistent , use ` Symbol package
5. Traverse ADS Layered TABLE statement , Get with regular expression Table name 、 All field names
6. It is said that Sqoop command

Code

ADS Layered TABLE statement （ADS Floor construction table .sql）

-- HIVE Create table statement , Field use ` Symbol package , The table name does not need to be wrapped 
CREATE EXTERNAL TABLE ads_purchase_order_info (
`prch_order_id` BIGINT COMMENT ' Purchase order header id',
`exfactory_total_price` DOUBLE COMMENT ' Total ex factory price ',
`insert_time` STRING COMMENT ' Data insertion date '
) COMMENT ' Purchasing information ';

MySQL Create table statement

CREATE TABLE purchase_order_info (
`prch_order_id` bigint COMMENT ' Purchase order header id',
`exfactory_total_price` DOUBLE COMMENT ' Total ex factory price ',
`insert_time` text COMMENT ' Data insertion date ',
PRIMARY KEY (`prch_order_id`)
) COMMENT ' Purchasing information ';

Python

class Sqoop(Shell):
    def sqoop(self, cmd):
        return self.sh_cmd_and_alert(' '.join(cmd.split()))

    def sqoop_export(self, mysql_tb, export_dir, columns='', update_mode='allowinsert', update_key='prch_order_id'):
        """ --columns The default is all columns ; Advice and , Avoid some inexplicable bug --update-mode The default is updateonly, Can be changed to allowinsert --update-key Is an anchor column for updating ; Multiple columns are separated by commas  """
        return self.sqoop(r''' {sqoop} export --connect jdbc:mysql://{host}:{port}/{database} --username '{username}' --password '{password}' --table {table} --num-mappers 1 --input-fields-terminated-by '\001' --input-null-string '\\N' --input-null-non-string '\\N' --export-dir '{export_dir}' {columns} '''.format(
            sqoop=self.get('sqoop', 'sqoop'),
            host=self.get('mysql_host', 'localhost'),
            port=self.get('mysql_port', '3306'),
            database=self['mysql_db'],
            username=self.get('mysql_user', 'root'),
            password=self['mysql_pwd'],
            table=mysql_tb,
            export_dir=export_dir,
            columns=columns,
        ))

from re import findall

s = get_sqoop()
for ads_ddl in read_sql_file('ADS Floor construction table .sql').split(';')[:-1]:
    columns = '--columns ' + ','.join(findall('`([^`]+)`', ads_ddl))
    hive_tb = findall(r'CREATE EXTERNAL TABLE (\S+)', ads_ddl)[0]
    mysql_tb = hive_tb.replace('ads_', '')
    print(s.sqoop_export(mysql_tb, EXPORT_DIR_PREFIX + hive_tb, columns))

---

because it is you