Tips for speeding up kettle insertion

Our company uses the most for data transmission Kettle,Kettle It's an open source software , It is convenient to configure data transmission between heterogeneous databases 、 Easy to use , Very easy to use . But I have neglected a problem before . He is still based on JDBC Of , So one parameter is particularly important rewriteBatchedStatements, Set this parameter to true You can perform batch processing with higher performance , Speed up insertion .

Setting mode

Interface settings

If we use the interface configuration, we can use the following settings

Code mode

Properties properties = new Properties();
properties.put("EXTRA_OPTION_MYSQL.serverTimezone","Asia/Shanghai");
properties.put("EXTRA_OPTION_MYSQL.useSSL","true");
properties.setProperty("EXTRA_OPTION_MYSQL.rewriteBatchedStatements","true");
DatabaseMeta targetDataMeta =
        new DatabaseMeta("test", "MySQL", "Native","localhost", "test", "3306", "root", "root");
targetDataMeta.setDBPort("3306");
targetDataMeta.setAttributes(properties);
transMeta.addDatabase(targetDataMeta);

Use the above method , Normal, we can speed it up Kettle Speed of insertion .