Using loops to process data in tables in kettle

occasionally , If kettle When the data of the source table in a transaction is very large , It is not advisable to read all the data in the source table into memory at once . stay mysql in , We can cycle , Use limit To quantitatively obtain part of the data for processing . namely , pivotal sql yes ：select * from table_name limit current_value, step_value; The following is a demonstration of ideas .
1： Get all the quantities in the record , Initialize the current cycle value, etc ;
2： The judgment condition of the cycle is ： The current cycle value is less than the maximum cycle value
2.1： modify sql The starting value of the query in the statement
2.2： Use a transformation to process the query results , This example is to append the result to the file ;
2.3： Add the current cycle value 1;

The total job flow chart

The first 1 Configuration information of step

The first 2 Configuration information of step

The first 2.1 Configuration information of step
var stepValue = new Number(parent_job.getVariable("STEP_VALUE"));
var i = new Number(parent_job.getVariable("CURRENT_LOOP"))*stepValue;
parent_job.setVariable("CURRENT_CURSOR",i);
true;

The first 2.2 Configuration information of step

The first 2.3 Configuration information of step
var i = new Number(parent_job.getVariable("CURRENT_LOOP"))+1;
parent_job.setVariable("CURRENT_LOOP",i);
true;