Kettle reads file split by line

Kettle It's an open source ETL Tools , Support data migration from many data sources , So is the author N It was used years ago , Recently, a batch of people from HDFS Text data of , I also thought about the use of this tool , Come on HDFS Text is often stored in rows , What the author encountered this time is 6 Thousands of lines json data , Total volume in 20G, After looking around the Internet, I couldn't find the operation of reading multiple lines of text , So I fiddled with it myself , For the record ：

The data format is as follows ：

{"type": "Feature", "properties": {},"geometry": {"type": "Polygon","coordinates": [[[-19.441364292429924, 65.52333266640196], [-19.441369748908993, 65.52335128397081], [-19.441456975615232, 65.52334689547278], [-19.441451519136184, 65.5233282779008], [-19.441364292429924, 65.52333266640196]]]}}
{"type": "Feature", "properties": {},"geometry": {"type": "Polygon","coordinates": [[[-22.023537553224426, 64.34071231744036], [-22.023766291888357, 64.34079685621145], [-22.023896688463573, 64.34073070136526], [-22.02366796389063, 64.340646160667], [-22.023537553224426, 64.34071231744036]]]}}
{"type": "Feature", "properties": {},"geometry": {"type": "Polygon","coordinates": [[[-20.53351958043648, 64.21502080919846], [-20.533556158627654, 64.21504083748084], [-20.533638377500047, 64.21501242439349], [-20.53360179930887, 64.21499239609055], [-20.53351958043648, 64.21502080919846]]]}}

Divided by rows JSON, You need to import it into postgresql in ：

stay kettle New China “ transformation ”, Select text file input in input , Add questions that need to be extracted :

stay “ Content ” Change the format of the management card to Unix, stay “ Field ” The management card can define the field name by clicking get field ：

 

 

Then you can preview the record ：

 

In order to read JSON One of the Key The next value , We can add one later “JSON Input ”, stay “ file ” Manage card selection “ The source is defined in a field ”,“ Get source from field ” Select the field output in the previous step :

 

 

stay “ Field ” For management card JSONPath Configuration fields ：

 

Next, you can preview the process by previewing the transformation ：

 

 

Then you can configure the output in the way of warehousing .

Finish off work , Test it out ,6 thousand 5 Millions of text data to postgresql when 37 minute , The execution efficiency should be very fast .