Kettle implements shared database connection and insert update component instances

Common components

1、 Shared database connection

The newly created database connection only belongs to a certain conversion program

If you want all conversion programs to use this connection , Sharing needs to be enabled

2、 Table input component

• demand ∶ Put the data in the table , Sync to another table .

• analysis

  • This is a transformation task
  • Input ︰ Read the data of the table
  • transformation : There is no conversion process
  • Output : Write the results into the table

• Realization

  • Development procedure

    

  • Configuration input

    First configure the database connection sharing

    

    

    

  • Configure output

    

    

    

  • test run

    

    

3、 Insert update component

• Job requirements ∶ take A The data of the table is synchronized to B In the table , Guarantee B Table data and A The data in the table is consistent , Implementation is an operation that is constantly updated

  • A The table has been updated , The updated data will also be synchronized to B In the table
  • A The table has not been updated , Even if the program runs ,B The table does not change
  • The process of data synchronization
    • Only the updated data is synchronized each time
    • Data that has been synchronized , No more synchronization
  • At work, you usually synchronize once a day , The program is executed once a day

• solve ∶ Insert updated output components

• function : Only the updated data will be synchronized , Data that has been synchronized will not be synchronized again

  • Data update
    • Insert a new piece of data
    • Modify an old piece of data

• Realization : anytime , The data synchronization of the two tables is consistent

  • Development and transformation task flow chart

    

  • Define input

    Consistent with the previous steps

    

  • Define insert / to update

    

    

    Only in id To make a comparison

    

    

  • test run

    

    Modify the data in the original table or add , Execute it again , Automatically update

summary

This paper mainly focuses on two knowledge points ：

1. Shared database connection
2. Insert update component

• Synchronization service
  • Total quantity ︰ Synchronize all data one at a time
    • Guarantee A and B It's consistent
      • Delete each time B All contents , then , Resynchronization
    • The performance of the program is poor , When the amount of data is large , Very slow , Not recommended
    • Table output ︰ Full amount of components
  • The incremental ︰ Every time an updated data synchronization occurs , If there is no update, the synchronized data is no longer synchronized
    • Guarantee A and B It's consistent
    • Incremental methods are used in work
    • Insert update ︰ Incremental components

The above two knowledge points are very suitable and suitable for working scenes , It can also be matched with homework , Yi Yi Yong Lao