Primary master-slave table integration process development process

The master-slave table means that the data of the slave table depends on the master table , Generally, when the data is finally queried, the primary table and the secondary table are associated for query

. Main table ： A data table created in a database , There is a primary key to associate with other tables , And as a unique identifier in the main table . From the table ： A table with the primary key value of the primary table as the foreign key , You can associate query with the main table through the foreign key . From the table and the main table through the foreign key associated query . The main table can be used to store the main information , Used to store extended information from tables .

Using master-slave tables can greatly save system resources , One master table data corresponds to multiple slave table data , If the form of master-slave table is not adopted , There will be a lot of repetition in the main table data , Generally, there are as many master table data as there are slave table data , A great waste of system resources , And this waste can be avoided .

Generate master-slave tables through primary and foreign keys , Using foreign keys as constraints , Realize one to many data form

. This document records the integrated process development process of data synchronization with the master-slave table as the source and target .

▎ Overall introduction

The data synchronization of master-slave tables differs from other data synchronization in that , The data of the master-slave table needs to be obtained together with the data of the master table and the associated slave table during data query , It's not about getting a piece of data alone ; The same is true of the data receiver , When receiving data, the master and slave data are stored together .

1. Overall description

Before the formal introduction of this work , First, introduce the working environment , This work is

stay ESB In the basic sample

,ESB The default generation of basic component sample project is mainly aimed at project implementers and beginners of enterprise service bus , For the convenience of the above personnel ESB Basic functions , The main functions include mapping and conversion of conventional data 、 Data adapter related operations 、 The related operations of the protocol adapter are combined with SMC Management control, application integration, service process monitoring and statistics, etc , At the same time, it is also convenient for project implementers to generate default samples and perform back checking on the use methods of general functions . This work is the data mapping and transformation part of the basic sample .

2. Integration Architecture

The data source tables of this function are account grouping table and account entity table （HR）; User grouping table and user entity table （OA）, These two parts are the source of data , Provide the other party with data to be synchronized , And as the receiving party, it receives the data synchronized by the other party , The integration architecture is shown below ：

Various data sources and target systems are as follows ：

3. functional requirement

This work is

In the original ESB The basic sample is adjusted and optimized based on the process of the application integration part

, Adjust the original data acquisition service and data receiving service to the master-slave table service , At the same time, adjust the output or input parameters corresponding to the service interface to dynamic / Format of static model ; On this basis , Adjust the original integration process , Yes Rest/Web The data obtained from service query is converted into corresponding format 、 Field mapping, etc , Transfer the converted data to the corresponding interface of the target service , Write data to the target database ; At the same time, during data synchronization , Whether it is data acquisition or data reception , All the data processed are in master-slave format .

▎ Demand analysis

The work to be completed this time is divided into two parts , One is

Development of data table related services of simulation business system

; Two is

The adjustment and development of the corresponding data synchronization process

. In this chapter , The demand of this work will be analyzed , Define the objectives of the work 、 Sort out the work ideas .

1. Preparation

After obtaining this requirement , Extract the specific objectives of this work from the requirements ：

Synchronization of master-slave data 、 Dynamic and static models are used to constrain the format of data transmission

. For the above key points , First, write the design documents before development according to the requirements , And confirm with the leader whether the idea is correct , Ensure that the work is carried out in the right direction , Secondly, it completes the design and creation of the data table of the simulation business system , Complete the preliminary preparations for the creation of follow-up services , Finally, according to the designed data table , Create dynamic and static models that constrain data formats ; The completion of these preparations can ensure the smooth implementation of the follow-up work .

2. Work objectives

Analyze according to the requirements in the previous article , It is concluded that the objectives to be achieved in this development work are ：

Synchronization of master-slave data

, That is, when synchronizing the master table data, the corresponding slave table data will be synchronized , Write to the master table and the slave table at the same time . At the same time, in the process of achieving this goal , There are other requirements , For example, it is necessary to use dynamic and static models to control the data format in the process of data transmission , At the same time, due to the different services of the source and target systems , You need to consider JSON And XML Conversion between format data .

3. Realize the idea

This work is carried out

It is necessary to create relevant services for simulating business system data tables 、 The creation of dynamic and static models that constrain data formats and the creation of integrated processes for master-slave format data synchronization

, The implementation idea is as follows ：

1. Create dynamic and static models based on master-slave format data , Dynamic and static models need to create master-slave forms , The main model needs to be list Form save from model ;

2. The service of the user data table is REST service , The data model used is a dynamic model , Get the information from the update interface of the service JSON Format data is directly converted to dynamic model and subsequent data storage operations are completed , In the query interface, the data obtained according to the conditions is converted into a dynamic model and output ;

3. The service of the account data sheet is WEB service , The data model used is static , Configure the input parameters corresponding to the interface in the service 、 Output parameters are static models , Perform corresponding mapping and conversion operations in the interface to complete data processing ;

4. According to the created service , Complete the scenario creation and generate the integration process in the management console , In the integration process, format conversion is performed according to the format of data 、 Field mapping, etc , Realize the synchronous operation of master-slave data .

▎ Implementation steps

Define specific work objectives , At the same time, according to the work objectives, complete the sorting of the realization ideas , You can start developing , In this chapter , The implementation steps of this development work will be described , Specifically divided into models 、 Service and process are described in three parts .

1. Model creation

In this development work

The models used are divided into dynamic models and static models , The steps to create a dynamic model are as follows

：

stay SMC In the service model of the management console service project , Click New , Pictured ：

Fill in the basic information of the model in the model details , At the same time, fill in the model sample with the desired model format JSON data , Pictured ：

The steps to create a static model are as follows

：

stay ESB Right click in the designer SM Model , Click new service model , Pictured ：

Fill in the basic information of the model in the service model wizard , Complete the creation of the model , Pictured ：

Click Select model... In the main model , Select the dependent model , Complete the creation of master-slave model , Pictured ：

2. Service creation

stay ESB In the designer , Create a service based on a data table , Mode select master-slave table mode , Pictured ：

Fill in the basic information of the service , Pictured ：

stay WEB In service , Set the input and output parameters of the relevant interface to the corresponding static model , Pictured ：

The service creation can be completed by completing the data conversion operation in the interface .

3. Integration process

In the admin console API In service , Import the created service , Pictured ：

Input parameters in the corresponding interface 、 Configuration of output parameters , Pictured ：

Create a scenario in the scenario configuration and complete the entry of basic information , Pictured ：

Click resolve... In the mapping parameters , Complete the mapping of parameters , Pictured ：

Create the corresponding integration process in the designer , And complete relevant adjustments , Pictured ：

▎ A functional test

After the development is complete , It is necessary to test the function of the development results , Verify whether the development results meet the initial requirements , Is there any incomprehensible consideration in the development process , In this chapter, the overall verification of the development results will be carried out , Check if there are any omissions .

1. Expected effect

In this function development

The ultimate goal is to realize the synchronous operation of master-slave data

, When the integration process is called, the master-slave data can be synchronized to the target data table ; At the same time, the development function is divided into trigger 、 Push and timing , When invoking the trigger process , Enter grouping CODE Synchronize the group and all entities under the group ; Push is to push the data that needs to be synchronized to JSON The format is synchronized as an input parameter of the integration process ; The timing integration process synchronizes data in batches according to the set timing policy .

2. Trigger process

Take account to user as an example , Enter groups in the administrative console code value , Click to call , Pictured ：

It can be seen from the corresponding output in the figure above , The process was executed successfully , You can also view it in the process log , Pictured ：

In the target database, check whether the group and its corresponding entities are synchronized successfully , Pictured ：

The entity effect is shown in the figure ：

3. Push process

Take account to user as an example , Enter the desired synchronization in the administrative console JSON data , Click to call , Pictured ：

It can be seen from the corresponding output in the figure above , The process was executed successfully , You can also view it in the process log , Pictured ：

In the target database, check whether the group and its corresponding entities are synchronized successfully , Pictured ：

The entity effect is shown in the figure ：

4. Timing process

Configure the timing policy of the timing process in the integration process , This time, the function is tested , Therefore, the timing policy is configured as interval 1 minute , Pictured ：

Start the timing process , Wait a minute , Check whether the process is called in process monitoring , Pictured ：

In the target database, check whether the group and its corresponding entities are synchronized successfully , Pictured ：

The entity effect is shown in the figure ：

▎ Job summary

The development of the integration process of master-slave table data synchronization , I started from getting the initial demand , To the subsequent creation of dynamic and static models 、 Use and adjustment of services and processes , There's a lot to gain from it , Here, I will summarize the gains and experiences in this work .

1. Work gains  

Through this development , I am

For dynamic models 、 The creation and use of static models have been further understood

, Define how to move in the process / The static model controls the format of the data , At the same time, we also have a certain degree of mastery on how to control the format of master-slave data through dynamic and static models ; besides , For data in master-slave format, during synchronization ,

How to operate the master table and the slave table data together

Also have a deeper grasp . In the formal project development work , Data can be normalized through dynamic and static models , It will be converted to the hump naming method for other subsequent operations ; For example, the subsequent work involves the processing of master-slave data , It can also complete the work faster according to the experience gained from this work .

2. Capability improvement

Through this work , about

Data format processing during synchronization

Understand more deeply , about ESB In the designer JAVA Conversion components 、 Mapping components and other data format conversion components are more skilled ; At the same time, due to this

The services used are REST/WEB Two services

, We also have a better understanding of which locations need to be adjusted for the data synchronization between the two services , It can improve the completion efficiency of such follow-up work .

3. summarizing

After the completion of this function development , You can use this example to check the use of dynamic and static models 、 Data processing and... In the process of master-slave data synchronization REST/WEB Between the two services JSON/XML How to convert data in format , So that users who are not familiar with these operations can quickly understand and master ; The project implementation personnel can also use this example to quickly modify the required process .

The functions of this development are different from those of previous development , In the process of this development , More data format conversion than previous synchronization processes , At the same time, more models are used , Check the functions of this development by creating samples , It can enable beginners of enterprise service bus to master the conversion methods of various data formats faster , It can also enable the project implementation personnel to quickly carry out the conversion 、 Mapping components and back checking of dynamic and static model usage methods .