Automatic upgrading of database structure in rainbow

Rainbond This product has been committed to getting through the whole process of enterprise application delivery , An indispensable part of this process is the continuous upgrading of enterprise applications 、 iteration .Rainbond Unique capabilities , It is an enterprise application system that can package multiple service components , And perform one click installation 、 Upgrade and rollback . The above content only solves the version control problem of the application itself . The upgrade iteration process of enterprise applications wants to be fully automated , You also need to be able to automatically handle the database table structure （Schema） Version control . After constant exploration ,Rainbond First, in the field of source code construction, with the help of industry-leading Liquibase Integrated with the database of the cloud native era Schema Version management capabilities .

Schema Version management challenges

Database table structure （Schema） Define the data table （Table） Name , And the data columns contained in each data table （Column） Name 、 Properties and other information . It describes the framework owned by a database , The data recorded in the database should follow Schema Definition in .

Different from the upgrade of the application itself ,Schema Version management issues , It is essentially an upgrade of persistent data , This feature is accompanied by two questions ：

• How to upgrade persistent data ： Delivery in the cloud native era , Can't jump out of containerization 、 Platform features . Major cloud native platforms are in the process of software delivery , Will not easily incorporate persistent data into the version control system . The reason is simple , The data in each delivery environment is different , During the upgrade process, it is difficult to choose a unified version management scheme for persistent data .
• What persistent data needs to be upgraded ： Since it is difficult to choose a unified version management scheme for persistent data , Then retreat to the next place , Whether the necessary persistent data can be preferentially selected for version management . After narrowing the scope , It highlights the special persistent data type of database table structure . The necessity of version management is obvious , The application itself starts from V1 Version upgraded to V2 edition , Then the corresponding database table structure also needs to add necessary new tables 、 New column .

These two questions lead to the main idea of this article ： In the field of enterprise software delivery , How to reasonably deal with the database table structure in the process of each upgrade （Schema） Version control ？

Traditional software delivery , stay Schema There are two mainstream solutions for version management ：

• Handle by hand ： That's the bottom line Schema Version management . Field delivery personnel not only need to deal with the application upgrade process , Also operate the database directly , complete Schema The upgrade . This method is the most direct , But processes that cannot be automated have some common problems ： Inefficient 、 Fallible .
• Code processing ： This is an advanced way . By introducing third-party libraries inside the application , To carry out Schema Version management . This operation can eliminate the manual process of delivery to the site , The deliverer only needs to update the application , The program itself will connect to the database , Yes Schema Make automated changes . The degree of automation of this method can meet the requirements , But it also has the common problem of introducing third-party libraries ： Technology cost increases 、 Invasive 、 Bound to language or framework .

Solutions in the cloud native era

The age of cloud Nativity , The user of the application 、 Deliverers hope to empower their applications through the platform they choose . In the field discussed in this paper , This expectation can be specifically described as ： With the help of platform capability , In a non intrusive way , take Schema Version management capabilities endow applications , Make the application upgrade with one click , Schema Also automatically complete the upgrade .

Rainbond As a cloud native application management platform , We are also constantly exploring ways to empower applications . stay Schema Version management , It realizes the integration in the process of source code construction Schema Version management capabilities . The application itself does not need to change any code , Just put the two types of files into the specified directory under the code root directory . These two kinds of files are ： The configuration file that defines the connection address of the database instance , upgrade Schema What is used Sql Script files .

About source code construction

Source code construction function , It's a kind of Rainbond Empowering applications . The age of cloud Nativity , Applications are moving towards containerization . The process of containerization seems unavoidable Dockerfile Compiling , But it is not . The source code construction function can directly connect with the source code , Compile it into a working container image . The whole process does not need the intervention of developers , Just provide the address of the code warehouse , It greatly reduces the technical burden of developers .

In the process of source code construction , Many capabilities are integrated in a non intrusive way . For example, by including Pinpoint-agent The way to integrate APM Ability . Another example is through inclusion jmx-exporter The way to integrate custom business monitoring capabilities . Today's key description , By incorporating Liquibase The way , Integrate Schema Version control capability .

About Liquibase

Liquibase Is a special version control for database table structure CI/CD Tools . from 2006 Year begins ,Liquibase The team has been committed to making database change management simpler , Especially in the field of agile software development . This tool is based on Apache 2.0 Open source agreement .

After long-term iteration ,Liquibase Already very mature and reliable , adopt sql、yaml、xml、json Various file formats including , Developers can quickly define compliance Liquibase Style database table structure change file , This kind of document is called changelog. be based on changelog Defined in the ,Liquibase It is very convenient to upgrade and rollback between multiple versions of change operations .

Liquibase Provide a variety of ways for developers to interact , It includes a general command line operation mode , The source code is built through command line integration Liquibase Of Schema Version management capabilities .

Code defined Schema Version control capability

Rainbond Source code construction advocates various abilities of code definition . about Schema Version control capability , It is also defined by the specified file in the code warehouse , We can briefly call it Schema As Code, This practice of code definition capability , Ask every time CI Work starts with a code warehouse address , such as Git. For every database instance , By specifying the configuration file in the directory and changelog To define the database table structure version . By default , It refers to Schema Catalog .

The following is an example of code structure ,Rainbond The official also provided a complete code example java-maven-demo :

.├── Procfile├── README.md├── Schema│   ├── changelog.sql       #  Define database table structure │   └── mysql.properties    #  Define database instance connection information ├── pom.xml└── src

Schema In the catalog mysql.properties and changlog.sql The document defines how Schema version control .

mysql.properties Defines the connection mode of the database instance , And the quoted changelog File address .

driver=com.mysql.cj.jdbc.Driverurl=jdbc:mysql://${MYSQL_HOST}:${MYSQL_PORT}/${MYSQL_DATABASE}?createDatabaseIfNotExist=true&characterEncoding=utf8username=${MYSQL_USER}password=${MYSQL_PASSWORD}changeLogFile=changelog.sql

The simplest definitions include ：

• driver： Designated to use jdbc drive , Driver support integrated in source code construction mysql、mariadb、mssql、mongo、postgresql、sqlite And other common types of databases .
• url： Define the database connection address , Can pass jdbc Standard writing method to pre create database instances .
• username&password： Define login credentials for database instances .
• changeLogFile： Define the path of the database instance table structure change file .

In the process of source code construction , Will traverse the identification Schema All under directory properties file , And handle the... Of each database instance at startup Schema Version control process . Through the combination of configuration files , It can work well in the following common scenarios .

• Single database instance
• Multiple database instances of the same type , For example, the application connects multiple at the same time mysql
• Multiple database instances of different types , For example, the application is connected at the same time mysql、mongo
• Multiple database instances in the same database , For example, applications use the same mysql Multiple library instances in

changlog Best practices

changelog file , Is management Schema The key . Here is an example ：

-- liquibase formatted sql-- changeset guox.goodrain:1create table person (id int primary key,name varchar(50) not null,address1 varchar(50),address2 varchar(50),city varchar(30));-- rollback drop table person;-- changeset guox.goodrain:2create table company (id int primary key,name varchar(50) not null,address1 varchar(50),address2 varchar(50),city varchar(30));-- rollback drop table company;

Recommended sql Type of changelog File to define Schema edition , Because this is most in line with the habits of developers .

changlog The file defines some behaviors through annotations . As follows ：

#  Definition  changelog  File format , This is every one  changelog  The beginning of the file -- liquibase formatted sql #  Define changesets , What follows is , Is the name of the developer , And the serial number of the change set , This serial number is very important , It is recommended to use ordered numbers to define -- changeset guox.goodrain:1#  Define rollback operation , Each change set should define the corresponding rollback operation , This makes when changes go wrong , Quickly rollback to the specified version of the changeset -- rollback drop table staff;

Liquibase Officials have put forward a series of best practices , There are some best practices that should be the default behavior of developers .

• Each change set contains only one change , By refining the changed version of the database table structure , This can prevent failed auto commit statements from leaving the database in an unexpected state .
• changeset Of ID, Choose an orderly and unique sequence , Or a developer friendly name .
• Make the version always rollback , For every one changeset Set reasonable rollback operation .

About mysql.properties and changlog.sql The way a document is written , For more features, please refer to liquibase file , These features can be inherited by source code construction .

Schema Life cycle process

1. Construction process

When executing the normal source code construction process , It will automatically identify the Schema Catalog , Get ready Schema The basic environment required for version management , Include jre and Liquibase tool kit .

The following prompts will appear in the build log ：

2. Start process

After completing the build process , The service component will automatically enter the startup process , Rainbond The platform will be based on the configuration file defined in the code , For each database instance , Perform automatic upgrade .

In the process , The header position in the log of the service component , Will print relevant records ：

The above figure demonstrates the same mysql Upgrade the table structure of multiple database instances . For empty library instances , This is also equivalent to an initialization operation .

In the example ,Rainbond Connect to the same application respectively mysql Two library instances in the database （ Respectively called Initializeanotherdb） Initializing the table structure , Create tables respectively company 、person as well as another_company 、another_person. In the database component Web After the terminal logs in , Can verify ：

3. Publish to component library

Rainbond Unique publishing mechanism , Business components and database components can be published as an application template . It is convenient for one click installation and delivery in different environments . Applications delivered through application templates , Still have Schema Version control capability . Newly installed application template , Its database will also be initialized to the above state . ad locum , We call published applications source applications , Applications installed from the application template are delivered applications .

4. Code update

When developers continue to iterate business systems ,Schema It also changes , Suppose a new version of the business system , requirement Initialize The new table staff, And for the existing person Add a new column to the table country. Then the developer should be the corresponding changelog.sql The following contents are added to the document , And submit it together with the new business code , Ensure business code and Schema bring into correspondence with .

-- changeset other.goodrain:3alter table person add column country varchar(2);create table staff (id int primary key,name varchar(50) not null,address1 varchar(50),address2 varchar(50),city varchar(30));-- rollback drop table staff;-- rollback alter table person drop column country;

Click build at the source application ,Rainbond Will pull the latest code , While updating business applications , by Schema upgrade .

There is no change in the construction process , But during startup , For updated Initialize And remain the same anotherdb Library instance ,Rainbond Give two different treatments ：

5. Upgrade based on application template

The source application has a new version , The delivered application should also be changed . First , The application template needs an updated version , Repeat the publishing process , Define a higher version number . Delivered applications can be based on Rainbond A hint of , One click upgrade to the updated version .

6. verification

Log in to the database component of the delivered Application , You can view the corresponding Schema change .

7. Roll back

Rollback operation of database table structure is a very serious problem . Based on the principle that the database table structure only increases but not decreases , Already effective Schema There will be no change with the one click rollback of the delivered Application . If rollback is necessary , Then the operation and maintenance personnel need to log in to the business component Web Manual operation of terminal .

Note the rollback sequence ： The database table structure should be rolled back before the application . This is because once the application rollback is completed , changlog The file itself has also been rolled back to the previous version , Unable to rollback the database table structure .

Execute the following command , According to the specified configuration file , Rollback the database table structure , The rollback range is in 1 individual changeset In units of .

cd Schema/liquibase rollbackCount 1 --defaults-file=mysql.properties

Once the business component after rollback is restarted or updated , Will compare changelog Upgrade the file again Schema, So after the rollback operation , Be sure to add environment variables ALLOW_SCHEMA_UPDATE=false To disable Schema Version management control function , Until the new version of the application template is upgraded .

common problem

1. How to be in *.properties Reasonably define the connection address and credentials of all database instances in the configuration file ？

Use environment variables instead of *.properties Data path instance connection address and credential information in the configuration file , See the example in the text for the definition method .Rainbond In the process of source code construction , All environment variables in the running environment will be picked up , Render the target configuration file , So the naming of environment variables is not important , Just ensure that the defined environment variables will be generated in the final delivery environment . Whether the environment variable comes from the customized environment configuration or Rainbond Unique connection information mechanism .

2. Rollback operation failed ？

How to rollback , It's defined in changlog In file . Be sure to ensure that every changeset There are corresponding rollback strategies , To ensure that each rollback will get the correct result .

3. perform Schema An error is reported during the upgrade ：!! Failed to check the database status. Check /app/Schema/xxx.properties.log

Every execution Schema In the process of change , Will be checked first , Including the connectivity of the database instance address 、changelog Enforceability of documents . If the inspection fails , No operation will be done on the database , But the results of the inspection will be recorded in the log file , You can log in. Web terminal , Check the log file contents in the prompt .

4. How do regular users get Schema Version control ？

This function is similar to Rainbond The version of is out of , So old users can get this ability by updating the source code and building related components . Execute the following set of commands ：

#  The following order is in  Rainbond  Any node in the cluster executes ; If you use  dind-allinone  edition , Should be in  rainbond-allinone  Container execution hubpassword=$(kubectl get rainbondcluster -o yaml -n rbd-system | grep password | awk '{print $2}')docker login --username=admin --password=${hubpassword} goodrain.meimages=(builder runner)for image in ${images[@]}  do    docker pull registry.cn-hangzhou.aliyuncs.com/goodrain/${image}:v5.5.0-release    docker tag registry.cn-hangzhou.aliyuncs.com/goodrain/${image}:v5.5.0-release goodrain.me/${image}    docker push goodrain.me/${image}  done

References Link

Liquibasehttps://www.liquibase.com

java-maven-demohttps://gitee.com/rainbond/java-maven-demo

About Rainbond

Rainbond It is an open source cloud native application management platform , Easy to use , You don't need to understand containers and Kubernetes, Support management of multiple Kubernetes colony , Provide full lifecycle management for enterprise applications , Functions include application development environment 、 Application market 、 Microservice architecture 、 Application continuous delivery 、 Application operation and maintenance 、 Application level multi cloud management, etc .

Github：https://github.com/goodrain/rainbond Official website ：https://www.rainbond.com Wechat group ： Focus on Rainbond The official account is joined by the technical exchange group. Nail group ： Please search for nail group number 31096419