Flyway management database

zero 、 In traditional DBA Upgrade the production database , It is troublesome to have multiple sets of environment maintenance 、 Cumbersome operation 、 Rollback complex , Can not be effectively used in practice Git Unified management of online scripts . Under many unfavorable conditions , Should be DevOps The requirements of , To visualize the maintenance and upgrade of database scripts 、 simple . Our team used flyway To achieve this effect . Organize some operations , For future reference .

flyway brief introduction ：Flyway Is a database migration （migration） Tools . It helps us synchronize databases in different environments , Reduce manual operation , Avoid the wrong order of data import , It also reduces the chance of omission . Think of it as git, Many people operate code at the same time

Supported databases ：Oracle, SQL Server, SQL Azure, DB2, DB2 z/OS, MySQL (including Amazon RDS), MariaDB, Google Cloud SQL, PostgreSQL (including Amazon RDS and Heroku), Redshift, Vertica, H2, Hsql, Derby, SQLite, SAP HANA, solidDB, Sybase ASE and Phoenix.

Depending on the conditions ： Need to rely on Git、 With the database ( this paper Mysql)

One 、 install flyway cli client  flyway-commandline-7.0.0

flyway Official website ：Documentation - Flyway by Redgate • Database Migrations Made Easy.https://flywaydb.org/documentation/flyway-cli Download address ：

Command-line - Command-line tool - Flyway by Redgate • Database Migrations Made Easy.https://flywaydb.org/documentation/usage/commandline/ After downloading and decompressing , take  flyway-7.0.0 Add folder to computer environment variable

Two 、 Apply for a unified secret key Key

To apply for the address , Every application is 30 God , You need to use a different email address each time ：Teams Edition Free trial - FlywayStart your 28-day free trial of Flyway Teams Edition.https://flywaydb.org/download/teams

3、 ... and 、 Common commands

# Initialize run
flyway -configFiles=./conf/flyway-local.conf migrate
# Specify baseline
flyway -configFiles=./conf/flyway-local.conf baseline -baselineVersion=2
# Undo the last action
flyway -configFiles=./conf/flyway-local.conf undo
# Clear all data and table structures
flyway -configFiles=./conf/flyway-local.conf clean

encryption ：sh Carry out orders  gpg --symmetric flyway-dev.conf # encryption

Be careful , The above order , All want to by git Under the tool Git Bash Here To use . stay windows Under the system cmd The command will report an error . stay Mac No error will be reported in the environment .

Four 、 Simple and practical

You only need two folders

--conf/

        --flyway-dev.conf

        --flyway-dev.conf.gpg

        --flyway-qa.conf

--sql

        --U1__uninit.sql

        --U2__drop_TAB123 .sql

        --V1__init.sql

        --V2__create_TAB123 .sql

The configuration file is flyway-dev.conf yes ：

flyway.driver=com.mysql.cj.jdbc.Driver
flyway.url=jdbc:mysql://1.2.3.4:3306/testFlyway?useUnicode=true&characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&allowMultiQueries=true&allowPublicKeyRetrieval=true
flyway.user=root
flyway.password=root
flyway.locations=filesystem:sql
flyway.licenseKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
flyway.baselineVersion=1
flyway.baselineOnMigrate=true

Note that the last two parameters are to set the baseline , Used in the old library

flyway.baselineVersion=1
flyway.baselineOnMigrate=true
This file flyway-dev.conf.gpg Yes, it is sh Encryption generated , The order is  gpg --symmetric flyway-dev.conf # Define encryption

sql The documents are as follows ：

V1__init.sql Initialize forward script , It usually defines some tables that already exist in the database . Set the baseline to 1, Naturally, this script will not be executed , Of course, if conditions permit , Yes, you can back up the entire library , Used on the new empty library , You can generate a copy of the same library , Of course, if you don't use this method, you can .

U1__uninit.sql Initialize reverse script , A forward script is a reverse script

V2__create_TAB123 .sql Forward script to execute ：

CREATE TABLE T_ABC (
	id varchar(32) NOT NULL COMMENT ' Primary key   ',
	name varchar(100) NULL COMMENT ' full name ',
	CONSTRAINT T_ABC_PK PRIMARY KEY (id)
)
ENGINE=InnoDB
DEFAULT CHARSET=utf8mb4
COLLATE=utf8mb4_general_ci
COMMENT='devops Data pipeline test table ';

insert into T_ABC values(1,' success 1');
insert into T_ABC values(2,' success 2');
insert into T_ABC values(3,' success 3');
insert into T_ABC values(4,' Failure 1');
insert into T_ABC values(5,' Failure 2');
insert into T_ABC values(6,' success ');

U2__drop_TAB123 .sql Reverse script to be executed

drop table T_ABC;

5、 ... and 、 Simple and practical script execution scheme

1、 For new libraries , Blank library execution

# Initialize run 
flyway -configFiles=./conf/flyway-local.conf migrate

follow-up , If you submit a new script , Also run this command , You can automatically execute subsequent scripts

After execution, the mysql Generate a flyway Operation corresponding record table  flyway_schema_history

 2、 For existing libraries , The baseline needs to be specified before the first execution

# Specify baseline 
flyway -configFiles=./conf/flyway-local.conf baseline -baselineVersion=2
# Initialize run 
flyway -configFiles=./conf/flyway-local.conf migrate

such V1 Version will not be executed , Only from the V2 Version starts execution .

3、 Undo the last action

# Undo the last action
flyway -configFiles=./conf/flyway-local.conf undo

You can undo the operation , At the same time in the database flyway_schema_history Delete the corresponding operation record in

4、 Clear all data and table structures ---- Use with caution ！！！

# Clear all data and table structures
flyway -configFiles=./conf/flyway-local.conf clean

All data and tables in the current database will be deleted , Delete library .

6、 ... and 、 General rules of use

1、 Branch management strategy ：

2、 Database change script naming rules

V{ Version number }__{ Change No }_{ operation }_{ Operational objectives }.sql

namely ：1、 Capitalization V start , Followed by the version number

        2、 Version number ：

        3、 Two underscores

         4、 Change No ： demand ID, Associated with Tianji

         5、 operation ：create insert update alter drop etc.

         6、 Operational objectives ： surface 、 Field 、 View 、 trigger etc.

3、 Promotion strategy

Promotion objective ：DEV->QA->UAT->PROD

Promotion plan ： Test environment just -> back -> just

   Production environment just

Because the data script fallback is complex , Need to be in DEV The successful implementation will promote the strategy to the next step , Because the successful and failed files will be submitted with Hash Forms exist in the database , It is not allowed to delete successful scripts submitted by others .

When a script fails recently , The pipeline will fail , And will not run the following script . Only the corresponding record of the currently failed script can be modified . It is not allowed to make changes on the previous file .

Handling of common problems ：1、 just ( Failure ) back ( success ) just ( Failure )

Positive script failed , None of the following scripts will run .

DDL Automatic submission ,DML Will withdraw

2、 just ( success ) back ( Failure ) just ( success )

Operation plan ：① Delete flyway_schema_history In the table , Just executed record .

② If there is ddl Generated tables , Delete the table by yourself

③ Resubmit the executed script

7、 ... and 、 Advanced operation

Script forward operation

dev file ：first.sh

SECRET=AKAJSHDKASHDKLAS
echo  ${SECRET}
echo '--------------'
gpg -d --batch --passphrase ${SECRET} ./conf/flyway-dev.conf.gpg > ./conf/flyway-dev.conf

flyway -configFiles=./conf/flyway-dev.conf info
flyway -configFiles=./conf/flyway-dev.conf migrate -X || { echo 'flyway command failed' ; exit 1; }
flyway -configFiles=./conf/flyway-dev.conf info

rm -rf ./conf/flyway-dev.conf

Reverse script ：last.sh

#!/bin/bash
SECRET=DAKJBFAKASD
echo  ${SECRET}
echo '--------------'
gpg -d --batch --passphrase ${SECRET} ./conf/flyway-dev.conf.gpg > ./conf/flyway-dev.conf

flyway -configFiles=./conf/flyway-dev.conf info
flyway -configFiles=./conf/flyway-dev.conf undo -teams || { echo 'flyway command failed' ; exit 1; }
flyway -configFiles=./conf/flyway-dev.conf info

rm -rf ./conf/flyway-dev.conf

Running scripts in sequence can complete the unified management of database scripts , adopt Git Manage relevant documents , You can manage the database according to the rules , It can also facilitate rollback