MySQL realizes read-write separation

brief introduction

Conventional MySql The read-write separation scheme is based on SQL The type of statement is realized by dynamically switching data sources , So is there any middleware that can automatically realize read-write separation ？ Xiaomi's open source database middleware Gaea Can be realized , Next, we will explain in detail how to use Gaea To achieve MySql Separation of reading and writing .
Gaea It was developed by Xiaomi China e-commerce R & D department based on MySql Protocol database middleware , At present, it is widely used in Xiaomi mall on the mainland and abroad , Include order 、 Community 、 Activities, etc .Gaea Support sub database and sub table 、SQL route 、 Read write separation and other basic characteristics , Among them, the scheme of sub database and sub table is compatible mycat and kingshard The routing of the two items .

Premise

The prerequisite to realize read-write separation is to realize MySql Master slave copy , The specific implementation can be viewed here ：Mysql Master slave copy

install Gaea

At present, the official offer is in Linux Under direct installation

install Go Language environment

because Gaea It's using Go language-written , So we need to install it first Go Language environment .

• Download the installation package first

wget https://dl.google.com/go/go1.13.5.linux-amd64.tar.gz

If you are prompted wegt Command not found , Executable

yum -y install wget

• Download completed after unzip to /mydata Under the table of contents

tar -zxvf go1.13.5.linux-amd64.tar.gz -C /mydata/

• add to /mydata/go/bin Directory to PATH variable

#  Edit environment variable configuration file 
vim /etc/profile
#  Add on last line 
export GOROOT=/mydata/go
export PATH=$PATH:$GOROOT/bin
#  Refresh profile 
source /etc/profile

• View version number , Test for successful installation

go version

• The following information is returned to indicate Go The locale has been successfully installed

go version go1.13.5 linux/amd64

install Gaea

because Gaea Installation package is not provided , So we need to compile the source code to get the executable file

• download Gaea Source code , Direct download zip Bag can
  Download address ：https://github.com/XiaoMi/Gaea
• Decompress the downloaded compressed package , Here we unzip to /mydata/gaea/ Under the table of contents ：

unzip Gaea-master.zip

• Get into /mydata/gaea/ Under the table of contents , Use make Command to compile the source code ：

make build

tip: some Go The dependency of will not be downloaded, resulting in compilation failure , There may be connection failure
At this time, you can execute the command ：

# Switch to a proxy address that can be accessed in China 
go env -w GOPROXY=https://goproxy.cn;
## To configure  GOPROXY  environment variable 
export GOPROXY=https://goproxy.io;

Then go ahead and execute make build The command will compile successfully

• When the compilation is complete /mydata/gaea/bin The directory will generate Gaea The executive documents of gaea
  
• Because we didn't build etcd Configuration center , So you need to modify the local configuration file /mydata/gaea/etc/gaea.ini, Change the configuration type to file

config_type=file

• add to namespace The configuration file , Used to configure our master-slave database information , Profile address ：/mydata/gaea/etc/file/namespace/wdj_namespace_1.json
  
• The contents of the configuration file are as follows

{
    "name": "wdj_namespace_1",
    "online": true,
    "read_only": false,
    "allowed_dbs": {
        "mall": true
    },
    "slow_sql_time": "1000",
    "black_sql": [
        ""
    ],
    "allowed_ip": null,
    "slices": [
        {
            "name": "slice-0",
            "user_name": "root",
            "password": "root",
            "master": "192.168.131.3:3307",
            "slaves": ["192.168.131.3:3308"],
            "statistic_slaves": null,
            "capacity": 12,
            "max_capacity": 24,
            "idle_timeout": 60
        }
    ],
    "shard_rules": null,
    "users": [
        {
            "user_name": "wdj",
            "password": "123456",
            "namespace": "wdj_namespace_1",
            "rw_flag": 2,
            "rw_split": 1,
            "other_property": 0
        }
    ],
    "default_slice": "slice-0",
    "global_sequences": null
}

namespace The configuration file

namespace The configuration format of is json, Include sub table 、 Non sub table 、 Instance and other configuration information , Can be changed at runtime

• Overall configuration description
  
• slice To configure
  
• users To configure
  

stay Docker Running in the container

Pack it up Docker Mirror image

Docker Hub There is no packaged Gaea Mirror image , We need to build a

• Here we use Dockerfile structure Docker Mirror image ,Dockerfile The contents in are as follows

#  This image needs a dependent base image 
FROM golang:latest
#  Set the gaea Copy the source package to docker Container of /go/Gaea-master Under the table of contents , about .tar.gz The files are automatically decompressed 
ADD Gaea-master.tar.gz /go/Gaea-master
#  Move the unzipped source code to /go/gaea Go to the catalogue 
RUN bash -c 'mv /go/Gaea-master/Gaea-master /go/gaea'
#  Get into /go/gaea Catalog 
WORKDIR /go/gaea
RUN bash -c 'go env -w GOPROXY=https://goproxy.cn'
RUN bash -c 'export GOPROXY=https://goproxy.io'
#  take gaea Package and compile the source code 
RUN bash -c 'make build'
#  Declare that the service is running at 13306 port 
EXPOSE 13306
#  Appoint docker Command to execute when container starts 
ENTRYPOINT ["/go/gaea/bin/gaea"]
#  Specify the name of the maintainer 
MAINTAINER wdj

TIP; Before that, we need to Gaea The source code compressed package of is converted to .tar.gz Format
Change orders ：

# take xxx.zip decompression 　　
unzip xxx.zip
# You'll get xxx Folder 
# take xxx Compressed into tar.gz 　　
tar -czf  xxx.tar.gz  xxx
# Got it. xxx.tar.gz

• Then use Docker Command build Gaea Of Docker Mirror image

docker build -t gaea:1.0.2 .

• Build successful console output
  
• Will be installed locally Gaea Copy the configuration file to /mydata/gaea-docker/etc/ Under the table of contents

cp -r /mydata/gaea/etc/ /mydata/gaea-docker/etc/

• Use Docker Command to start Gaea Containers

docker run -p 13306:13306 --name gaea \
-v /mydata/gaea-docker/etc:/go/gaea/etc \
-d gaea:1.0.2

Test read-write separation

Test ideas ： First, we turn off the master-slave copy of the slave instance , And then through Gaea Agent to operate the database , Insert a piece of data , If there is this data in the primary instance but not in the secondary instance , Note that the write operation is the main database . And then through Gaea Agent queries the table data , Without this data , Indicates that the read operation is from the Library , Prove that the read-write separation is successful .

• adopt Navicat Connect to Gaea agent , Note that the account password here is Gaea Of namespace The content of configuration in , Port is Gaea Service port for
  
• adopt Navicat Connect to the master library and the slave library respectively , Used to view data , At this point, the following three database connections are established
  
• adopt stop slave Order to shut down mysql-slave The master-slave copy function of the instance
  
• adopt Gaea The agent is in test Insert a piece of data into the table
  
• View in the main library test The data table , It is found that the data already exists
  
• View from library test The data table , No such data found , Prove that the write operation goes to the main database
  
• View directly in the agent test Table data , No such data found , Prove that the read operation is from the Library
  

combination SpringBoot Use

In our SpringBoot Application , We just have to take Gaea The proxy service of can be directly used as a database service to realize read-write separation .

Test the connection code

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
/**
 * @author wdj
 * Created on 2021/2/5 11:40
 */

public class JdbcTest {
    // attribute 
    private static String driver="com.mysql.jdbc.Driver";
    //JDBC Driver class name 
    private static final String URL="jdbc:mysql://192.168.131.3:13306/mall?serverTimezone=GMT%2B8&useUnicode=true&characterEncoding=utf8&useSSL=false";
    private static final String USER="wdj";
    // user name 
    private static final String PASSWORD="123456";
    // password 
    // Create connection 
    public static Connection getConnection(){
        Connection conn=null;
        try {
            Class.forName(driver);
            // load jdbc drive 
            conn=DriverManager.getConnection(URL, USER, PASSWORD);
// Create connection 
            //System.out.println(" Connection created successfully ！");
        } catch (Exception e) {
            e.printStackTrace();
        }

        return conn;
    }
    // Close the connection 
    public static void close(Connection conn,Statement st,ResultSet rs){
        if(rs!=null){
            try {
                rs.close();
                //System.out.println(" close rs success ！");
            } catch (Exception e) {
                // TODO: handle exception
                e.printStackTrace();
            }
        }
        if(st!=null){
            try {
                st.close();
                //System.out.println(" close st success ！");
            } catch (Exception e) {
                // TODO: handle exception
                e.printStackTrace();
            }
        }
        if(conn!=null){
            try {
                conn.close();
                //System.out.println(" close conn success ！");
            } catch (Exception e) {

                // TODO: handle exception
                e.printStackTrace();
            }
        }
    }


    public static void main(String[] args) {

        try        {
            Connection conn = JdbcTest.getConnection();
            if(conn!=null)
            {
                System.out.println(" The database connection is normal ！");
            }
            else            {
                System.out.println(" Database connection exception ！");
            }
        }
        catch(Exception ex)
        {
            ex.printStackTrace();
        }

    }
}