Mysql database

Mysql 5.7

DML

1. replace

   Before inserting a record , It is necessary to confirm whether the value of a field of this record in the table is repeated , If repeat , Need to update other columns , If there is no repetition , Just insert

   replace into t(id,update_time) values(1,now());
   

   Re execution replace into When ,mysql We will first look for the primary key or unique key to determine whether the records conflict , If conflict , Is executed delete+insert operation , If there is no conflict, execute insert operation

   stay mysql In some versions of , When there is conflict , The main library will execute delete+insert operation , and binlog It's recorded in update operation , This leads to the self increment of the slave Library id Not in line with reality . Manual adjustment is required during master-slave switching

DDL

1. The design principle of the table
   • See the name and know the meaning
   • Satisfaction paradigm
   • Reasonable model selection
   • Character set selection utf8,utf8mb64
   • Engine selection innodb
     • Row lock
     • High performance
     • Support transactions
   • Try to avoid long and wide tables , The field does not exceed 50 individual
   • Add comments to tables and fields
   • It is forbidden to store images in the database , Big data such as files , It is suggested to put it in oss The server
   • Naming specification
     • Lowercase letters , And use underline segmentation
     • Temporary storage , Table tmp The prefix , And suffixed with date , for example tmp_table_20220104
   • Stored procedures are not recommended 、 trigger 、 View etc.
2. Business
   • Atomicity A
   • Uniformity C
   • Isolation, I
   • persistence D
3. Transaction isolation level
   • RU Read uncommitted 【 Dirty reading allowed 】
   • RC Read submitted 【 It can't be read repeatedly 】
   • RR Repeatable degrees 【 Fantasy reading 】
   • S Serialization
4. Field design specification
   • Follow less but better
   • Try to choose the smallest data type
   • Appropriate redundancy
   • All fields need to be added not null attribute
   • Use unsigned Store nonnegative values
     • add to unsigned attribute , You can use the same space , Double the data storage range , for example int type (-2147483648,2147483647), And add unsigned after , The value range is (0,4294967295)
   • Use int Store ip value
     • inet_ntoa() and inet_aton Can achieve ip And decimal numbers
   • Use as small as possible varchar Field
   • Don't arbitrarily convert types between fields
   • Fields need to have default values
   • Control index 、 Contains the number of fields
   • No redundancy 、 Duplicate index
   • No indexing null value
   • Foreign keys are not allowed
   • Reasonably use index coverage to avoid io
   • Update and disassemble in large quantities to reduce the particle size
5. sql Design
   • use in Instead of or
   • Implicit type conversions are prohibited — Will invalidate the index
   • Avoid using join And subquery
     • Subqueries and more than 3 Above table join It is very easy to cause slow sql And affect mysql Overall performance of
   • To avoid the mysql The index column of is used for mathematical and functional operations
   • Avoid big business
   • data , It is recommended to obtain data in batches , Less data is obtained each time 500 strip , Result set should be less than 1M
   • use union all Instead of union
     • union Will remove duplicate data , And sort it out
     • union all Will ignore sorting and de duplication , Merge the results directly , Compared with sorting, it saves a lot of computing resources
   • Avoid full table scanning
   • Do not use leading % Go to query
     • A separate slave library can be used for similar query operations
   • No right null Make relevant judgments
   • Splicing sql Pay attention to prevent injection
   • update and delete belt where Conditions

Redis

1. Naming specification
2. Use standard
3. Capacity ,qps forecast
   • A single node qps stay 2w within
4. Structure selection
   • String
     • k|v
   • Hash
   • Set
   • Sorted Set
   • List
5. Use standard
   • Separation of hot and cold data
   • Different business data are stored separately
   • Storage key Be sure to set the timeout
   • The stored large text data must be compressed first and then stored , Do not store pictures , Video etc.
   • on-line redis Do not use dangerous commands
     • hmget
     • hgetall
     • hvals
     • hkeys
     • smembers
     • keys
     • …
   • When using lists as queues , Monitor the queue length

troubleshooting

1. Complete table scanning without cable guidance
2. The use of functions on index columns leads to index invalidation
3. Implicit type conversion of data type leads to index invalidation
4. Use union all Instead of or【 Demolition sql sentence , You can use index 】

Master slave database building 【docker】

1. Pull the mirror image

   docker pull mysql:5.7.37
   

2. docker run Create the main container

   docker run  \
   -p 3307:3306 \
   --name=mysql-master \
   -v /mydata/mysql-master/log:/var/log/mysql \
   -v /mydata/mysql-master/data:/var/lib/mysql \
   -v /mydata/mysql-master/conf:/etc/mysql \
   -e MYSQL_ROOT_PASSWORD=root \
   -d mysql:5.7.37
   

3. docker ps Check operation

4. stay /mydata/mysql-master/conf/ Create under folder my.cnf Profile file

   [mysqld]
   ##  Set up server-id, The same LAN is globally unique 
   server-id=101
   ##  Specify the name of the database that does not need to be synchronized 
   binlog-ignore-db=mysql
   ##  Turn on binary log function 
   log-bin=mall-mysql-bin
   ##  Set the memory size of binary log ( Business )
   binlog_cache_size=1M
   ##  Set the log format using binary （mixed,statement,row）
   binlog-format=mixed
   ##  Binary log expiration cleanup time , The default value is 0, Does not automatically clean up 
   expire_logs_days=7
   ##  If all errors encountered in master-slave replication or specified types of errors , avoid slave End copy interrupt 
   ##  Such as ：1062 An error is a duplicate of some primary keys ,1032 The error is because the master-slave database is inconsistent 
   slave_skip_errors=1062
   

5. After creating the above configuration file , Restart the container instance

6. Into the container , And in maser Create a data synchronization user inside the container instance

   CREATE USER 'slave'@'%' IDENTIFIED BY '123456';
   GRANT REPLICATION SLAVE,REPLICATION CLIENT ON *.* TO 'slave'@'%';
   

7. docker run Create from container

   docker run  \
   -p 3308:3306 \
   --name=mysql-slave \
   -v /mydata/mysql-slave/log:/var/log/mysql \
   -v /mydata/mysql-slave/data:/var/lib/mysql \
   -v /mydata/mysql-slave/conf:/etc/mysql \
   -e MYSQL_ROOT_PASSWORD=root \
   -d mysql:5.7.37
   

8. docker ps Check operation

9. stay /mydata/mysql-slave/conf/ Create under folder my.cnf Profile file

   [mysqld]
   ##  Set up server-id, The same LAN is globally unique 
   server-id=102
   ##  Specify the name of the database that does not need to be synchronized 
   binlog-ignore-db=mysql
   ##  Turn on binary log function , in preparation for slave1 As an instance of other databases master When using 
   log-bin=mall-mysql-slave1-bin
   ##  Set the memory size of binary log ( Business )
   binlog_cache_size=1M
   ##  Set the log format using binary （mixed,statement,row）
   binlog-format=mixed
   ##  Binary log expiration cleanup time , The default value is 0, Does not automatically clean up 
   expire_logs_days=7
   ##  If all errors encountered in master-slave replication or specified types of errors , avoid slave End copy interrupt 
   ##  Such as ：1062 An error is a duplicate of some primary keys ,1032 The error is because the master-slave database is inconsistent 
   slave_skip_errors=1062
   ## relay_log Configure relay logs 
   relay_log=mall-mysql-relay-bin
   ## log_slave_updates Express slave Write the copy event to your own binary log 
   log_slave_updates=1
   ## slave Set to read only （ have super Privileged user ）
   read_only=1
   

10. Restart from container , And enter the container to view

    mysql -uroot -p;
    root
    

11. Execute the following statement ：

    change master to master_host=' host ip',master_user='slave',master_password='123456',master_port='3307',master_log_file='mall-mysql-bin.000001',mysql_log_pos=617,mysql_connect_retry=30;
    

     Parameter description 
    master_host  Master database ip Address 
    master_user  Create a user account for synchronizing data in the master database 
    master_password  Create a user password in the master database for synchronizing data 
    master_port  Main database running port 
    master_log_file  Specify the log file for copying data from the database , By checking the master data of Zhuang Yi , obtain File Parameters 
    mysql_log_pos  Specify where to start copying data from the database , By checking the master data status , obtain position Parameters 
    mysql_connect_retry  The connection failed and the time interval between retries , Company s
    

    binlog Open Settings

win10

1. view the database binlog Whether to open

show variables like 'log_bin';
show variables like 'log_bin%'; 

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-2aqv85da-1644390879899)(C:\Users\admin\AppData\Roaming\Typora\typora-user-images\image-20210616101339651.png)]

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-c4QNpv2M-1644390879901)(C:\Users\admin\AppData\Roaming\Typora\typora-user-images\image-20210616112031907.png)]

2. open binlog journal

   2.1 find mysql Configuration file for mysql.ini【 stay mysql Installation directory 】

   2.2 stay mysqld The configuration is as follows

   # Turn on binlog journal 
   server_id=1
   log-bin=mysql-bin
   binlog-format=ROW
   

   [ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-m7xZ8hsl-1644390879903)(C:\Users\admin\AppData\Roaming\Typora\typora-user-images\image-20210616112246377.png)]

   binlog-format: There are three forms of master-slave synchronization

   statement Will operate on the database sql The statement is written to binlog in

   row Every data change will be written to binlog in .

   mixed statement And row Mixing .Mysql Decide when to write statement Format , When to write row Format binlog

   2.3 Restart the service , Check again

   net stop mysql
   net start mysql
   

   show variables like 'log_bin%'; 
   

   [ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-P9FWI9EW-1644390879903)(C:\Users\admin\AppData\Roaming\Typora\typora-user-images\image-20210616112407603.png)]

linux

1. see binlog state

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-1CVTzz6S-1644390879904)(C:\Users\admin\AppData\Roaming\Typora\typora-user-images\image-20210616113128885.png)]

2. Locate the configuration file and modify it

   1. The location of the configuration file

      /etc/my.cnf
      

   2. Add the configuration

      #  Turn on binary log 
      server-id=1
      log-bin=/usr/local/mysql/data/mysql-bin
      

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-4fD6aWD3-1644390879905)(C:\Users\admin\AppData\Roaming\Typora\typora-user-images\image-20210616113847798.png)]

docker

1. The configuration is mapped to a local file , The local file configuration is as follows

   [mysqld]
   #binlog setting
   log-bin=/var/lib/mysql/mysql-bin
   server-id=123454
   binlog-format=ROW
   

2. restart mysql, Search found binlog It's not turned on

   docker restart mysql
   

3. It took a long time to find this hole on the Internet , Record the process

   First , see docker About my.cnf I found this sentence in my log ：

   mysqld: [Warning] World-writable config file '/etc/mysql/``my.cnf``' is ignored.
   

   Viewing on the Internet is caused by the permission of the configuration file https://blog.csdn.net/qilovehua/article/details/45508925, For details, please refer to this blog post

   Then enter docker Of mysql Container to check , Sure enough my.cnf The permissions for are 777

   [ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-LPOhkoXk-1644390879905)(C:\Users\admin\AppData\Roaming\Typora\typora-user-images\image-20210616142116847.png)]

   Change it to 644

   chmod 644 my.cnf
   

   Restart again mysql Containers , Check it out binlog Successfully opened

   however , If you make any changes to the mapped configuration file later , He will be mapped again to /etc/mysql/my.cnf This file and the permissions are reset to 777, Need to modify again

canal Use research

canal-server

https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz

canal-adapter

https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.adapter-1.1.5.tar.gz

Database synchronization command

/opt/zbox/run/mysql/mysqldump -uroot -pZenTao-Cestc123 --databases zentaoep | /opt/zbox/run/mysql/mysql -h 10.255.237.120 -uroot -pwhut.VINCENT920812



/opt/zbox/run/mysql/mysqldump -uroot -pZenTao-Cestc123 --databases zentaoep | /opt/zbox/run/mysql/mysql -h  10.101.42.163 -uroot -proot

/opt/zbox/run/mysql/mysqldump -uroot -pZenTao-Cestc123 --lock-tables=0 zentaoep > /opt/zbox/mysql_bak_`date +%Y%m%d`.sql