当前位置：网站首页>Gbase 8A MPP cluster capacity expansion practice

Gbase 8A MPP cluster capacity expansion practice

2022-07-27 09:51:00 【51CTO】

The authors introduce

Wang Hao , New torch network core business system DBA, Mainly involves Oracle、Greenplum、Gbase Wait for database management 、IT Operation and maintenance management , Have rich practical experience in database multi business scenario performance optimization , Focus on database performance optimization 、IT Operation and maintenance automation .

One 、 background

Due to the development of business model and the need of data cycle retention , Recently a provincial operator plans to GBase 8a Cluster expansion , From existing 3coor+21data Expand the node to 3coor+61data node .

At present GBase 8a The cluster version is GBase8a_MPP_Cluster-NoLicense-8.6.2_build33-R12-redhat7.3-x86_64. Newly added 40 Nodes are only data nodes . I have the honor to participate in , Responsible for specific capacity expansion and subsequent data redistribution operations .

Two 、 environmental information

1、 Hardware configuration information

CPU：

CPU Count ：4*8C (4 Physics cpu, Every physics cpu Yes 8 A logic cpu)

Memory ：

MemTotal: 512GB

2、 Software version

GBase 8a Cluster version GBase8a_MPP_Cluster-NoLicense-8.6.2_build33-R12-redhat7.3-x86_64

3、 Expansion machine planning

To ensure the application access interface ip The address group does not change , After expansion , Still retain 3 individual coordinator（ The management node ） The nodes remain the same , Capacity expansion 40 All nodes are data（ data ） node . The planning host name is gbase25-gbase64.

3、 ... and 、 Preparation before implementation

1、 Network environment requirements for capacity expansion implementation

The network environment of the site is the original cluster 24 Servers and new expansion 40 The servers are all in the intranet , The Internet is gigabytes , Double network card binding , The network test results meet the expansion requirements .

2、 Storage space requirements for expansion implementation

In order to ensure the absolute safety of expansion implementation , Each server should have enough space for redistributed temporary data . Every node of the existing nodes in the cluster opt The directory has free space 13TB, Root free space 439GB; New node opt There is free space 22TB, Root free space 149GB, Meet the expansion requirements .

The inspection found that there were two servers （IP The address is 190、193） The disk write speed is obviously abnormal , The host personnel confirm that it is RAID Card battery failure , After the repair, the disk read and write speed is normal .

GBase 8a MPP Cluster expansion practice _ database

3、 Server requirements for expansion implementation

Unified MPP The operating system version of the cluster node . Before the expansion, the new expansion node operating system has been unified and re integrated , It is consistent with the existing node operating system version of the cluster , by rhel7.3, Meet the expansion requirements .

Four 、 Expansion implementation

1、 Add new node root And gbase Users trust each other

       
       -- root user 
       
scp -r ~/.ssh 192.168.200.12:~/
       
-- gbase user 
       
scp -r ~/.ssh 192.168.200.12:~/
      
1.
2.
3.
4.

2、 To configure C3 Tools ( The tool is used for GBASE Each node executes the execution command at the same time )

       
       -- root user 
       
vi /etc/c3.conf  add to 
       
cluster new {
       
     192.168.200.11:redhat1
       
     192.168.200.12
       
}
      
1.
2.
3.
4.
5.
6.

3、 Use C3 The tool configures the expansion node environment

       
       --  operating system 
       
cexec new_data: new: 'cat /etc/redhat-release'
       
--  kernel 
       
cexec new_data: new: 'uname -a'
       
--  A firewall 
       
cexec new_data: new: 'service iptables status'
       
cexec new_data: new: 'service ip6tables status'
       
cexec new_data: new: 'chkconfig | grep iptables'
       
cexec new_data: new: 'chkconfig | grep ip6tables'
       
-- selinux
       
cexec new_data: new: 'sestatus'
       
cexec new_data: new: 'grep ^SELINUX= /etc/selinux/config'
       
--  Memory parameters 
       
cexec new_data: new: 'ulimit -H'
       
cexec new_data: new: 'ulimit -S'
       
cexec new_data: new: 'ulimit -m'
       
-- vi /etc/security/limits.config  add to 
       
-- * soft as unlimited
       
-- * hard as unlimited
       
-- * rss as unlimited
       
--  Transparent big page 
       
cexec new_data: new: 'cat /sys/kernel/mm/redhat_transparent_hugepage/enabled'
       
-- echo "never" > /sys/kernel/mm/redhat_transparent_hugepage/enabled
       
--  Host name check 
       
cexec new_data: new: 'hostname'
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.

4、 Set up the cluster readonly Then back up the cluster information

       
       --  modify /etc/hosts
       
vi /etc/hosts  Add new node information and synchronize to new node 
       
--  Confirm that the cluster is normal 
       
gcadmin
       
gcadmin showddlevent
       
gcadmin showdmlevent
       
gcadmin showdmlstorageevent
       
--  Set up the cluster readonly
       
gcadmin switchmode readonly
       
-- scn tableid  Backup 
       
cexec 'python -c "import gcware;print gcware.getscn()"'
       
cexec 'python -c "import gcware;print gcware. gettableid ()"'
       
--  View version information 
       
cexec "/opt/gcluster/server/bin/gclusterd -V"
       
cexec "gcadmin -V"
       
cexec data: "/opt/gnode/server/bin/gbased -V"
       
cexec 'gccli -ugbase -pgbase20110531 -Nse "select @@version"'
       
--  Database information backup 
       
sh backup_database.sh
       
ls -l /home/gbase/gbase_expand/201811
       
-- nodedatamap Backup 
       
gccli -ugbase -pgbase20110531 -vvv -e"rmt:select * from gbase.nodedatamap  into outfile '/home/gbase/gbase_expand/201811/nodedatamap.dat' fields terminated by '|'"
       
wc -l /home/gbase/gbase_expand/201811/nodedatamap.dat
       
--  Back up the cluster configuration file 
       
cexec "mkdir -p /home/gbase/gbase_expand/201811/gcluster"
       
cexec "cp -r /opt/gcluster/config/ /home/gbase/gbase_expand/201811/gcluster/"
       
cexec "ls /home/gbase/gbase_expand/201811/gcluster/config"
       
cexec data: "mkdir -p /home/gbase/gbase_expand/201811/gnode"
       
cexec data: "cp -r /opt/gnode/config/ /home/gbase/gbase_expand/201811/gnode/"
       
cexec coor: "ls /home/gbase/gbase_expand/201811/gnode/config"
       
--  Backup corosync The configuration file 
       
cexec "cp -r /etc/corosync /home/gbase/gbase_expand/201811/"
       
cexec "ls /home/gbase/gbase_expand/201811/corosync | wc -l"
       
--  Backup gcware The configuration file 
       
cexec "cp -r /var/lib/gcware /home/gbase/gbase_expand/201811/"
       
cexec 'ls /home/gbase/gbase_expand/201811/gcware | wc -l'
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.

5、 Perform the expansion

       
       --  Stop the cluster 
       
cexec "service gcware stop"
      
1.
2.

       
       -- Find the directory where the original upgrade package is located 
       
--  modify demo.options
       
cd gcinstall/
       
vi demo.options
       
installPrefix= /opt
       
coordinateHost = 
       
dataHost = 134.32.48.8,134.32.48.11,134.32.48.13,134.32.48.14,134.32.48.46,134.32.48.47,134.32.48.48,134.32.48.50
       
existCoordinateHost =
       
134.32.48.208,134.32.48.209,134.32.48.210,134.32.48.211,134.32.48.212,134.32.48.213,134.32.48.214,134.32.48.215,134.32.48.216,134.32.48.217,134.32.48.218,134.32.48.219,134.32.48.220,134.32.48.221,134.32.48.222,134.32.48.223,134.32.48.224,134.32.48.225,134.32.48.226,134.32.48.227
       
existDataHost =134.32.48.208,134.32.48.209,134.32.48.210,134.32.48.211,134.32.48.212,134.32.48.213,134.32.48.214,134.32.48.215,134.32.48.216,134.32.48.217,134.32.48.218,134.32.48.219,134.32.48.220,134.32.48.221,134.32.48.222,134.32.48.223,134.32.48.224,134.32.48.225,134.32.48.226,134.32.48.227
       
loginUser= root
       
loginUserPwd = ' Huawei#123'
       
#loginUserPwdFile = loginUserPwd.json
       
dbaUser = gbase
       
dbaGroup = gbase
       
dbaPwd = gbase
       
rootPwd = ' Huawei#123'
       
#rootPwdFile = rootPwd.json
       
dbRootPwd = '[email protected]'
       
#mcastAddr = 226.94.1.39
       
mcastPort = 5493
       
--  Perform the expansion ​
       
./gcinstall.py --silent=demo.options
       
--  Profile comparison 
       
diff /opt/gcluster/config/gbase_8a_gcluster.cnf /home/gbase/gbase_expand/201811/gcluster/config/gbase_8a_gcluster.cnf 
       
diff /opt/gnode/config/gbase_8a_gbase.cnf /home/gbase/gbase_expand/201811/gnode/config/gbase_8a_gbase.cnf
       
cexec data: md5sum /opt/gnode/config/gbase_8a_gbase.cnf
       
--  Generate a new distribution（ Backup ）
       
gcadmin distribution gcChangeInfo.xml p 1 d 1
       
--  Generate a new hashmap
       
gccli -ugbase -pgbase20110531 -vvv -e"initnodedatamap"
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.

6、 Basic verification of cluster availability after capacity expansion

       
        Add, delete, check and test 
       
create database db_test;
       
create table db_test.t1(c1 int,c2 int) distributed by ('c1');
       
insert into db_test.t1 values (1,1),(2,2),(3,3);
       
update db_test.t1 set c2=10 where c1=1;
       
select * from db_test.t1;
       
delete from db_test.t1 where c1>=3;
       
select * from db_test.t1;
       
truncate table db_test.t1;
       
 Data loading test 
       
load data infile 'sftp://gbase:[email protected]/tmp/t1.txt' into table db_test.t1 fields terminated by ':';
       
select count(1) from db_test.t1;
       
drop table db_test.t1;
       
drop database db_test;
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

5、 ... and 、 Data redistribution

be-all MPP Clusters are distributed in many data nodes , So after the expansion operation is completed , To avoid data skew , All business table data needs to be redistributed to all data nodes （ Including expansion nodes ）.

       
       --  Redistribute 
       
--  Set redistribution concurrency to 0​
       
gccli -ugbase -pgbase20110531 -vvv -e"set global gcluster_rebalancing_concurrent_count=0"
       
gccli -ugbase -pgbase20110531 -Ns -e"select @@gcluster_rebalancing_concurrent_count"
       
--  Redistribute the entire instance 
       
gccli -ugbase -pgbase20110531 -vvv -e"rebalance instance"
       
gccli -ugbase -pgbase20110531 -Ns -e"select count(1) from gclusterdb.rebalancing_status"
       
--  Adjust priorities 
       
create table test.reb_tab(db_name varchar(64),table_name varchar(64),priority int) replicated;
       
--  Insert a table of high priority 
       
insert into test.reb_tab values ('test','t1',1),('test','t2',2);
       
update gclusterdb.rebalancing_status a, test.reb_tab b set a.priority=b.priority where a.db_name=b.db_name and a.table_name=b.table_name ;
       
select count(1) from gclusterdb.rebalancing_status where priority<5; 
       
--  Adjust redistribution concurrency 
       
gccli -ugbase -pgbase20110531 -vvv -e"set global gcluster_rebalancing_concurrent_count=1"
       
gccli -ugbase -pgbase20110531 -Ns -e"select @@gcluster_rebalancing_concurrent_count"
       
--  Pause redistribution 
       
gccli -ugbase -pgbase20110531 -vvv -e"pause rebalance instance"
       
gccli -ugbase -pgbase20110531 -Ns -e"select status,count(1) from gclusterdb.rebalancing_status group by 1"
       
--  Continue to redistribute 
       
gccli -ugbase -pgbase20110531 -vvv -e"continue rebalance instance"
       
gccli -ugbase -pgbase20110531 -Ns -e"select status,count(1) from gclusterdb.rebalancing_status group by 1"
       
--  Wait for redistribution to complete 
       
--  Resume business 
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.

6、 ... and 、 Efficiency analysis

The time consumption of each step in the expansion ：

Capacity expansion ：24 Japan 18:30 ~ 24 Japan 20:20, It takes about 2 Hours ;
Redistribute ： altogether 8802 A watch ,231T The amount of data ,24 Japan 20.25 ~ 26 Japan 10.36, It takes about 38 Hours , The original plan 91 Hours （ With engineering experience 35MB/s Speed calculation of ）.

notes ： Because there is a very uneven table , All data falls on one node ,70 A field ,75 Billion records ,13 Compress , Single slice 350GB. This table alone is used for redistribution 12 Hours . Except for this watch ,8801 The actual time of a watch 27 Hours （24 Japan 20：25~25 Japan 23：25）, achieve 118MB/s, The redistribution rate is much faster than expected .

7、 ... and 、 Summary of experience

1、MPP When doing data redistribution operations, the cluster usually , The execution time of business scheduling must be considered , Because the redistributive operation may cause the business table to lock the table and affect the normal execution of business scheduling , The data synchronization time is before the expansion operation 2 Point to the afternoon 15 spot , Scheduling takes a long time , All business tables needed for scheduling are used before scheduling execution , Increase redistribution priority , Complete redistribution ahead of time , Reduce redistributive concurrency during scheduling execution , So that we can do 24 Hour redistribution , And does not affect production scheduling . If the daily scheduling time is short or there are too many tables to filter those tables needed for scheduling execution , It is recommended to redistribute the data in the wrong time .

2、 In addition to connecting the new node with the network of the cluster , Need to consider with data loader , And hadoop colony （ If the hdp When loading data ） Network connectivity .

3、 It is better to check the inclination of the following table before expansion , It is recommended to adjust the distribution key for tables with large tilt , In order to prevent similar expansion “ Because there is a very uneven table , All data falls on one node ,70 A field ,75 Billion records ,13 Compress , Single slice 350GB. This table alone is used for redistribution 12 Hours ” The situation of .

From the past 40 So far this year , The form of database basically experienced the traditional commercial database 、 The evolution process from open source database to cloud native database . How to innovate and innovate the database in the cloud era ？ How to move and build the core database of financial industry safely and smoothly ？ Come on Gdevops Beijing station of global agile operation and maintenance Summit To find the answer ：

《All in Cloud Time , Next generation cloud native database technology and trend 》 Vice president of Alibaba Group / Chief database scientist of Dharma Hall Fei Fei Li （ Flying cutter ）
《AI And the way of database evolution in the cloud era 》 General manager of Tencent database product center Lin Xiaobin （ Ding Qi ）
《ICBC Of MySQL The road to explore 》 Software development center of ICBC Weiyadong
《 Financial industry MySQL High availability practice 》 Technical director of akerson Mingxiyuan
《 Minsheng Bank is in SQL Exploration and practice in auditing 》 Minsheng Bank Senior database expert Li Ningning
《OceanBase The implementation and practice of distributed database in Bank of Xi'an 》 The ant gold dress P9 Senior experts /OceanBase The core leader Jiang Zhiyong