PostgreSQL highly available repmgr (1 master 2 slave +1witness) + pgpool II realizes master-slave switching + read-write separation

1. One 、 Overall framework
2. Two 、 Environmental applications
3. 3、 ... and 、 To configure 4 Mutual trust between hosts
4. Four 、 To configure repmgr
   1. 4.1、 Master database modification pg_hba.conf Parameter file
   2. 4.2、 Master database modification postgresql.conf Parameter file
   3. 4.3、 The main library creates related users and databases
   4. 4.4、4 Nodes are modified separately repmgr.conf
   5. 4.5、 Register the main database service in the main database
   6. 4.6、 Clone the standby database 1 Spare warehouse 2
      1. 4.6.1、4 Each node is equipped with ~/.pgpass Password file
      2. 4.6.2、 Clone the standby database 1
5. 8、 ... and 、 summary

One 、 Overall framework

The final goal of this paper ：

1、1 Lord 2 Copy from stream

2、 Read / write separation + Load balancing （pgpool-II）

3、 Master slave auto switch （repmgr）

All nodes ： Installing the operating system 、 establish PG User directory 、 install PG Software 、 install repmgr

Main library ： Only initialize the main library 、 Start the main database archiving

Two 、 Environmental applications

Apply first 4 Taiwan machine ,6.66 The node finally applies , Anyway docker Containers are ready to use .

--  Main library docker rm -f lhrrepmgr64361docker run -d --name lhrrepmgr64361 -h lhrrepmgr64361 \  --net=pg-network --ip 172.72.6.61 \  -p 64361:5432  \  -v /sys/fs/cgroup:/sys/fs/cgroup \  --privileged=true lhrbest/lhrpgall:1.0 \  /usr/sbin/initdocker network connect bridge lhrrepmgr64361--  Slave Library 1docker rm -f lhrrepmgr64362docker run -d --name lhrrepmgr64362 -h lhrrepmgr64362 \  --net=pg-network --ip 172.72.6.62 \  -p 64362:5432  \  -v /sys/fs/cgroup:/sys/fs/cgroup \  --privileged=true lhrbest/lhrpgall:1.0 \  /usr/sbin/initdocker network connect bridge lhrrepmgr64362--  Slave Library 2docker rm -f lhrrepmgr64363docker run -d --name lhrrepmgr64363 -h lhrrepmgr64363 \  --net=pg-network --ip 172.72.6.63 \  -p 64363:5432  \  -v /sys/fs/cgroup:/sys/fs/cgroup \  --privileged=true lhrbest/lhrpgall:1.0 \  /usr/sbin/initdocker network connect bridge lhrrepmgr64363-- Witness library docker rm -f lhrrepmgr64364docker run -d --name lhrrepmgr64364 -h lhrrepmgr64364 \  --net=pg-network --ip 172.72.6.64 \  -p 64364:5432  \  -v /sys/fs/cgroup:/sys/fs/cgroup \  --privileged=true lhrbest/lhrpgall:1.0 \  /usr/sbin/initdocker network connect bridge lhrrepmgr64364--  restart 4 Taiwan machine docker restart lhrrepmgr64361 lhrrepmgr64362 lhrrepmgr64363 lhrrepmgr64364[root@docker35 ~]# docker ps | grep repmgr 31d3d31c1073        lhrbest/lhrpgall:1.0                       "/usr/sbin/init"         41 hours ago        Up 41 hours         5433-5435/tcp, 0.0.0.0:64364->5432/tcp                                                               lhrrepmgr64364572d4ea2c072        lhrbest/lhrpgall:1.0                       "/usr/sbin/init"         41 hours ago        Up 41 hours         5433-5435/tcp, 0.0.0.0:64363->5432/tcp                                                               lhrrepmgr643636ded416b2016        lhrbest/lhrpgall:1.0                       "/usr/sbin/init"         41 hours ago        Up 41 hours         5433-5435/tcp, 0.0.0.0:64362->5432/tcp                                                               lhrrepmgr64362fa38b58b8f3d        lhrbest/lhrpgall:1.0                       "/usr/sbin/init"         41 hours ago        Up 41 hours         5433-5435/tcp, 0.0.0.0:64361->5432/tcp                                                               lhrrepmgr64361[root@docker35 ~]#

Be careful ：

1、 Mirror image lhrbest/lhrpgall:1.0 in , Applied 4 All hosts have been installed PG 13 and repmgr Software

2、PG The installation method is compile installation , The database has been initialized , The user is pg13

3、 The image has been installed PG13、PG12、PG11、PG10 and PG9.6 These versions , Are compiled and installed , This article USES the PG13 To operate

3、 ... and 、 To configure 4 Mutual trust between hosts

--  Only in the main library 61 Node , With root User execution ：chmod +x sshUserSetup.sh./sshUserSetup.sh -user pg13  -hosts "172.72.6.61 172.72.6.62 172.72.6.63 172.72.6.64" -advanced exverify -confirm--  Modify permissions on each host , Otherwise, the report will be wrong ：Bad owner or permissions on /home/pg13/.ssh/configchmod 600 /home/pg13/.ssh/config

Input in sequence ：yes, Just enter and password .

Script Download ：https://share.weiyun.com/57HUxNi

verification ：

ssh pg13@lhrrepmgr64361 datessh pg13@lhrrepmgr64362 datessh pg13@lhrrepmgr64363 datessh pg13@lhrrepmgr64364 date

The second execution will not prompt for input yes, And can successfully execute commands , said SSH Peer configuration succeeded .

Execution process ：

[pg13@lhrrepmgr64361 /]# ssh [email protected] dateTue Apr 27 17:15:04 CST 2021[root@lhrrepmgr64361 /]# [pg13@lhrrepmgr64361 /]# ssh [email protected] dateTue Apr 27 17:15:08 CST 2021[root@lhrrepmgr64361 /]# [pg13@lhrrepmgr64361 /]# ssh [email protected] dateTue Apr 27 17:15:13 CST 2021[pg13@lhrrepmgr64361 /]# 

Four 、 To configure repmgr

4.1、 Master database modification pg_hba.conf Parameter file

--  Enter the main library docker exec -it lhrrepmgr64361 bashsu - pg13cat >> /pg13/pgdata/pg_hba.conf <<"EOF"local repmgr repmgr md5host repmgr repmgr 127.0.0.1/32 md5host repmgr repmgr 172.72.6.0/24 md5local replication repmgr md5host replication repmgr 127.0.0.1/32 md5host replication repmgr 172.72.6.0/24 md5EOF

4.2、 Master database modification postgresql.conf Parameter file

--  Modify the parameters cat >> /pg13/pgdata/postgresql.conf <<"EOF"#  Archive parameters wal_level='replica'archive_mode='on'archive_command='test ! -f /pg13/archive/%f && cp %p /pg13/archive/%f'restore_command='cp /pg13/archive/%f %p'#  Master slave stream replication hot_standby=onmax_wal_senders=10wal_sender_timeout=60swal_keep_size=16MB#  Master slave switching parameters , Enable PG Replication slot of database ,PG12 Unwanted "use_replication_slots=true" This parameter .max_replication_slots=10wal_log_hints=on#  Automatic switch shared_preload_libraries ='repmgr'EOF--  restart pg_ctl start--  Inquire about psql -U postgres -h 192.168.66.35 -p 64361select * from pg_settings where name in ('wal_level','archive_mode','archive_command');--  Switch Archive select pg_switch_wal();

4.3、 The main library creates related users and databases

--  Create relevant users and databases su - pg13createuser -s repmgrcreatedb repmgr -O repmgrpsql -h 127.0.0.1 -c "alter user repmgr with password 'lhr';"psql -h 127.0.0.1 -c "alter user repmgr set search_path to repmgr, \"\$user\",public;"

result ：

[root@lhrrepmgr64361 /]# systemctl start pg13[root@lhrrepmgr64361 /]# systemctl status pg13● pg13.service - PostgreSQL database server   Loaded: loaded (/etc/systemd/system/pg13.service; disabled; vendor preset: disabled)   Active: active (running) since Tue 2021-04-27 16:25:24 CST; 6s ago     Docs: man:postgres(1)  Process: 769 ExecStart=/pg13/pg13/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS) Main PID: 771 (postgres)   CGroup: /docker/a777ef12d5ff83a9d47be51a98531bd45b42d2b008f7a25f894f3244ce9cc0d4/system.slice/pg13.service           ├─771 /pg13/pg13/bin/postgres -D /pg13/pgdata -p 5432           ├─772 postgres: logger            ├─774 postgres: checkpointer            ├─775 postgres: background writer            ├─776 postgres: walwriter            ├─777 postgres: autovacuum launcher            ├─778 postgres: stats collector            └─779 postgres: logical replication launcher Apr 27 16:25:23 lhrrepmgr64361 systemd[1]: Starting PostgreSQL database server...Apr 27 16:25:24 lhrrepmgr64361 pg_ctl[769]: 2021-04-27 16:25:24.024 CST [771] LOG:  redirecting log output to logging collector processApr 27 16:25:24 lhrrepmgr64361 pg_ctl[769]: 2021-04-27 16:25:24.024 CST [771] HINT:  Future log output will appear in directory "pg_log".Apr 27 16:25:24 lhrrepmgr64361 systemd[1]: Started PostgreSQL database server.[root@lhrrepmgr64361 /]# su - pg13Last login: Tue Apr 27 16:24:50 CST 2021 on pts/0[pg13@lhrrepmgr64361 ~]$ [pg13@lhrrepmgr64361 ~]$ createuser -s repmgr[pg13@lhrrepmgr64361 ~]$ [pg13@lhrrepmgr64361 ~]$ createdb repmgr -O repmgr[pg13@lhrrepmgr64361 ~]$ [pg13@lhrrepmgr64361 ~]$ psql -h 127.0.0.1 -c "alter user repmgr with password 'lhr'"ALTER ROLE[pg13@lhrrepmgr64361 ~]$ [pg13@lhrrepmgr64361 ~]$ psql -h 127.0.0.1 -c "alter user repmgr set search_path to repmgr, \"\$user\",public";ALTER ROLEC:\Users\lhrxxt>psql -U postgres -h 192.168.66.35 -p 64361Password for user postgres:psql (13.2)Type "help" for help.postgres=# \du                                   List of roles Role name |                         Attributes                         | Member of-----------+------------------------------------------------------------+----------- postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {} repmgr    | Superuser, Create role, Create DB                          | {}postgres=# \l                                 List of databases   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges-----------+----------+----------+------------+------------+----------------------- postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | repmgr    | repmgr   | UTF8     | en_US.utf8 | en_US.utf8 | template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +           |          |          |            |            | postgres=CTc/postgres template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +           |          |          |            |            | postgres=CTc/postgres(4 rows)

4.4、4 Nodes are modified separately repmgr.conf

--  With pg13 The user to change su - pg13--  Main library cat > /pg13/pg13/repmgr.conf << "EOF"node_id=1node_name=lhrrepmgr64361conninfo='host=172.72.6.61 user=repmgr password=lhr dbname=repmgr connect_timeout=2'data_directory='/pg13/pgdata'pg_bindir='/pg13/pg13/bin'EOF--  Slave Library 1cat > /pg13/pg13/repmgr.conf << "EOF"node_id=2node_name=lhrrepmgr64362conninfo='host=172.72.6.62 user=repmgr password=lhr dbname=repmgr connect_timeout=2'data_directory='/pg13/pgdata'pg_bindir='/pg13/pg13/bin'EOF--  Slave Library 2cat > /pg13/pg13/repmgr.conf << "EOF"node_id=3node_name=lhrrepmgr64363conninfo='host=172.72.6.63 user=repmgr password=lhr dbname=repmgr connect_timeout=2'data_directory='/pg13/pgdata'pg_bindir='/pg13/pg13/bin'EOF-- witness node cat > /pg13/pg13/repmgr.conf << "EOF"node_id=4node_name=lhrrepmgr64364conninfo='host=172.72.6.64 user=repmgr password=lhr dbname=repmgr connect_timeout=2'data_directory='/pg13/pgdata'pg_bindir='/pg13/pg13/bin'EOF--  test psql 'host=172.72.6.61 user=repmgr password=lhr dbname=repmgr connect_timeout=2'psql 'host=172.72.6.62 user=repmgr password=lhr dbname=repmgr connect_timeout=2'psql 'host=172.72.6.63 user=repmgr password=lhr dbname=repmgr connect_timeout=2'psql 'host=172.72.6.64 user=repmgr password=lhr dbname=repmgr connect_timeout=2'

4.5、 Register the main database service in the main database

--  Registration service repmgr -f /pg13/pg13/repmgr.conf primary register--  Inquire about repmgr -f /pg13/pg13/repmgr.conf cluster showpsql -U repmgr -h 192.168.66.35 -p 64361 -d repmgr

Execution process ：

[pg13@lhrrepmgr64361 pg13]$ repmgr -f /pg13/pg13/repmgr.conf primary registerINFO: connecting to primary database...NOTICE: attempting to install extension "repmgr"NOTICE: "repmgr" extension successfully installedNOTICE: primary node record (ID: 1) registered[pg13@lhrrepmgr64361 pg13]$ [pg13@lhrrepmgr64361 pg13]$ repmgr -f /pg13/pg13/repmgr.conf cluster show ID | Name           | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string                                                        ----+----------------+---------+-----------+----------+----------+----------+----------+--------------------------------------------------------------------------- 1  | lhrrepmgr64361 | primary | * running |          | default  | 100      | 1        | host=172.72.6.61 user=repmgr password=lhr dbname=repmgr connect_timeout=2  C:\Users\lhrxxt>psql -U repmgr -h 192.168.66.35 -p 64361 -d repmgrPassword for user repmgr:psql (13.2)Type "help" for help.repmgr=# select * from repmgr.nodes; node_id | upstream_node_id | active |   node_name    |  type   | location | priority |                                 conninfo                                  | repluser | slot_name |      config_file---------+------------------+--------+----------------+---------+----------+----------+---------------------------------------------------------------------------+----------+-----------+------------------------       1 |                  | t      | lhrrepmgr64361 | primary | default  |      100 | host=172.72.6.61 user=repmgr password=lhr dbname=repmgr connect_timeout=2 | repmgr   |           | /pg13/pg13/repmgr.conf(1 row)

4.6、 Clone the standby database 1 Spare warehouse 2

4.6.1、4 Each node is equipped with ~/.pgpass Password file

su - pg13echo "*:*:*:repmgr:lhr" > ~/.pgpasschmod 0600 ~/.pgpass--  test , If you don't need a password , That proves the configuration is correct psql 'host=172.72.6.61 user=repmgr dbname=repmgr connect_timeout=2'psql 'host=172.72.6.62 user=repmgr dbname=repmgr connect_timeout=2'psql 'host=172.72.6.63 user=repmgr dbname=repmgr connect_timeout=2'psql 'host=172.72.6.64 user=repmgr dbname=repmgr connect_timeout=2'

4.6.2、 Clone the standby database 1

--  among --dry-run It means that the cloning process is unreal , Just check if there is a problem repmgr -h 172.72.6.61 -U repmgr -d repmgr -f /pg13/pg13/repmgr.conf standby clone --force --dry-run--  Execute the clone command , In fact, it is still called pg_basebackuprepmgr -h 172.72.6.61 -U repmgr -d repmgr -f /pg13/pg13/repmgr.conf standby clone --force --  start-up pg_ctl -D /pg13/pgdata start--  Inquire about psqlselect * from pg_stat_wal_receiver;

Execution process ：

[pg13@lhrrepmgr64362 ~]$ repmgr -h 172.72.6.61 -U repmgr -d repmgr -f /pg13/pg13/repmgr.conf standby clone --force --dry-runNOTICE: destination directory "/pg13/pgdata" providedINFO: connecting to source nodeDETAIL: connection string is: host=172.72.6.61 user=repmgr dbname=repmgrDETAIL: current installation size is 29 MBINFO: "repmgr" extension is installed in database "repmgr"WARNING: target data directory appears to be a PostgreSQL data directoryDETAIL: target data directory is "/pg13/pgdata"HINT: use -F/--force to overwrite the existing data directoryINFO: replication slot usage not requested;  no replication slot will be set up for this standbyINFO: parameter "max_wal_senders" set to 10NOTICE: checking for available walsenders on the source node (2 required)INFO: sufficient walsenders available on the source nodeDETAIL: 2 required, 10 availableNOTICE: checking replication connections can be made to the source server (2 required)INFO: required number of replication connections could be made to the source serverDETAIL: 2 replication connections requiredWARNING: data checksums are not enabled and "wal_log_hints" is "off"DETAIL: pg_rewind requires "wal_log_hints" to be enabledNOTICE: standby will attach to upstream node 1HINT: consider using the -c/--fast-checkpoint optionINFO: all prerequisites for "standby clone" are met[pg13@lhrrepmgr64362 ~]$ repmgrrepmgr -h 172.72.6.61 -U repmgr -d repmgr -f /pg13/pg13/repmgr.conf standby clone^C[pg13@lhrrepmgr64362 ~]$ repmgr -h 172.72.6.61 -U repmgr -d repmgr -f /pg13/pg13/repmgr.conf standby cloneNOTICE: destination directory "/pg13/pgdata" providedINFO: connecting to source nodeDETAIL: connection string is: host=172.72.6.61 user=repmgr dbname=repmgrDETAIL: current installation size is 29 MBERROR: target data directory appears to be a PostgreSQL data directoryDETAIL: target data directory is "/pg13/pgdata"HINT: use -F/--force to overwrite the existing data directory[pg13@lhrrepmgr64362 ~]$ repmgr -h 172.72.6.61 -U repmgr -d repmgr -f /pg13/pg13/repmgr.conf standby clone --forceNOTICE: destination directory "/pg13/pgdata" providedINFO: connecting to source nodeDETAIL: connection string is: host=172.72.6.61 user=repmgr dbname=repmgrDETAIL: current installation size is 29 MBINFO: replication slot usage not requested;  no replication slot will be set up for this standbyNOTICE: checking for available walsenders on the source node (2 required)NOTICE: checking replication connections can be made to the source server (2 required)WARNING: data checksums are not enabled and "wal_log_hints" is "off"DETAIL: pg_rewind requires "wal_log_hints" to be enabledWARNING: directory "/pg13/pgdata" exists but is not emptyNOTICE: -F/--force provided - deleting existing data directory "/pg13/pgdata"NOTICE: starting backup (using pg_basebackup)...HINT: this may take some time; consider using the -c/--fast-checkpoint optionINFO: executing:  /pg13/pg13/bin/pg_basebackup -l "repmgr base backup"  -D /pg13/pgdata -h 172.72.6.61 -p 5432 -U repmgr -X stream Password: WARNING:  skipping special file "./.s.PGSQL.5432"WARNING:  skipping special file "./.s.PGSQL.5432"NOTICE: standby clone (using pg_basebackup) completeNOTICE: you can now start your PostgreSQL serverHINT: for example: pg_ctl -D /pg13/pgdata startHINT: after starting the server, you need to register this standby with "repmgr standby register"[pg13@lhrrepmgr64362 ~]$ pg_ctl -D /pg13/pgdata startwaiting for server to start....2021-04-27 19:14:08.750 CST [9298] LOG:  redirecting log output to logging collector process2021-04-27 19:14:08.750 CST [9298] HINT:  Future log output will appear in directory "pg_log". doneserver started[pg13@lhrrepmgr64362 ~]$ psqlpsql (13.2)Type "help" for help.postgres=# select * from pg_stat_wal_receiver; pid  |  status   | receive_start_lsn | receive_start_tli | written_lsn | flushed_lsn | received_tli |      last_msg_send_time       |     last_msg_receipt_time     | latest_end_lsn |        latest_end_time        | slot_name | sender_host | sender_port |                                                                                                                                                           conninfo                                                                                                                                                            ------+-----------+-------------------+-------------------+-------------+-------------+--------------+-------------------------------+-------------------------------+----------------+-------------------------------+-----------+-------------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9304 | streaming | 0/3000000         |                 1 | 0/30001F0   | 0/30001F0   |            1 | 2021-04-27 19:15:08.862599+08 | 2021-04-27 19:15:08.862719+08 | 0/30001F0      | 2021-04-27 19:14:08.830865+08 |           | 172.72.6.61 |        5432 | user=repmgr password=******** channel_binding=disable connect_timeout=2 dbname=replication host=172.72.6.61 port=5432 application_name=lhrrepmgr64362 fallback_application_name=walreceiver sslmode=disable sslcompression=0 ssl_min_protocol_version=TLSv1.2 gssencmode=disable krbsrvname=postgres target_session_attrs=any(1 row)

Information “NOTICE: standby clone (using pg_basebackup) complete” Indicates that the test is complete .

8、 ... and 、 summary

1、 Due to the master-slave switch, the database needs to be started , therefore , In the whole environment , be-all PG Databases cannot be used systemctl To manage , Otherwise, switching will cause problems .

2、 To clear the registration information of the node , You can also enter the main library , operation repmgr.nodes Table information .