当前位置：网站首页>Pgbouncer lightweight PG connection pool management tool

Pgbouncer lightweight PG connection pool management tool

2022-06-24 04:40:00 【Tencent cloud database tencentdb】

introduction ： Once heard a saying , use PostgreSQL Without connection pooling , It's definitely what my father did …… Even if it's like pgbouncer In this way “ Lightweight ” Connection pool , The difference between "yes" and "no" is not so big .

Although this sentence is quite absolute , But in a way , There is some truth in this sentence .

pgbouncer advantage ：

It can effectively improve the utilization of connections , Avoid too many invalid connections , This causes the database to consume too much resources ,CPU Occupy too much .
Restrict client connections , Prevent excessive or malicious connection requests .

pgbouncer Introduce ：

PG It's a multiprocess structure , Each new session adds a new process , Relatively speaking, the cost to the database will be huge .

Because in a normal business session , Quite a few session There's been a long time idle The state of , And this state leads to , At this point, if you need to connect , You need to build a new process , To access the database , So the connection number comes up . While using pgbouncer The main reason is

PGBrouncer Ability to cache and PostgreSQL The connection of , When a connection request comes in , Direct distribution pgbouncer And postgresql The idle connection between , Without the need for PostgreSQL fork Create a new process to establish a connection , To save on creating new processes , The resource consumption of creating a connection .

The key is pgbouncer It's using libevent Conduct socket signal communication , High communication efficiency . Each connection consumes only 2kB Memory . relative pg At every turn 4MB Of work_mem It is very lightweight .

If you want to use vernacular to say , Not used pgbouncer It's a private car , If there are too many cars , Then the road is full of , And used pgbouncer It's like a bus or a minibus , Someone got in the car, someone got out of the car , The seats are fixed , So if the bus itself has 30 A seat , But in fact, it's not just about supporting 30 personal , As for how many people supported , It depends on the speed of transactions connected to the database , Whether a connection can be reused , It's a little CPU The concept of time sharing of .

pgbouncer Connection pool mode ：

pgbouncer Three connection pooling modes are supported ：

session This is for session Speaking of , When the user actively releases the connection ,pgbouncer Will release and reuse the connection between it and the database . The setting itself is similar to PG The meaning of the connection logic is basically the same . So it's better not to , This mode is not commonly used .
Transaction The most commonly used and appropriate model , A connected channel is time-sharing Business Used by the , This is better than the above session The pattern contrast is much more obvious , The utilization of the connection will follow the relevant allocation with higher reuse , And performance improvements . After the transaction is sent from the client ,pgbouncer A connection will be found in the connection with the database idle Go and throw it in and do it , After completing the transaction , The connection will be restarted pgbouncer To be used by other new affairs .
SQL Be similar to autocommit, The smallest particle size , One SQL Release connection upon execution , The connection will be occupied by other sessions . Although the use of resources is the most extreme , But it does not conform to the general business scenario . Breaking the transaction logic .

pgbouncer Installation configuration method ：

Official website address ：PgBouncer - lightweight connection pooler for PostgreSQL

1、 Download the latest release edition ,1.160.0 edition .

Prepare resources ,1 Cloud servers ,1 A cloud database PostgreSQL Main instance ,1 A cloud database PostgreSQL Read only instance .
Both ECs and cloud database are required to be unified VPC Inside , Require to be able to pass CVM Access to the database instance .
Be careful ： I use the test here centos 7.2 Version ECS , The database version is PostgreSQL 11.8.

2、 Log in to ECs , And upload pgbouncer To ECs .

# To install a pgbench, It can be used to test later ：

# yum install -yhttps://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

# yum install -y postgresql11-server

# establish pgbouncer Storage directory of

# mkdir /opt/pgbouncer

# Enter the uploaded pgbouncer Directory , unpack ：

# tar -xf pgbouncer-1.16.0.tar.gz

# cd pgbouncer-1.16.0

# Perform compilation and installation , And specify the installation path as /opt/pgbouncer：

# ./configure --prefix=/opt/pgbouncer

# make

# make install

# Be careful , When performing the first step , You will be prompted that various dependent libraries are not installed , At this point, you only need to install the prompted dependent libraries devel Bag can . Such as ：libevent-devel,openssl-devel

# yum install -y libevent-devel openssl-devel

# Finally, give the entire folder permissions to postgres Users to manage

# chown postgres:postgres /opt/pgbouncer -R

3、 After executing the above statement , Can be pgbouncer installation is complete . At this point, you need only pgbrouncer The configuration of the .

The main execution file is ：/opt/pgbouncer/bin/pgbouncer
The configuration file is ：/opt/pgbouncer/share/doc/pgbouncer/pgbouncer.ini
The user whitelist configuration file is ：/opt/pgbouncer/share/doc/pgbouncer/userlist.txt

4、 Of course, you can also copy the above three files to a directory for management , It's more convenient , The related commands are described as follows ：

-v Parameters Execute... For foreground output , Relevant logs can be printed to the screen , But after closing the window , The program will close , Commonly used in debug. Example ：/opt/pgbouncer/bin/pgbouncer -v /opt/pgbouncer/share/doc/pgbouncer/pgbouncer.ini
-d Parameters by daemon perform , The normal program can be executed in this way . Example ：/opt/pgbouncer/bin/pgbouncer -d /opt/pgbouncer/share/doc/pgbouncer/pgbouncer.ini
-R Parameter is Restart the program , After modifying the configuration file , You can restart to load the configuration file . /opt/pgbouncer/bin/pgbouncer -R /opt/pgbouncer/share/doc/pgbouncer/pgbouncer.ini
Other parameters are not important , Executable pgbouncer --help Check it out. .

4、pgbouncer.ini How to configure files , Just look at the example below ：

[databases] block , It mainly configures the access connection string of the database .
[users] block , It mainly configures some private configurations at the user level , For example, configure the connection pool mode of a user
[pgbouncer] block , It mainly configures global pgbouncer Pattern .

[databases]

db_rw = host=172.27.32.11port=5432dbname=test

db_ro = host=172.27.32.12port=5432dbname=test

; here At the very beginning xxx= It can be configured at will , Mainly connection representation , Easy to remember . It is recommended to configure according to business conditions .

; Note that the application is connected to... Through this connection string database In the after , Unable to switch database.

; host, port, dbname Must provide access connection information for the database , Here, I configure the primary instance of cloud database as db_rw, Configure the read-only instance of this primary instance as db_ro, When I use the client to access the database , visit db_ro That is, access the following libraries .

; The user name and password can also be configured here , But it is not recommended to configure it here , Because the business or database has multiple users , If the user is configured here , You can only use this user to access the database . If you want to set which users can access , It can be used userlist File to limit .

[users]

; Example ：user1 = pool_mode=transaction max_user_connections=10

; The user level connection pool mode is configured here , If there is a special configuration for a user, it can be configured here , If not , Leave it blank .

[pgbouncer]

listen_addr =172.27.32.17

listen_port =5432

; The service listens for addresses and ports .

logfile = /opt/pgbouncer/pgbouncer.log

pidfile = /opt/pgbouncer/pgbouncer.pid

unix_socket_dir = /var/run/pgbouncer

unix_socket_mode =0777

unix_socket_group =

auth_type = trust

auth_file = /opt/pgbouncer/share/doc/pgbouncer/userlist.txt

; here auth_type You can configure the md5, But on the database side pg_hba.conf The file also needs to be configured as md5, And userlist The configuration file needs to be saved md5 The password after , Cannot be configured as clear text , If set to trust, You need to set it to plaintext .

pool_mode = transaction

; Connection pool mode , Configurable session,transaction,sql, It is recommended to use transaction. The default is session.

max_client_conn =100

; Maximum number of connections configuration

default_pool_size =20

; Default connection pool size

min_pool_size =0

reserve_pool_size =5

reserve_pool_timeout =3

5、 To configure userlist.txt：

The format is ："user1" "password" , The first column is User name allowed to access , The second column is the password of this user .
As in the main configuration file auth_type Set to trust, The password is configured as clear text .
As in the main configuration file auth_type Set to md5, The password is configured as md5 value , The super user can log in to the database , Execute statement ：select rolname,rolpassword from pg_authid, Copy the user name and the displayed encrypted password .

6、 When the configuration is complete , You can pass postgres User start pgbouncer.

$ Start command ：/opt/pgbouncer/bin/pgbouncer -d /opt/pgbouncer/share/doc/pgbouncer/pgbouncer.ini

7、 After starting up , You can check whether the startup is successful through the following command

$ ps -ef |grep pgbouncer

postgres 17171 1 0 16:41 ? 00:00:00 /opt/pgbouncer/bin/pgbouncer -d /opt/pgbouncer/share/doc/pgbouncer/pgbouncer.ini

$ netstat -anlp|grep pgbouncer

tcp 0 0 172.27.32.17:5432 0.0.0.0:* LISTEN 17171/pgbouncer

Performance experience ：

test result , Use pgbouncer Very low impact on performance , And make full use of connection resources , The number of sets currently tested is not large , Based on the current test results , I believe that in large-scale concurrent scenarios , It can also reduce the burden on the database , But the stability needs to be verified .

adopt PostgreSQL Native connection and pgbouncer Conduct access pressure test to check the implementation , Set the connection to 100 individual , Pressure measurement for each 2 minute , One minute in between .

Pressure test the main instance through the primary connection ： pgbench -U haha -p 5432 -h 172.27.32.11 -d test -r -P 1 -c 100 -j 32 -T 120 -D scale=10000 -D range=100000000
adopt pgbouncer Main instance of pressure measurement ： pgbench -U haha -p 5432 -h 172.27.32.17 -d db_rw -r -P 2 -c 100 -j 32 -T 120 -D scale=10000 -D range=100000000

View database monitoring , You can see that it works pgbouncer The number of connections is reduced 33%,QPS Almost no change , The average response time delay does not change significantly .

The first bulge is Use pg The number of connections accessed by native connections , On average 113 about , It is consistent with the pressure measurement , The lower line in the back is through pgboucer The results of pressure measurement , The number of visible connections has decreased 30%, On average 30 about .