Detailed description of gaussdb (DWS) complex and diverse resource load management methods

This article is shared from Huawei cloud community 《

GaussDB（DWS）  Resource load management ： Concurrency control and CPU Actual measurement and analysis of control effect 【 Gauss is not a mathematician this time 】

》, author ： Malick .

background

GaussDB（DWS） It provides complex and diverse resource load management methods ： It can be from a single cn The total concurrent number of limits the number of jobs （max_active_statements）, You can also create resource pools , The concurrency limit is applied to the users of the specified resource pool . On the resource pool , That is, the memory 、CPU The limitation of , You can also do without resource restrictions . about CPU Control of resources , That is, you can use a hard limit that specifies a specific number of cores , You can also use idle on-demand allocation ,cpu The soft limit of allocating resources according to the ratio when running full .

​ Because there are so many functional configurations , To make DWS In different business scenarios , Adopt different configuration schemes , Maintain business stability , Ensure the use of resources for important business .

​ For so many control functions , What is the actual effect of control , This article is based on the latest version , Carry out effect measurement , And make some analysis and explanation . It is mainly divided into the following parts ：

• The role of concurrency limit in resource bottleneck

• CPU The actual use effect of the limit

• CPU The actual effect of quotas , The quota CPU And limits CPU Ability comparison

Scene one ： The role of concurrency limit in resource bottleneck

The so-called resource bottleneck , namely CPU、 Memory 、IO、 One or more of the network has reached the bottleneck , Compete for resources before jobs appear , Resulting in a significant reduction in performance . For such scenes , When we solve problems everyday , Several methods usually thought of ：

​1. Reduce business concurrency ;2. Catch those who consume high resources sql sentence , Optimize it ;3. For consumption cpu High job resource constraints , Ensure that adequate resources are available for other operations .

​ Theoretically, every method has an effect , But what's the effect , It can not be simply stated clearly , Data is required for some confirmation .

Environment building

1. To configure ：3 Taiwan physics machine , specifications ：

2. Gauss database （DWS） Cluster specification ：

PS： The cluster version has little effect on the test results , The functional specifications of each version are basically unchanged .

Data structure

test CPU The impact of resource control on flexible short queries and complex queries , Complex queries take TPCDS Data and flexible queries take TPCC data . The structure here 1500x Of TPCDS/100xTPCC.

Data sources ：

• tpcds Data from tpcds Tool construction . It took nearly a night . Start a local gds The server , establish tpcds Corresponding original table and appearance , Direct import .HDD disc , Import performance is also poor .

• tpcc Data is readily available in other test data servers , Create the original table surface directly gds Import ,100x data , Import approx 10min about .

Test ideas

• find tpcds Middle and high CPU Consumed statements , Testing several concurrencies will CPU Full , And the running time should not be too long , Avoid affecting test efficiency .

• Find the statement , Set the concurrent number of a batch of jobs , For example, the overall number of jobs is 30 individual , just 4 Concurrency will CPU Full , Then test under different concurrency control , Operation performance .

• Different and issue the first completion time due to CPU The degree of competition varies , Time is different , So it also needs to be recorded .

Test data

explain ：tpcds-Q9, In this test environment 1500x Under the data , Single concurrency can make physical machines cpu achieve 30%-50%, The single concurrent running time is 100s about .; This test takes Q9*30 As a batch of work . Control different concurrency numbers , Record the operation of each batch ;4 Concurrent cpu Basically, the bottleneck has been reached , Therefore, this round of test starts from 4 Concurrency starts .

The test results are as follows ：

Conclusion analysis

• First, we plot the concurrency number and the overall execution time , Trend chart of single execution time ：

The chart is as follows ：

2. Chart analysis , As can be seen from the line chart above ：

• As the number of concurrent increases , The overall running time has been slightly improved , Description in CPU In the case of bottlenecks , Reduction of concurrency , It does not improve the overall performance of batch jobs .

• The overall average running time of the job is also relatively stable , The average time spent running each job , There is no big difference under different concurrency numbers .

• Run time of the first completed job , The concurrency number is 4 Under the circumstances , Only 400s+, And in the concurrent number 30 The situation of fullness , Reached 1620s+, have a long way to go , The change trend is basically linear with the increase of concurrency .

General description

Analyze according to the test conclusion , stay CPU In the case of bottlenecks , Limit concurrency , In fact, it can not improve the performance of the overall operation ; But in different scenarios , Different configuration policies can be selected .

for example ： Need to have timely response to work , You can limit the number of concurrency to less , This will ensure that there will always be jobs that can be completed at a faster speed ; It is required that the overall job run faster , According to the test data , You can set the concurrency number to large , In this way, the overall running time is the shortest .

Click to follow , The first time to learn about Huawei's new cloud technology ~