CODIS stress test (PHP)

One ). summary ： 

  1.   Without affecting the application delay , Short link (3.3W) And long links (15.5W) QPS Differences in 5 times .

  2.   In long link ( No significant increase in delay ) Under the circumstances , No error is reported in the application , And the average delay is reduced 50% about .

  3.  codis proxy A single node QPS The maximum bearing capacity is as follows ：   

Machine model	QPS	Maximum system index (cpu)
8c32G	～15.5W	75%
32c12G	～22.5W	75%

Two ). Test details ： 

  1. The test model ：

Number of back-end services	codis proxy Number	proxy To configure	Back end server configuration	Pressure measuring tool	Pressure test order
20 individual	8 individual	8c 32G	32C 128G	ab + jmeter	ab -c 10000 -n 2000000 http://interact.int.yidian-inc.com/test/ttt/redistest
20 individual	8 individual	8c 32G	32C 128G	ab + jmeter	jmeter -n -t redis3.jmx -l log1.jtl -Jthn=100000 -Jloopn=999999999

remarks ：redis With pressure measuring tools

./redis-benchmark -t set,get -h 10.138.20.97 -p 19000 -n 5000000 -d 384 -k 0 -c 1000 --threads 10

2. Short link pressure test ：

        Link request method ： One connection , Two operations ( Read and write )

Number of machines * Number of single machine concurrent threads													Total number of requests	Pressure test times	avg A single node proxy QPS	avg 8 President Tai QPS	The average number of business errors	Back end service latency
1*10000													6000000	4	0.97W （0.97W 0.97W 0.98W 0.96W）	7.6W （7.7W 7.6W 7.8W 7.7W）	1.3 （0, 0, 4 ,1）	16ms
2*10000													6000000*2	4	1.9W （1.9W 1.9W 1.8W 1.9W）	15.3W （15.4W 15.4W 15.2W 15.2W）	0 （0, 0, 0 ,0）	44ms
3*10000													6000000*3	3	2.85W （2.85W 2.85W 2.85W ）	22.5W （22.5W 22.5W 22.5W）	10.3 (12, 11 ,8)	340ms
4*10000													6000000*4	3	3.3W （3.3W 3.3W 3.3W ）	26.5W （26.5W 26.5W 26.5W）	19 (32 ,7 ,18)	400ms
5*10000													6000000*5	2	3.3W （3.3W 3.3W ）	26.5W （26.5W 26.5W ）	16	420ms

summary ： When using short links to access codis proxy, Single proxy achieve 3w qps, Need to expand proxy To increase the number of the entire cluster qps, When pressing a single proxy When ,proxy Maximum attainable 5.4 ten thousand ,axe The service delay has risen to 1500ms. There are many errors reported in the backend log .

3.   Short link adjust kernel after pressure test ：

How to optimize ：

  Kernel parameters tcp time_wait reusing   net.ipv4.tcp_timestamps 0 ---> 1

        Link request method ： One connection , Two operations ( Read and write )

Number of pressure measuring machines * Number of threads													Total number of requests	Pressure test times	avg A single node proxy QPS	Mean value when not optimized	avg total QPS	avg Number of business errors	Not optimized error mean
1*10000													6000000	1	0.96W	0.97W	7.5w	0	1.3
2*10000													6000000*2	3	1.9W	1.9W	15.3W	68,39, 77	0
3*10000													6000000*3	1	2.85W	2.85W	22.5W	116	10.3
4*10000													6000000*4	1	3.4W	3.3W	27W	503	19

          Conclusion ： The effect is not improved , The number of application errors increased slightly .

4. Long link pressure measurement ：

        Single pressure measurement ：

        Link request method ： One connection , Two operations ( Read and write )

Number of pressure measuring machines * Number of threads													Total number of requests	Pressure test times	avg A single node proxy QPS	avg Number of business errors	proxy CPU Use	O & M monitoring threshold
2*10000													6000000 * 2	3	32C：15.5W 8C：15.5W	0	32C：20% 8C：75%	70%
3*10000													6000000 * 3	3	32C：22.5W 8C：20W	0	32C：75% 8C：98%	70%
4*10000													6000000 * 4	3	32C：30W	0	32C：80%	70%

summary ：32c 128G Of proxy node qps Maximum 15.5W,CPU utilization 20%, Business monitoring indicator backend latency 10ms about , There is no business error report .

Cluster pressure measurement ：

        Link request method ： One connection , Two operations ( Read and write )

Number of pressure measuring machines * Number of threads													Pressure test times	avg total QPS
6 * 10000													2	46W
4 * 100000													1	70W

summary ：32c 128G Of proxy node qps Maximum 15.5W,CPU utilization 20%, Business monitoring indicator backend latency 10ms about , There is no business error report .