TCP million concurrent server optimization parameters

C语言TCPServer million concurrent parameter tuning optimization

背景

combine previously doneC语言TCP服务端程序,The maximum number of concurrency it can accept is low,故我们希望Make some coding or system parameter changesto improveTCP服务器的最大并发数

注：The maximum number of concurrency mentioned here（or maximum concurrency）,**is the number of clients（**即通信socket的数量）

instead of the number of requests per secondqps,注意区别！

The final result of this experiment did not reach one million concurrent,只接近80w,Therefore, the focus of this blog is onWhat are the tuning parameters and code modifications?.

实验准备

准备四台虚拟机 版本皆为 20.04

服务器：4g内存,双核

three clients：2g内存,单核

若想要修改Ubuntu静态IP地址可参考:https://blog.csdn.net/baidu_39332177/article/details/123131601

优化调参

出现Connection refused错误

描述

当建立1024个socket的clientfd之后,Other clients will appear when they want to connect again.Connection refused错误

问题原因

通过ulimit -a View the server-side defaultopen filesThe number of parameters is only1024个

解决办法

修改open files数量,方法如下

• 方法一：临时修改,输入sudo ulimit -n 1048576,However, after restarting, the original settings are restored.
• 方法二：永久修改,输入 sudo vim /etc/security/limits.conf Then add the relevant settings at the end of the file

hardRefers to the warning settings,可以超过这个值,But there is a warning after exceeding

softmeans strict setting,不允许超过这个值

When a client connect to the serverToo many open files错误

描述

原因和解决办法

​ Since we only modified it on the server sideopen files的数量,必须要在客户端也将open files设置为1048576

出现Cannot assign requested address错误

描述

原因分析

​ 首先,The problem is that the client address or the server address cannot be assigned？sockfdWhat is the relationship between the network address？

​ 根据socketA quintuple can be found socket–>(远程IP,远程端口,本机IP,本机端口,协议),socketis a one-to-one relationship with quintuple,Through this quintuple it is possible to passrecv和send函数来收发数据.

​ So the cause of the problem is that the combination of quintuple is exhausted,服务器的本机IP以及远程IPand the remote port number has been determined,So the only thing that can be used is to increase the number of native ports to increasesocket的数量.

​ 注意：The maximum number of local ports can only be65535

解决办法

​ 使用100port number and server address andsocket的绑定,得到100monitorfd,and put it onepoll树,由epollto monitor the arrival of events,The key code is modified as follows：

int epfd = epoll_create(1); // 1.创建一个监视事件的管理员fd
    int sockfds[MAX_PORT] = {
    0}; // listen fd集合

    int i = 0;
    //开MAX_PORT个端口进行监听
    for(i = 0;i < MAX_PORT; ++i)
    {
    
        //创建监听的文件描述符,相当于酒店门口迎宾的人员
        //为客户提供服务由其他服务员来做
        int sockfd = socket(AF_INET, SOCK_STREAM, 0);
        
        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(struct sockaddr_in));//清空结构体 
        addr.sin_family = AF_INET;
        addr.sin_port = htons(port + i); //主机字节序转换为网络字节序 8888, 8889....8987
        addr.sin_addr.s_addr = INADDR_ANY;
        

        if(bind(sockfd, (struct sockaddr *)&addr, sizeof(struct sockaddr_in)) < 0)
        {
    
            perror("bind");
            return -2;
        }
        //利用sockfd进行监听
        //第二个参数指定能处理的最大连接请求
        if(listen(sockfd, 5) < 0)
        {
    
            perror("listen");
            return -3;
        }
        
        printf("tcp server listen on port: %d\n", port + i);

        struct epoll_event ev;

        //有数据到来socketfd,表示可读了,则会触发EPOLLIN
        //对端（客户端）读取了一些数据走,则表示可以往这个socketfd写了,则会触发EPOLLOUT
        ev.events = EPOLLIN;
        ev.data.fd = sockfd;

        //EPOLL_CTL_ADD:在epoll的监视列表中添加一个文件描述符（即参数fd）,指定监视的事件类型（参数event）
        epoll_ctl(epfd, EPOLL_CTL_ADD, sockfd, &ev);                           

        sockfds[i] = sockfd;
    }

Connection timed out 错误

描述

这个问题在ubuntu20.04上不存在,但还是记录一下.

原因与解决方案

首先查看file-maxWhether the parameter value is insufficient100w

file-max : sets the maximum number of file-handles that the Linux kernel will allocate.

[email protected]:~$ cat /proc/sys/fs/file-max
9223372036854775807

已经大于100w,So then checknf_conntrack_max(Set the maximum value for connection tracking)

[email protected]:~$ sudo modprobe ip_conntrack  #This sentence is used to startip_conntrack,If not added, the following statement fails
[email protected]:~$ cat /proc/sys/net/netfilter/nf_conntrack_max 
262144

Here we can see that the connection tracking maximum value is only262144,没有到100w,So the solution is to set that parameter to100w之上

sudo vim /etc/sysctl.conf

将net.nf_conntrack_max = 1048576add to config file

Then enter the following statement to take effect

sudo sysctl -p

Cannot open /proc/meminfo:Too many open files in system file-max

Explanation of the problemfile-max不够大,在etc/sysctl.conf修改参数即可

[email protected]:~$ sudo modprobe ip_conntrack  #This sentence is used to startip_conntrack,If not added, the following statement fails
[email protected]:~$ sudo vim /etc/sysctl.conf

添加下面的语句

Then enter the following statement effective configuration

sudo sysctl -p

Tuning of memory reclamation settings

The following configuration code is added toetc/sysctl.conf文件中,Be careful to addsudo modprobe ip_conntrack

1. 第一个参数：tcp协议栈内存

   net.ipv4.tcp_mem = 252144 524288 786432
   

   tcp_mem(3个INTEGER变量)：low, pressure, high

   low：当TCP使用了低于该值的内存页面数时,TCP不会考虑释放内存.
   pressure：当TCP使用了超过该值的内存页面数量时,TCP试图稳定其内存使用,进入pressure模式,当内存消耗低于low值时则退出pressure状态.
   high：允许所有tcp sockets用于排队缓冲数据报的页面量,when memory usage超过此值,系统拒绝分配socket,后台日志输出“TCP: too many of orphaned sockets”

2. 第二个参数：tcp发送缓冲区

   net.ipv4.tcp_wmem = 1024 1024 2048
   

   表示tcp连接(socket)的发送（write）minimum buffer size、默认、最大值,分别为1KB ,1KB,2KB

3. 第三个参数：tcp接收缓冲区

   net.ipv4_tcp_rmem = 1024, 1024, 2048
   

   表示tcp连接(socket)的接受（recive）minimum buffer size、默认、最大值,分别为1KB ,1KB,2KB

这样设置之后,默认的1个socket的file descriptor大小为2KB（rmem + wmem）,百万个fd就大概是2g内存.

杂项

• 最后测试的时候,My configuration can only go up to82万左右,不知道什么原因,Also increased the memory of the client and server
• When a large number of clients are disconnected and down,Processor usage will soar
• 内存和CPUUtilization is best maintained at80%一下

注意

The main purpose of this experiment is to understand the optimizations required for millions of concurrent,So the code is not given in detail.如果面试时候,If anyone asks about this server tuning,Tell the interviewer what problems you have solved,Don't fail to reach100wjust yes,不敢回答.