Network learning (III) -- highly concurrent socket programming (epoll)

Catalog

One 、 introduction

Two 、 High concurrency background

------> 2.1、select

3、 ... and 、epoll

------> 3.1、 Functional separation

------> 3.2、 Ready list

------> 3.3、 Code explanation

------> 3.4、 Process summary

One 、 introduction

The previous chapter talked about socket Programming for , This chapter introduces the relevant mechanisms of high concurrency servers
The original text refers to this article to a large extent Epoll Principle analysis

Two 、 High concurrency background

Let's review the last chapter socket Programming , The link between the host and the server is successful , After starting communication , Host calls recv() To get data , When the program runs to Recv when , It will always wait , Do not execute until data is received

One Socket It corresponds to a port number , And the network packet contains IP And port information , The kernel can find the corresponding... Through the port number Socket.
that , How to monitor multiple Socket Well ？ This is what this chapter is about

The server needs to manage multiple client connections , and Recv You can only monitor a single Socket, In this contradiction , People began to look for surveillance of multiple Socket Methods .Epoll The point is to effectively monitor multiple Socket. And in the epoll Before , What has been used is select and pol Mechanism , The two are very similar , That's all select 了

1、select

In the code below , Prepare an array first FDS, Give Way FDS Store all the things that need to be monitored Socket.
And then call Select, If FDS All in Socket No data ,Select It will block , Until there is one Socket Data received ,Select return , Wake-up process .
Users can traverse FDS, adopt FD_ISSET Decide which one Socket Receive the data , And then deal with it .

int s = socket(AF_INET, SOCK_STREAM, 0);   
bind(s, ...) 
listen(s, ...) 
 
int fds[] =   Store what needs to be monitored socket 
 
while(1){
     
    int n = select(..., fds, ...) 
    for(int i=0; i < fds.count; i++){
     
        if(FD_ISSET(fds[i], ...)){
     
            //fds[i] Data processing of  
        } 
    } 
} 

Select The process of

Select It's very direct , If the program simultaneously monitors Sock1、Sock2 and Sock3 Three Socket, So in calling Select after
1、 The operating system puts the process A Add these three Socket Waiting in the queue .
2、 When any one Socket After receiving the data , Interrupting the program will call up the process . The so-called arousal process , Is to remove the process from all the waiting queues , Join the work queue
3、 Through these steps , When a process A After being awakened , It knows at least one Socket Received data . The program only needs to traverse once Socket list , You can get ready Socket.

select The shortcomings of

This simple way works , In almost all operating systems there are corresponding implementations . But simple methods often have disadvantages , Mainly ：
1、 Secondary call Select All processes need to be added to all monitoring Socket Waiting queue , Every wake-up needs to be removed from each queue . This involves two iterations , And every time I have to put the whole FDS List passed to kernel , There is a certain cost . It's because of the overhead of traversal , For the sake of efficiency , Will stipulate that Select The maximum number of monitors , By default, you can only monitor 1024 individual Socket.
2、 After the process is awakened , The program doesn't know what Socket Receive the data , You need to go through it again .

In order to solve these shortcomings , It's introduced epoll Mechanism

3、 ... and 、epoll

Epoll Is in Select appear N It was invented years later , yes Select and Poll(Poll and Select Is essentially the same , There are a few improvements ) Enhanced version of .Epoll Improve efficiency through the following measures ：

1、 Functional separation

Select One of the reasons for the inefficiency is to “ Maintain waiting queues ” and “ Blocking process ” The two steps merge into one .

comparison Select,Epoll Split the function

As shown in the figure above , Every time you call Select Both of these two steps are needed , However, in most application scenarios , Need to be monitored Socket Relatively fixed , It doesn't need to be changed every time .

Epoll Separate these two operations , First use epoll_ctl Maintain waiting queues , Call again epoll_wait Blocking process . Obviously , Efficiency can be improved .

For the convenience of understanding the following content , Let's get to know Epoll Usage of . In the following code , First use epoll_create Create a Epoll object Epfd, Re pass epoll_ctl Will need to be monitored Socket Add to Epfd in , Last call epoll_wait Waiting for data ：

int s = socket(AF_INET, SOCK_STREAM, 0);    
bind(s, ...) 
listen(s, ...) 
 
int epfd = epoll_create(...); 
epoll_ctl(epfd, ...); // All that needs to be monitored socket Add to epfd in  
 
while(1){
     
    int n = epoll_wait(...) 
    for( Receiving data socket){
     
        // Handle  
    } 
} 

2、 Ready list

Select Another reason for inefficiency is that the program doesn't know what Socket Receive the data , It can only be traversed one by one . If the kernel maintains a “ Ready list ”, Quote... From received data Socket, You can avoid traversal .

When a process calls epoll_create When the method is used , The kernel will create a eventpoll object ( In the process Epfd Represented by ).

establish Epoll After the object , It can be used epoll_ctl Add or remove the Socket. If you pass epoll_ctl add to Sock1、Sock2 and Sock3 Surveillance , The kernel will eventpoll Add to these three Socket Waiting in the queue .

When Socket After receiving the data , The interrupt program will operate eventpoll object , Instead of operating the process directly , At the same time, interrupting the program will give eventpoll Of “ Ready list ” add to Socket quote .

eventpoll The object is equal to Socket And the intermediary between the process ,Socket Data receiving does not directly affect the process , But by changing eventpoll To change the state of the process .

When the program is executed to epoll_wait when , If Rdlist Has quoted Socket, that epoll_wait Go straight back to , If Rdlist It's empty , Blocking process .

epoll_wait When blocking a process, the kernel will block the process A Put in eventpoll Waiting in the queue , Blocking process .

When Socket Data received , On the one hand, the interrupt program is modified Rdlist, On the other hand, wake up eventpoll Wait for the process in the queue , process A Enter the operation state again
Epoll Wake up process also because Rdlist The existence of , process A You can know what Socket There is a change .

3、 Code explanation

int main(int argc, char* argv[])
{
    
    int i, maxi, listenfd, connfd, sockfd,epfd,nfds, portnumber;
    ssize_t n;
    char line[MAXLINE];
    socklen_t clilen;

......
    // Statement epoll_event Structural variables ,ev Used to register Events , Arrays are used to return events to be processed 
    struct epoll_event ev,events[20];
    
    // Generate for processing accept Of epoll Dedicated file descriptors 
    epfd=epoll_create(256);
    
    struct sockaddr_in clientaddr;
    struct sockaddr_in serveraddr;
    listenfd = socket(AF_INET, SOCK_STREAM, 0);

    // Set the file descriptor associated with the event to be processed 
    ev.data.fd=listenfd;
    
    // Set the type of event to process 
    ev.events=EPOLLIN|EPOLLET;
    //ev.events=EPOLLIN;

    // register epoll event 
    epoll_ctl(epfd,EPOLL_CTL_ADD,listenfd,&ev);
    bzero(&serveraddr, sizeof(serveraddr));
    serveraddr.sin_family = AF_INET;
    char *local_addr="127.0.0.1";
    inet_aton(local_addr,&(serveraddr.sin_addr));//htons(portnumber);

    serveraddr.sin_port=htons(portnumber);
    bind(listenfd,(sockaddr *)&serveraddr, sizeof(serveraddr));
    listen(listenfd, LISTENQ);
    maxi = 0;
    for ( ; ; ) {
    
        // wait for epoll The occurrence of the incident 

        nfds=epoll_wait(epfd,events,20,500);
        // Deal with all the events that happen 

        for(i=0;i<nfds;++i)
        {
    
            if(events[i].data.fd==listenfd)// If a new one is detected SOCKET The user is connected to the bound SOCKET port , Establish a new connection .
            {
    
                connfd = accept(listenfd,(sockaddr *)&clientaddr, &clilen);
                if(connfd<0){
    
                    perror("connfd<0");
                    exit(1);
                }
                //setnonblocking(connfd);

                char *str = inet_ntoa(clientaddr.sin_addr);
                cout << "accapt a connection from " << str << endl;
                // Set the file descriptor for the read operation 

                ev.data.fd=connfd;
                // Set the read operation event used for annotation test 

                ev.events=EPOLLIN|EPOLLET;
                //ev.events=EPOLLIN;

                // register ev
                epoll_ctl(epfd,EPOLL_CTL_ADD,connfd,&ev);
            }
            else if(events[i].events&EPOLLIN)// If it's a connected user , And receive data , So read in .
            {
    
                cout << "EPOLLIN" << endl;
                if ( (sockfd = events[i].data.fd) < 0)
                    continue;
                if ( (n = read(sockfd, line, MAXLINE)) < 0) {
    
                    if (errno == ECONNRESET) {
    
                        close(sockfd);
                        events[i].data.fd = -1;
                    } else
                        std::cout<<"readline error"<<std::endl;
                } else if (n == 0) {
    
                    close(sockfd);
                    events[i].data.fd = -1;
                }
                line[n] = '/0';
                cout << "read " << line << endl;
                // Set file descriptors for write operations 

                ev.data.fd=sockfd;
                // Set the write operation event for annotation test 

                ev.events=EPOLLOUT|EPOLLET;
                // modify sockfd The event to be handled on is EPOLLOUT

                //epoll_ctl(epfd,EPOLL_CTL_MOD,sockfd,&ev);
            }
            else if(events[i].events&EPOLLOUT) //  If there's data to send 
            {
    
                sockfd = events[i].data.fd;
                write(sockfd, line, n);
                // Set the file descriptor for the read operation 

                ev.data.fd=sockfd;
                // Set the read operation event used for annotation test 

                ev.events=EPOLLIN|EPOLLET;
                // modify sockfd The event to be handled on is EPOLIN

                epoll_ctl(epfd,EPOLL_CTL_MOD,sockfd,&ev);
            }
        }
    }
    return 0;
}

4、 Process summary

The main process is as follows
1、epoll_create Create for processing accept Of epoll Dedicated file descriptors
2、 Create a listener socket, Put it in the epoll_event File descriptor to be processed in , And fill in Event type , among EPOLLLT and EPOLLET The difference between
3、 after bind、listen Start listening for connection sockets
4、epoll_wait First judge epoll_event File descriptors in , If it is a listening descriptor , It means that a new connection has occurred ,accept A new socket , Refill epoll_event , Reuse epoll_ctl Sign up to epoll In the descriptor of
5、epoll_wait If it's a connected user , Judge epoll_wait type (EPOLLIN/EPOLLOUT), Respectively called read、write