Core principle and code explanation of epoll reactor model

[Ⅰ] Epoll Principle and Application && ET Patterns and LT Pattern

The first part of the article links ： Epoll Principle and Application && ET Patterns and LT Pattern

[Ⅱ] Epoll Reactor model core principle and code explanation

One 、 Reactor core principles

epoll The three elements of the reactor model ：

• epoll ET Pattern

• Non blocking polling processing

• struct epoll_event In the structure epoll_data_t In the consortium void *ptr The pointer – Implement callback mechanism

  Structure review ：

  【 Important to understand 】 The struct epoll_event Structure can be understood as （ adopt epoll——ctl()） Mounted to the kernel epoll Listen to the structure on the red black tree （ A mechanism similar to deep copy ）：

  struct epoll_event {
         
  	__uint32_t events; /* Epoll events */
  	epoll_data_t data; /* User data variable */
  };
  
  typedef union epoll_data {
         
  	void *ptr;
  	int fd;				//  The fd It's the introduction epoll_ctl() Of the corresponding listening event fd
  	uint32_t u32;
  	uint64_t u64;
  } epoll_data_t;
  

  A consortium is also called a consortium , The variables in the union share an address space .

  In the most basic use , The value put into the consortium is fd, The following example pseudocode ：

  /* int connfd  yes accpt() Back to socket Connection handle  */
  struct epoll_event event = {
         0, {
         0}};
  event.events = EPOLLIN;
  event.data.fd = connfd;
  epoll_ctl(efd, EPOLL_CTL_ADD, connfd, &event);
      
  ......  //  Business logic 
      
  while (1) {
         
      /* Monitor the red black tree efd,  Add the event descriptor to the file events Array ,  Blocking wait*/
      int nfd = epoll_wait(efd, events, MAX_EVENTS+1, -1);
  
      for (i = 0; i < nfd; i++) {
         
  		/*  Use int type ,  Receiving Consortium data Of fd member  */
  		int readyfd = events[i].data.fd;  
      
   		...... //  Business logic 
       
  	}
  }
  

  but The reactor model is not directly put into fd, Instead, put in a custom structure pointer （ The forced conversion became void * type ）, such epoll_wait() When you return, you can retrieve the user-defined structure stored before .

  /*  User defined structure  */
  /*  Describes information about ready file descriptors  */
  struct myevent_s {
         
      int fd;                                                 // The file descriptor to listen to 
      int events;                                             // The corresponding listening event 
      void *arg;                                              // The generic parameter 
      void (*call_back)(int fd, int events, void *arg);       // Callback function 
  
      int status;                                             // Is it listening :1-> On the red and black trees ( monitor ), 0-> be not in ( No monitoring )
      char buf[BUFLEN];
      int len;
      long last_active;                                       // Record every time you add a red black tree  g_efd  Time value of 
  };
  
  ...... //  Business logic 
  
  /* struct myevent_s *ev  It is a user-defined structure  */
  struct epoll_event epv = {
         0, {
         0}};
  epv.events = ev->events = EPOLLIN;   		//EPOLLIN  or  EPOLLOUT
  epv.data.ptr = ev;							//  Note that this is not epv.data.fd = connfd
  epoll_ctl(efd, EPOLL_CTL_ADD, ev->fd, &epv)
      
  ...... //  Business logic 
      
  while(1) {
         
      
  /* Monitor the red black tree g_efd,  Add the event descriptor to the file events Array , 1 Second no event satisfies ,  return  0*/
  	int nfd = epoll_wait(g_efd, events, MAX_EVENTS+1, 1000);
  
  	...... // Error handling 
      
  	for (i = 0; i < nfd; i++) {
         
  		/* Use custom structs myevent_s Type a pointer ,  receive   Consortium data Of void *ptr member */
  		struct myevent_s *ev = (struct myevent_s *)events[i].data.ptr;  
      
   		...... //  Business logic 
       
  	}
  }
  

  Implementation of callback mechanism ：

  Store pointer functions in a custom structure ,epoll_wait Take out after returning events[i].data.ptr Point to a custom structure , then Call the callback function stored in the structure ：

  struct myevent_s *ev = (struct myevent_s *)events[i].data.ptr;
  ...... // Judge 
  ev->call_back(ev->fd, events[i].events, ev->arg);
  

Two 、 Reactor model example

2.1 The whole logic

• socket、bind、listen – epoll_create Create a listener Red and black trees – return epfd

• epoll_ctl() Add a... To the red and black tree listenfd（ monitor socket）

• while(1) {

  • 【 Optional 】 Monitor each iteration 100 A connection , If there is a timeout connection （ Deposition users ） Then it will be closed automatically ;

  • epoll_wait() monitor --> Corresponding to listening fd Something happened --> Return listening satisfaction structure set （ namely struct epoll_event Array of structs ）;

  • Determine whether to return array elements :

    • lfd Satisfy EPOLLIN event （ Read events ） --> Callback acceptconn() function （ Mainly complete accept() The task of ）

    acceptconn() function ：

    • call accept() Accept new connection , Turn on cfd socket, Set to non blocking ;
    • From the global custom structure array struct myevent_s g_events[MAX_EVENTS+1] Find a free element in ev ;
    • call eventset() Function will cfd and Callback function senddata write in ev in ;
    • call eventadd() take Overall &ev As struct epoll_event Structure of the data Of the consortium void *ptr The pointer , Set listening EPOLLIN（ Read events ）, Mount to epoll On the red and black trees .

    • cfd Satisfy EPOLLIN event （ Read events ） --> Callback recvdata() function （ It mainly completes the read operation ）

    recvdata() function ：

    1. call epoll_ctl() And macros EPOLL_CTL_DEL take cfd Pluck from the red and black trees ;
    2. Process input , And save the processing structure to the user-defined structure char buf[BUFLEN] In the member variable ;
    3. Change the listening event to EPOLLOUT, change cfd The corresponding callback function is senddata()
    4. call epoll_ctl() And macros EPOLL_CTL_ADD take cfd Re mount to the red black tree ;

    ---

    Be careful ：

    Strictly speaking ,write Also confirm whether it is writable through the listening mechanism , Because in the actual network environment , If the opposite end of the communication is half closed , Or the sliding window is full , You can't make it wtrite data .

    ---

    • cfd Satisfy EPOLLOUT event （ Write events ）–> Callback senddata() function （ It mainly completes the write operation ）

    senddata() function ：

    1. call epoll_ctl() And macros EPOLL_CTL_DEL take cfd Pluck from the red and black trees ;
    2. Save to... In the user-defined structure char buf[BUFLEN] Copy the data to cfd Send buffer for （ namely write()/send() function ）;
    3. Change the listening event to EPOLLIN, change cfd The corresponding callback function is recvdata()
    4. call epoll_ctl() And macros EPOLL_CTL_ADD take cfd Re mount to the red black tree ;

  }// while(1) end

2.2 Important functions

① eventset() function

Statement ：void eventset(struct myevent_s *ev, int fd, void (*call_back)(int, int, void *), void *arg)

function ： The structure will be customized myevent_s Member variables initialization

Invoke the sample ：

1. eventset(&g_events[MAX_EVENTS], lfd, acceptconn, &g_events[MAX_EVENTS]);
2. eventset(ev, fd, recvdata, ev);
3. eventset(ev, fd, senddata, ev);

void eventset(struct myevent_s *ev, int fd, void (*call_back)(int, int, void *), void *arg)
{
    
    ev->fd = fd;				//  To be monitored fd
    ev->call_back = call_back;	//  Set the callback function 
    ev->events = 0;				//  The listening event is controlled by the function  eventadd() The function specifies 
    ev->arg = arg;				// ev Of arg namely ev Parameters of the callback function in the , yes ev In itself （ It's hard to understand , But this is the key ）
    ev->status = 0;				//  The structure status is marked as " Occupied "
    memset(ev->buf, 0, sizeof(ev->buf));	// Clear the... Of the structure char * buffer 
    ev->len = 0;				//  Set the buffer data length to 0
    ev->last_active = time(NULL); // call eventset Time of function （ Optional , Used to disconnect the sediment connection ）
    return;
}

② eventadd() function

Statement ：void eventadd(int efd, int events, struct myevent_s *ev)

function ： towards epoll Add a listening node to the listening red black tree

Invoke the sample ：

1. eventadd(g_efd, EPOLLIN, &g_events[i]);
2. eventadd(g_efd, EPOLLIN, ev);
3. eventadd(g_efd, EPOLLOUT, ev);

/* eventadd(efd, EPOLLIN, &g_events[MAX_EVENTS]); */
void eventadd(int efd, int events, struct myevent_s *ev)
{
	/*  Pointer from custom structure struct myevent_s * The variable of ev in   Extract data to one that can be hung to epoll Listen to the... On the red and black tree struct epoll_event Variable  epv On  */ 
    struct epoll_event epv = {0, {0}};
    int op;
    
    epv.data.ptr = ev;
    epv.events = ev->events = events;  //EPOLLIN  or  EPOLLOUT

    if (ev->status == 0) {     // Already in the red and black tree  g_efd  in 
        op = EPOLL_CTL_ADD;    // Add it to the red black tree  g_efd,  And will status Set up 1
        ev->status = 1;
    }

    if (epoll_ctl(efd, op, ev->fd, &epv) < 0)  	// Actual addition 
        printf("event add failed [fd=%d], events[%d]\n", ev->fd, events);
    else
        printf("event add OK [fd=%d], op=%d, events[%0X]\n", ev->fd, op, events);

    return ;
}

③ eventdel() function

Statement ：void eventadd(int efd, int events, struct myevent_s *ev)

function ： from epoll Remove a listening node from the listening red black tree

Invoke the sample ：eventdel(g_efd, ev);

void eventdel(int efd, struct myevent_s *ev){
    
    // Not on the black and red tree 
    if (ev->status != 1) 	return;
    
	// modify state 
    ev->status = 0;       
    
    // From the red and black trees  efd  Admiral  ev->fd  Enucleation 
    epoll_ctl(efd, EPOLL_CTL_DEL, ev->fd, NULL);                
    
    return ;
}

④ acceptconn() function

Statement ：void acceptconn(int lfd, int events, void *arg)

framework ：

• call accept() Accept new connection , Turn on cfd socket, Set to non blocking ;
• From the global custom structure array struct myevent_s g_events[MAX_EVENTS+1] Find a free element in ev ;
• call eventset() Function will cfd and Callback function senddata write in ev in ;
• call eventadd() take Overall &ev As struct epoll_event Structure of the data Of the consortium void *ptr The pointer , Set listening EPOLLIN（ Read events ）, Mount to epoll On the red and black trees .

/*  When lfd The read event for is ready , epoll return ,  Call this function   Link with client  */
/*  stay acceptconn Do it internally accept */ 
void acceptconn(int lfd, int events, void *arg)
{
    
    struct sockaddr_in cin;
    socklen_t len = sizeof(cin);
    int cfd, i;

    if ((cfd = accept(lfd, (struct sockaddr *)&cin, &len)) == -1) {
    
        if (errno != EAGAIN && errno != EINTR) {
    
            /*  No error handling for the time being  */
        }
        printf("%s: accept, %s\n", __func__, strerror(errno));
        return ;
    }

    do {
    
        for (i = 0; i < MAX_EVENTS; i++)                                // From global array g_events Find a free element in 
            if (g_events[i].status == 0)                                // Be similar to select The median value is -1 The elements of 
                break;                                                  // Jump out of  for

        if (i == MAX_EVENTS) {
    
            printf("%s: max connect limit[%d]\n", __func__, MAX_EVENTS);
            break;                                                      // Jump out of do while(0)  Do not execute subsequent code 
        }

        int flag = 0;
        if ((flag = fcntl(cfd, F_SETFL, O_NONBLOCK)) < 0) {
                 // take cfd Also set to non blocking 
            printf("%s: fcntl nonblocking failed, %s\n", __func__, strerror(errno));
            break;
        }

        /*  to cfd Set up a  myevent_s  Structure ,  Callback function   Set to  recvdata */
        eventset(&g_events[i], cfd, recvdata, &g_events[i]);   
        // take cfd Add to the red black tree g_efd in , Listen to read events 
        eventadd(g_efd, EPOLLIN, &g_events[i]);                      
    } while(0);

    printf("new connect [%s:%d][time:%ld], pos[%d]\n", 
            inet_ntoa(cin.sin_addr), ntohs(cin.sin_port), g_events[i].last_active, i);
    return ;
}

⑤ recvdata() function

Statement ：void recvdata(int fd, int events, void *arg)

framework ：

1. call epoll_ctl() And macros EPOLL_CTL_DEL take cfd Pluck from the red and black trees ;
2. Process input , And save the processing structure to the user-defined structure char buf[BUFLEN] In the member variable ;
3. Change the listening event to EPOLLOUT, change cfd The corresponding callback function is senddata()
4. call epoll_ctl() And macros EPOLL_CTL_ADD take cfd Re mount to the red black tree ;

void recvdata(int fd, int events, void *arg)
{
    
    struct myevent_s *ev = (struct myevent_s *)arg;
    int len;
    
    // Remove the node from the red and black tree 
    eventdel(g_efd, ev);        
    
	 // Read file descriptor ,  Data stored in myevent_s member buf in 
    len = recv(fd, ev->buf, sizeof(ev->buf), 0);           

    if (len > 0) {
    
        ev->len = len;
        // Manually add the string end tag to avoid buffer overflow 
        ev->buf[len] = '\0';                               
        printf("C[%d]:%s\n", fd, ev->buf);
        
		// Set the  fd  The corresponding callback function is  senddata
        eventset(ev, fd, senddata, ev); 
         // take fd Join the red and black trees g_efd in , Listen to its write events 
        eventadd(g_efd, EPOLLOUT, ev);                     

    } else if (len == 0) {
    
        close(ev->fd);
        /* ev-g_events  The offset element position is obtained by subtracting the address  */
        printf("[fd=%d] pos[%ld], closed\n", fd, ev-g_events);
    } else {
    
        close(ev->fd);
        printf("recv[fd=%d] error[%d]:%s\n", fd, errno, strerror(errno));
    }

    return; 
    /*  If used with thread pool , The expectation is that the thread will return to the thread pool after it finishes the task , Instead of being recycled by the system , So this part of the thread cannot set the separation property  */ 
}

⑥ senddata() function

Statement ：void senddata(int fd, int events, void *arg)

framework ：

1. call epoll_ctl() And macros EPOLL_CTL_DEL take cfd Pluck from the red and black trees ;
2. Save to... In the user-defined structure char buf[BUFLEN] Copy the data to cfd Send buffer for （ namely write()/send() function ）;
3. Change the listening event to EPOLLIN, change cfd The corresponding callback function is recvdata()
4. call epoll_ctl() And macros EPOLL_CTL_ADD take cfd Re mount to the red black tree ;

void senddata(int fd, int events, void *arg)
{
    
    struct myevent_s *ev = (struct myevent_s *)arg;
    int len;
    // From the red and black trees g_efd Remove 
    eventdel(g_efd, ev); 
	// Directly transfer data   Write back to the client . Not processed 
    len = send(fd, ev->buf, ev->len, 0);    
	   
    if (len > 0) {
    
        printf("send[fd=%d], [%d]%s\n", fd, len, ev->buf);
        // Will be fd Of   The callback function is changed to  recvdata
        eventset(ev, fd, recvdata, ev);   
        // From the new add to the red and black tree ,  Set to listen for read events 
        eventadd(g_efd, EPOLLIN, ev);                       
    } else {
    
        close(ev->fd);                                      // Close links 
        printf("send[fd=%d] error %s\n", fd, strerror(errno));
    }

    return ;
}

2.3 Sample source code （Server End ）

/* *epoll Based on non blocking I/O Event driven  */
#include <stdio.h>
#include <sys/socket.h>
#include <sys/epoll.h>
#include <arpa/inet.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>

#define MAX_EVENTS 1024 // The maximum number of listeners 
#define BUFLEN 4096
#define SERV_PORT 8080

void recvdata(int fd, int events, void *arg);
void senddata(int fd, int events, void *arg);

/*  Describes information about ready file descriptors  */

struct myevent_s {
    
    int fd;                                                 // The file descriptor to listen to 
    int events;                                             // The corresponding listening event 
    void *arg;                                              // The generic parameter 
    void (*call_back)(int fd, int events, void *arg);       // Callback function 
    int status;                                             // Is it listening :1-> On the red and black trees ( monitor ), 0-> be not in ( No monitoring )
    char buf[BUFLEN];
    int len;
    long last_active;                                       // Record every time you add a red black tree  g_efd  Time value of 
};

int g_efd;                                                  // Global variables ,  preservation epoll_create File descriptor returned 
struct myevent_s g_events[MAX_EVENTS+1];                    // Custom structure type array . +1-->listen fd


/* The structure  myevent_s  Member variables   initialization */
/* eventset(&g_events[MAX_EVENTS], lfd, acceptconn, &g_events[MAX_EVENTS]); */ 
void eventset(struct myevent_s *ev, int fd, void (*call_back)(int, int, void *), void *arg)
{
    
    ev->fd = fd;
    ev->call_back = call_back;
    ev->events = 0;
    ev->arg = arg;
    ev->status = 0;
    memset(ev->buf, 0, sizeof(ev->buf));
    ev->len = 0;
    ev->last_active = time(NULL);                       // call eventset Time of function 

    return;
}

/*  towards  epoll Monitoring the red and black tree   Add one   File descriptor  */
/* eventadd(efd, EPOLLIN, &g_events[MAX_EVENTS]); */
void eventadd(int efd, int events, struct myevent_s *ev)
{
    
	/*  Pointer from custom structure struct myevent_s * The variable of ev in   Extract data to one that can be hung to epoll Listen to the... On the red and black tree struct epoll_event Variable  epv On  */ 
    struct epoll_event epv = {
    0, {
    0}};
    int op;
    epv.data.ptr = ev;
    epv.events = ev->events = events;       //EPOLLIN  or  EPOLLOUT

    if (ev->status == 0) {
                                              // Already in the red and black tree  g_efd  in 
        op = EPOLL_CTL_ADD;                 // Add it to the red black tree  g_efd,  And will status Set up 1
        ev->status = 1;
    }

    if (epoll_ctl(efd, op, ev->fd, &epv) < 0)                       // Actual addition / modify 
        printf("event add failed [fd=%d], events[%d]\n", ev->fd, events);
    else
        printf("event add OK [fd=%d], op=%d, events[%0X]\n", ev->fd, op, events);

    return ;
}

/*  from epoll  Monitoring   Delete one from the red and black tree   File descriptor */

void eventdel(int efd, struct myevent_s *ev)
{
    
    struct epoll_event epv = {
    0, {
    0}};

    if (ev->status != 1)                                        // Not on the black and red tree 
        return ;

    //epv.data.ptr = ev;
    epv.data.ptr = NULL;										// Erase the pointer  
    ev->status = 0;                                             // modify state 
    epoll_ctl(efd, EPOLL_CTL_DEL, ev->fd, &epv);                // From the red and black trees  efd  Admiral  ev->fd  Enucleation 

    return ;
}

/*  When a file descriptor is ready , epoll return ,  Call this function   Link with client  */
/*  stay acceptconn Do it internally accept */ 
void acceptconn(int lfd, int events, void *arg)
{
    
    struct sockaddr_in cin;
    socklen_t len = sizeof(cin);
    int cfd, i;

    if ((cfd = accept(lfd, (struct sockaddr *)&cin, &len)) == -1) {
    
        if (errno != EAGAIN && errno != EINTR) {
    
            /*  No error handling for the time being  */
        }
        printf("%s: accept, %s\n", __func__, strerror(errno));
        return ;
    }

    do {
    
        for (i = 0; i < MAX_EVENTS; i++)                                // From global array g_events Find a free element in 
            if (g_events[i].status == 0)                                // Be similar to select The median value is -1 The elements of 
                break;                                                  // Jump out of  for

        if (i == MAX_EVENTS) {
    
            printf("%s: max connect limit[%d]\n", __func__, MAX_EVENTS);
            break;                                                      // Jump out of do while(0)  Do not execute subsequent code 
        }

        int flag = 0;
        if ((flag = fcntl(cfd, F_SETFL, O_NONBLOCK)) < 0) {
                 // take cfd Also set to non blocking 
            printf("%s: fcntl nonblocking failed, %s\n", __func__, strerror(errno));
            break;
        }

        /*  to cfd Set up a  myevent_s  Structure ,  Callback function   Set to  recvdata */
        eventset(&g_events[i], cfd, recvdata, &g_events[i]);   
        eventadd(g_efd, EPOLLIN, &g_events[i]);                         // take cfd Add to the red black tree g_efd in , Listen to read events 

    } while(0);

    printf("new connect [%s:%d][time:%ld], pos[%d]\n", 
            inet_ntoa(cin.sin_addr), ntohs(cin.sin_port), g_events[i].last_active, i);
    return ;
}

void recvdata(int fd, int events, void *arg)
{
    
    struct myevent_s *ev = (struct myevent_s *)arg;
    int len;

    len = recv(fd, ev->buf, sizeof(ev->buf), 0);            // Read file descriptor ,  Data stored in myevent_s member buf in 

    eventdel(g_efd, ev);        // Remove the node from the red and black tree 

    if (len > 0) {
    

        ev->len = len;
        ev->buf[len] = '\0';                                // Add string end tag manually 
        printf("C[%d]:%s\n", fd, ev->buf);

        eventset(ev, fd, senddata, ev);                     // Set the  fd  The corresponding callback function is  senddata
        eventadd(g_efd, EPOLLOUT, ev);                      // take fd Join the red and black trees g_efd in , Listen to its write events 

    } else if (len == 0) {
    
        close(ev->fd);
        /* ev-g_events  The offset element position is obtained by subtracting the address  */
        printf("[fd=%d] pos[%ld], closed\n", fd, ev-g_events);
    } else {
    
        close(ev->fd);
        printf("recv[fd=%d] error[%d]:%s\n", fd, errno, strerror(errno));
    }

    return; /*  The expectation is that the thread will return to the thread pool after it finishes the task , Instead of being recycled by the system , So this part of the thread cannot set the separation property  */ 
}

void senddata(int fd, int events, void *arg)
{
    
    struct myevent_s *ev = (struct myevent_s *)arg;
    int len;

    len = send(fd, ev->buf, ev->len, 0);                    // Directly transfer data   Write back to the client . Not processed 

    eventdel(g_efd, ev);                                // From the red and black trees g_efd Remove 

    if (len > 0) {
    

        printf("send[fd=%d], [%d]%s\n", fd, len, ev->buf);
        eventset(ev, fd, recvdata, ev);                     // Will be fd Of   The callback function is changed to  recvdata
        eventadd(g_efd, EPOLLIN, ev);                       // From the new add to the red and black tree ,  Set to listen for read events 

    } else {
    
        close(ev->fd);                                      // Close links 
        printf("send[fd=%d] error %s\n", fd, strerror(errno));
    }

    return ;
}

/* establish  socket,  initialization lfd */

void initlistensocket(int efd, short port)
{
    
    struct sockaddr_in sin;

    int lfd = socket(AF_INET, SOCK_STREAM, 0);
    fcntl(lfd, F_SETFL, O_NONBLOCK);                                            // take socket Set to non blocking 

	memset(&sin, 0, sizeof(sin));                                               //bzero(&sin, sizeof(sin))
	sin.sin_family = AF_INET;
	sin.sin_addr.s_addr = INADDR_ANY;
	sin.sin_port = htons(port);

	bind(lfd, (struct sockaddr *)&sin, sizeof(sin));

	listen(lfd, 20);

    /* void eventset(struct myevent_s *ev, int fd, void (*call_back)(int, int, void *), void *arg); */
    eventset(&g_events[MAX_EVENTS], lfd, acceptconn, &g_events[MAX_EVENTS]);

    /* void eventadd(int efd, int events, struct myevent_s *ev) */
    eventadd(efd, EPOLLIN, &g_events[MAX_EVENTS]);

    return ;
}

int main(int argc, char *argv[])
{
    
    unsigned short port = SERV_PORT;

    if (argc == 2)
        port = atoi(argv[1]);                           // Use user specified port . If not specified , Use default port 

    g_efd = epoll_create(MAX_EVENTS+1);                 // Create a red black tree , Back to the big picture  g_efd 
    if (g_efd <= 0)
        printf("create efd in %s err %s\n", __func__, strerror(errno));

    initlistensocket(g_efd, port);                      // Initialize listening socket

    struct epoll_event events[MAX_EVENTS+1];            // Save an array of file descriptors that have satisfied the ready event   In order to offer epoll_wait Use  
	printf("server running:port[%d]\n", port);

    int checkpos = 0, i;
    while (1) {
    
        /*  Timeout verification , Every test 100 A link , Don't test listenfd  When the client 60 No communication with the server in seconds , Then close the client link  */

        long now = time(NULL);                          // current time 
        for (i = 0; i < 100; i++, checkpos++) {
             // Primary cycle detection 100 individual .  Use checkpos Control test object 
            if (checkpos == MAX_EVENTS)					// The root node does not participate in the detection  
                checkpos = 0;
            if (g_events[checkpos].status != 1)         // Not in the red and black trees  g_efd  On 
                continue;

            long duration = now - g_events[checkpos].last_active;       // The world where the client is not active 

            if (duration >= 60) {
    
                close(g_events[checkpos].fd);                           // Close the link with the client 
                printf("[fd=%d] timeout\n", g_events[checkpos].fd);
                eventdel(g_efd, &g_events[checkpos]);                   // Put the client   From the red and black trees  g_efd remove 
            }
        }

        /* Monitor the red black tree g_efd,  Add the event descriptor to the file events Array , 1 Second no event satisfies ,  return  0*/
        int nfd = epoll_wait(g_efd, events, MAX_EVENTS+1, 1000);
        if (nfd < 0) {
    
            printf("epoll_wait error, exit\n");
            break;
        }


        for (i = 0; i < nfd; i++) {
    
            /* Use custom structs myevent_s Type a pointer ,  receive   Consortium data Of void *ptr member */
            struct myevent_s *ev = (struct myevent_s *)events[i].data.ptr;  

            if ((events[i].events & EPOLLIN) && (ev->events & EPOLLIN)) {
               // Read ready event 
                ev->call_back(ev->fd, events[i].events, ev->arg);
                //lfd EPOLLIN 
            }
            if ((events[i].events & EPOLLOUT) && (ev->events & EPOLLOUT)) {
             // Write ready event 
                ev->call_back(ev->fd, events[i].events, ev->arg);
            }
        }
    }

    /*  Release all resources before exiting  */
    return 0;
}