connection reset by peer

1、connection reset by peer?

From the perspective of application layer logs , It's almost impossible to be sure connection reset by peer The underlying reason . therefore , We started the work of capturing bags . The specific way is ：

• We need to choose one end to do bag grabbing , This is the client ;
• Check the application log , Found that it appeared in a few minutes connection reset by peer The error of ;
• Compare the error log with the packet capture file , Look for clues .

Let's take a look first , What do these error logs look like ：

2015/12/01 15:49:48 [info] 20521#0: *55077498 recv() failed (104: Connection reset by peer) while sending to client, client: 10.255.252.31, server: manager.example.com, request: "POST /WebPageAlipay/weixin/notify_url.htm HTTP/1.1", upstream: "http:/10.4.36.207:8080/WebPageAlipay/weixin/notify_url.htm", host: "manager.example.com"
2015/12/01 15:49:54 [info] 20523#0: *55077722 recv() failed (104: Connection reset by peer) while sending to client, client: 10.255.252.31, server: manager.example.com, request: "POST /WebPageAlipay/app/notify_url.htm HTTP/1.1", upstream: "http:/10.4.36.207:8080/WebPageAlipay/app/notify_url.htm", host: "manager.example.com"
2015/12/01 15:49:54 [info] 20523#0: *55077710 recv() failed (104: Connection reset by peer) while sending to client, client: 10.255.252.31, server: manager.example.com, request: "POST /WebPageAlipay/app/notify_url.htm HTTP/1.1", upstream: "http:/10.4.36.207:8080/WebPageAlipay/app/notify_url.htm", host: "manager.example.com"
2015/12/01 15:49:58 [info] 20522#0: *55077946 recv() failed (104: Connection reset by peer) while sending to client, client: 10.255.252.31, server: manager.example.com, request: "POST /WebPageAlipay/app/notify_url.htm HTTP/1.1", upstream: "http:/10.4.36.207:8080/WebPageAlipay/app/notify_url.htm", host: "manager.example.com"
2015/12/01 15:49:58 [info] 20522#0: *55077965 recv() failed (104: Connection reset by peer) while sending to client, client: 10.255.252.31, server: manager.example.com, request: "POST /WebPageAlipay/app/notify_url.htm HTTP/1.1", upstream: "http:/10.4.36.207:8080/WebPageAlipay/app/notify_url.htm", host: "manager.example.com"

Log analysis

• recv() failed： there recv() It's a system call , That is to say Linux Network programming interface . Its function is , It's easy to understand literally , It is used to receive data . We can directly man recv, See the details of this system call , It also includes its various abnormal status codes .
• 104： This number is also related to system calls , It is recv() Call a status code when an exception occurs , This is given by the operating system . stay Linux In the system ,104 The corresponding is ECONNRESET, It is also a TCP Connected by RST Abnormal closing of message .
• upstream： stay Nginx In the terminology of reverse proxy software ,upstream Refers to the back-end server . in other words , The client sends the request to Nginx,Nginx Will forward the request to upstream, Wait for the latter to reply HTTP After response ,Nginx Reply this response to the client . Be careful , there “ client <->Nginx” and “Nginx<->upstream” It's two separate TCP Connect , It's just like the picture below ：

2、wireshark Common filters

ip.addr eq my_ip： Filter out source IP Or purpose IP by my_ip Message of 
ip.src eq my_ip： Filter out source IP by my_ip Message of 
ip.dst eq my_ip： Filter out the purpose IP by my_ip Message of 
tcp.seq eq 1 and tcp.ack eq 1 RST  The serial number of is  1, The confirmation number is also  1
frame.time >="dec 01, 2015 15:49:48" and frame.time <="dec 01, 2015 15:49:49" #frame.time  filter 
frame.time >="dec 01, 2015 15:49:48" and frame.time <="dec 01, 2015 15:49:49" and ip.addr eq 10.255.252.31 and tcp.flags.reset eq 1 and !(tcp.seq eq 1 or tcp.ack eq 1)

find TCP RST message . This requires another type of filter , That is to say tcp.flags, And here it is flags, Namely SYN、ACK、FIN、PSH、RST etc. TCP Sign a .

about RST message , The filter condition is ：

tcp.flags.reset eq 1
ip.addr eq 10.255.252.31 and tcp.flags.reset eq 1

3、 Kernel call of client handshake

Client initiates connection , These system calls are called in turn ：

• socket()
• connect()

The server listens to the port and provides services , Then the following system calls should be called in turn ：

• socket()
• bind()
• listen()
• accept()

The user space program of the server should use TCP Connect , First, get the last interface above , That is to say accept() Return of call . and accept() The premise that the call can return successfully , It is normal to complete three handshakes .