Case summary of SSH service suddenly unable to connect

a Oracle database The server （Linux Version is Oracle Linux Server release 5.7） This afternoon, there was a brief ssh It's not connected ,ssh When you can't connect ,ping The server normal , Use psping Detection port 22 It's normal （ Only return 5 A package , It didn't last ping）, Use SQL Developer You can log in to the database for any operation , in addition , adopt DPA The tool found this The server Of CPU And other resources consumption is very low （ After finding that the database services are normal , I went out to eat ）, Come back when , Feedback from colleagues ssh It's normal , Missed a good time for diagnosis , During this period, another colleague also did some examinations ：

The test found ping normal , however psping testing 8088 The network delay of port discovery is very long , There's even a timeout . He made a screenshot comparison , As shown below .

ping It's a network layer protocol , It just shows that the network is 3 The layer is connected ;tomcat It's the application layer protocol

After dinner , Find out ssh You can log in normally The server , The inspection found that this process has been running for more than 200 days , So in other words sshd The service didn't die ,sshd The service has not been restarted .

Use ps -ef | grep sshd find sshd The process of , Execute the following command

[[email protected] ~]# ps -eo pid,lstart,etime | grep 3423  3423 Sun Feb 18 13:56:11 2018 234-09:01:48

Check log information , I found several in it Did not receive identification string from xxx Information about （ Some information has been desensitized ）.

[[email protected] log]# tail -100 /var/log/secure Oct  8 14:50:48 mylnx01 sshd[4341]: pam_unix(sshd:session): session opened for user oracle by (uid=0) Oct  8 14:50:49 mylnx01 sshd[4341]: pam_unix(sshd:session): session closed for user oracle Oct 10 12:26:41 mylnx01 sshd[742]: Did not receive identification string from 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[743]: Did not receive identification string from 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[790]: Did not receive identification string from 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[789]: Did not receive identification string from 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[745]: Did not receive identification string from 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[744]: Did not receive identification string from 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[1007]: Connection closed by 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[1006]: Connection closed by 192.168.xxx.xxx Oct 10 12:26:41 mylnx01 sshd[746]: Did not receive identification string from 192.168.xxx.xxx

I searched the relevant information about this error , Generally, errors occur because ：

This one below means ssh server waited and did not receive what it needed in a timely fashion. This is typically due to connectivity issues. In an ssh connection, the server first provides its identification string, then waits for the client to then provide its identification string. If there is a loss in connection, or the client just bails, this is what you will see in the logs. If someone uses telnet or netcat to fetch your ssh banner, or other various scans, the logs on the server side will show this as well.

This error message means ssh The service didn't receive what it needed in time , And wait . Usually caused by connection problems . stay ssh Connecting , The server First, provide its identification string , Then wait for the client to provide its identification string . If the connection is lost , Or the client just exited , What you see in the log will appear .

Although it is suspected that it is a routing problem , But individuals lack detailed evidence of network monitoring , But there is also some supporting evidence ： Recently, there have been a lot of network problems in the two places , The day before yesterday, I also found that network packet switching was serious , The network administrator asked the supplier for feedback , But it's not clear what happened later . Because I don't deal with this matter .