The server is poisoned - the dish is the original sin

WeChat official account ： O & M development story , author ： Jock

Friday friend's birthday , Just finished eating and ready to sing , Received the news that the business payment failed , The problem is that the front-end machine of the bank cannot establish a normal connection with the bank .

chart 1

I'm numb …

This server was built by the bank itself , To be honest, I seldom manage ：
1、 Is a Windows The server , I'm not familiar with （ Too much food ： The road is narrow ）
2、 I don't know what's in the bank , Will it cause other problems if it moves （ It's still too much ： I'm afraid ）
3、 The volume of business is not large , Usually there is no problem （ It's still too much ： Habitual thinking ）

But something went wrong , I have to manage , The song has not been sung , Run to the garage , Turn on the computer , Start a blind operation .

First, I checked all the monitoring data （ This time depends on it ）.

First determine CPU、 Memory 、 disk IO、 The network is normal .

But it was found that at the time of the problem , The connection number of front-end processor is very high （ Maybe this is the key to the problem ）.

Can't , First try whether you can enter the server remotely , Found to be OK Of .

First use the network to check the three board axe ：（1） Open the browser to access the domain name , It was found that the connection could not be established .（2）Ping domain name , It is found that the domain name can be resolved normally .（3）Telnet IP+Port, Find out Telnet no .

At that time, I didn't think that the connection number had been used up （ Button feet of vegetables ）.

Then go find Windows The event log of , There are too many logs , Too much numbness , Looking dizzy , But Kung Fu is not inferior to those who have a heart , I still found some problems , as follows ：

What do you mean by that? ？

The local port is used up , No extra ports can be allocated and used, which will result in failure to communicate with the outside .

What do you mean by that? ？

This is to say TCP Three handshakes ,TCP When the client connects to the server , The client must assign a dynamic port , However, I will continue to repeat here , Everyone is a big man .

Windows The default dynamic port range is 1024-5000, That is, you can only initiate a contract 4000 individual Socket Connect , Then the first reaction is to increase the dynamic port allocation range （ It's so smart ）.

Then I changed the registry （[HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters]）, Increase the port range to 2000, Then restart the server （ veritable SRE（Server Reboot Enginer））.

After the server restarts , It must be able to serve normally , After all, there is no problem that cannot be solved by restarting , If there is , Then restart several times .

But it's over ？

Of course not , We have to find out why there are so many connections , Under normal business conditions , There won't be such a problem .

So open the console , Use natstat -ano, Don't see, don't know , It's a shock at first sight , I found that many links are with 47.95.x.x On going .

I use netstat -ano | find "47.95.x.x" | find /c "80" A: , New machine , There is 2000 Multiple connections , It really scared me .

First ask the business side , Make sure that IP Is it a third party , Getting a clear answer is not , I'm starting to panic .

Then check online , Found this IP It's from Alibaba cloud Beijing Data Center .

At first, I thought it was some public cloud services of Alibaba cloud , For example, Alibaba cloud shield 、 Cloud monitoring, etc （ Know everything. ）.

So go straight to Ali , I got a reply that Alibaba cloud Dun did not provide these services IP, I panic. , A bad premonition haunts me .

Put this right away IP Disabled , Limit it in the security group .

And then from netstat -ano It can be found mainly through 1060 This process establishes a connection

uqccmg This process looks unconventional （ It does not conform to the conventional naming rules ）, Of course, we should also confirm .

After getting the exact answer , Try to kill the process , Kill again , Kill again , I feel numb .

Then through the process to find the file ,

Look at this non mainstream icon , Poisoning is beyond doubt .

Try deleting the file , Of course it failed , however , Also know that the process is through .net Service hosted , Then I'll put .net Stop , Anyway, I didn't use this service .

After stopping , It can really kill the process , And it didn't start again , There are also fewer connections .

BUT, Don't be so optimistic …

The virus is still there , It's just not up , That's it .

So only with the help of anti-virus software , Put down a tinder , I found out 17 Risk projects .

First, kill the virus through anti-virus software , I don't know whether it's done or not （ Food is original sin ）.

Business can now be used normally ,socket The connection is also normal , No more suspicious processes were found .

however , I always have no bottom in my heart , The best way is to reinstall , It also needs to evaluate the migration and installation costs , The main thing is not that we install it ourselves , I don't know. …

Through this experience , I found myself too delicious ：
1、 The system security reinforcement is not done well , There is no anti-virus software installed , The main reason is that Alibaba cloud's yundun is too expensive …
2、 The server is not checked frequently , Did not do a good job in daily inspection
3、 Open ports that don't need to be opened （ The bank opened , I dare not turn it off , To do ？）
4、 Yes Windows The server is naturally rebellious

Last , One sentence summary ： too TM fail .

I am a Jock ,《 O & M development story 》 A member of the official account team. , Front line operation and maintenance workers , Cloud native practitioners , It's not just hard core technology , And our thinking and perception of Technology , Welcome to our official account , Looking forward to growing up with you ！