IP black and white list

1、DDOS attack

        DDOS Also known as distributed denial of service , The full name is Distributed Denial of Service.DDOS This is to use reasonable requests to cause resource overload , Service Unavailable .

         For example, a parking lot has a total of 100 A parking space , When 100 After all the parking spaces are full , Another car wants to park , You have to wait for the existing car to go out first . If the existing car doesn't go out all the time , Then there will be a long line at the entrance of the parking lot , The parking lot is overloaded , It doesn't work , This situation is “ Denial of service ”. Our system is like a parking lot , The resources in the system are parking spaces . Resources are limited , And the service must be provided all the time . If the resources are already occupied , Then the service will be overloaded , Causes the system to stop new responses .

         Distributed denial of service attacks , Magnified the normal request several times , Attack through several network nodes at the same time , To achieve scale effect . These network nodes are often controlled by hackers “ chicken ”, When the quantity reaches a certain scale , There is a “ Botnet ”. Large botnets , Even tens of thousands 、 Hundreds of thousands of units . Launched by a botnet of this size DDOS attack , Almost unstoppable .

         common DDOS The attack has SYN flood、UDP flood、ICMP flood etc. . among SYN flood Is one of the most classic DDOS attack , It was found in 1996 year , But it still maintains a very strong vitality .SYN flood So rampant because it takes advantage of TCP Defects in the design of the protocol , and TCP/IP Protocol is the foundation of the whole Internet , Pull one hair and move the whole body , Nowadays, it is almost impossible to fix such defects .

Syn_Flood Attack principle ：

         The attacker first forges the address to launch the attack on the server SYN request （ Can I establish a connection ？）, The server will respond with ACK+SYN（ Sure + Please make sure the ）. And the real IP Will think , I didn't send a request , No response . The server did not receive a response , Will try again 3-5 And wait for one SYNTime（ commonly 30 second -2 minute ） after , Drop this connection .

         If an attacker sends a large number of such forged source addresses SYN request , The server side will consume a lot of resources to handle this semi connection , Saving traversal consumes a lot of CPU Time and memory , What's more, we have to constantly analyze the IP Conduct SYN+ACK Retry .TCP It's a reliable agreement , Then the message will be retransmitted , The default number of retries is 5 Time , The retry interval from 1s Start doubling every time , Respectively 1s+ 2s + 4s + 8s +16s = 31s, The first 5 We'll have to wait after the next delivery 32s Only then did I know that 5 I'm out of time , So the total is 31 + 32 = 63s.

         That is to say, a fake syn message , Will occupy TCP Prepare the queue 63s For a long time , That is to say, without any protection , Frequently send forged forgeries syn package , Will run out of connection resources , So that the real connection cannot be established , Unable to respond to normal request . The end result is that the server has no time to ignore normal connection requests — Denial of service .

Syn_Flood defense ：

cookie Source Authentication ：

         The principle is syn The message is first sent by DDOS Protection system to respond syn_ack. Bring specific sequencenumber （ Write it down as cookie）. The real client will return a ack also Ack number by cookie+1. And forged clients , Will not respond . So we can know those IP The corresponding client is real , The real client IP Join the white list . The next visit will be directly through , And other forgeries syn The message is intercepted .

reset authentication ：

        Reset Certification uses TCP The reliability of the protocol , First of all, by DDOS Protection system to respond syn. Protective equipment received syn Post response syn_ack, take Ack number ( Confirmation no. ) Set to a specific value （ Write it down as cookie）. When the real client receives this message , It is found that the confirmation number is incorrect , Will send reset message , also sequence number by cookie + 1. And the forged source , There will be no response . In this way, we can integrate the real client IP Join the white list .

         In many confrontations DDOS In the products of , Generally, a variety of algorithms are used , Combine some DDOS Characteristics of attack , Clean the flow . against DDOS The network devices can be connected in series or in parallel at the network outlet . but DDOS It is still a problem in the industry , When the attack traffic exceeds the network device , Even the maximum load of bandwidth , The network will still be down . Generally speaking , The reason why large websites look more “ resist ”DDOS attack , Because the bandwidth of large websites is sufficient , The number of servers in the cluster is also relatively large . But the resources of a cluster are limited after all , In the actual attack ,DDOS The traffic can even reach several G To dozens G, In this case , Only cooperate with network operators , Joint completion DDOS The response to the attack .

2、CC attack

CC The attack was DDOS One way of attacking , It can be understood as the application layer DDOS attack .

The attacker uses the proxy server to generate a legitimate request to the victim host , Realization DDOS And camouflage ：CC(Challenge Collapsar).

        CC The principle of attack is very simple , It is to constantly make normal requests for some application pages that consume large resources , In order to achieve the purpose of consuming server resources . stay Web Application , Query the database 、 read / Write hard disk files and other operations , It will consume more resources . A typical example ：

String sql = " select * from post where targid=${targid} order by postid desc limit ${start},30";

         When post The table data is huge , Turn pages frequently ,${start} When the number increases sharply , Query result set =${start}+30; The query efficiency shows a significant downward trend , And multiple concurrent and frequent calls , The query cannot be completed immediately , The resource cannot be released immediately , It will cause too many database requests to connect , The database is blocked , The website cannot be opened normally .

        CC Is to make full use of this feature , Simulate multiple users to constantly access those high computing 、 high IO The data of . Why use a proxy ？ Because agents can effectively hide their identity , You can also bypass all firewalls , Because basically all firewalls detect concurrency TCP/IP Number of connections , Beyond a certain number, a certain frequency is considered to be Connection-Flood.

         The Internet is full of all kinds of search engines 、 Crawlers of information collection and other systems （spider）, It often happens that crawlers directly crawl small websites to death , This is related to the application layer DDOS The result of the attack is very similar to .

         application layer DDOS The attack can also be done in the following ways ： After hackers invaded a website with a lot of traffic , By tampering with the page , Divert huge user traffic to target websites .

<!-- Then the users who visit this page , Will be right target To launch a get request , This may directly lead to target Denial of service -->
<iframe src="http://target" height="0" width="0">
</iframe>

         application layer DDOS An attack is an attack against server performance , So many ways to optimize server performance , Can mitigate this attack more or less . For example, put the frequently used data on memcache in , Compared with the resources consumed by querying the database , Inquire about memcache The resources consumed are negligible .

         But many performance optimization schemes are not designed to fight the application layer DDOS Designed to attack , Therefore, it is not difficult for an attacker to find a page that consumes a lot of resources . For example, when memcache When the query fails to hit , The server must query the database , This increases the consumption of server resources , The attacker only needs to find such a page .

         At the same time, the attacker can trigger “ read ” Outside of data operation , It can also trigger “ Write ” Data manipulation ,“ Write ” The behavior of data will generally cause the server to operate the database .

3、CC protective

application layer DDOS Attack is not an insurmountable problem , Generally speaking , We can start from the following aspects .

         First , Performance optimization should be done for application code . Reasonably make cache It is a good optimization scheme , Transfer the pressure of the database to memory as much as possible . In addition, resources need to be released in time , For example, close the database connection in time , Reduce the consumption of empty connections .

         secondly , Optimize the network architecture . Be good at using load balancing and shunting , Avoid the concentration of user traffic on a single server . At the same time, we can make full use of CDN And the role of the image site , Relieve the pressure of the main station .

         There's more , Use page static technology , Use the client browser's caching function or the server's caching service , as well as CDN Buffering services for nodes , Can reduce the pressure of data retrieval and calculation on the server side , Respond quickly to the results and release the connection process .

         Last , And the most important point , Implement some countermeasures , For example, limit each IP Address request frequency , Dynamically join the blacklist after exceeding the limit policy

（1） Verification Code

         The following is a page where users submit comments , Embedding verification code can effectively prevent resource abuse , Because usually the script cannot automatically recognize the verification code . But the verification code is also divided into three, six, nine and so on , Some verification codes are easy to identify , Others are more difficult to identify . The original intention of the invention of verification code , To identify people and machines . But if the verification code is designed too complex , Then it's hard for people to recognize , So the verification code is a double-edged sword .

（2）Detecting system abuse

        Yahoo It provides us with a solution . If the application layer is initiated DDOS The attack IP The addresses are real , So in reality , Of the attacker IP In fact, the address can not grow indefinitely . Suppose the attacker has 1000 individual IP Address attack , If requested 10000 Time , On average, each IP The address requests the same page to reach 10 Time , If the attack continues , Single IP There will also be more requests for addresses , But anyway , It's all here 1000 individual IP Do polling within the address range .

         So Yahoo Implemented a set of algorithms , according to IP Address and Cookie Etc , The request frequency of the client can be calculated and intercepted .Yahoo This system is also designed for Web Server A development module , But there will be one in the overall architecture master The server computes all data in a centralized way IP Address request frequency , And synchronize the policy to each Webserver On .

        Yahoo The defense system designed , Tested by practice , It can effectively resist the application layer DDOS Attacks and some similar resource abuse attacks . but Yahoo It is not open source , Therefore, for some Internet companies with strong R & D ability , According to the description in the patent , Implement a similar system .

4、IP Black and white list

• Web Application firewall - IP Black and white list configuration
• CDN - To configure IP Black and white list
• DDoS protective - Configure black and white lists
• Development IP Black and white list function  

（1）OpenResty

        OpenResty It's based on Nginx Scalable Web platform , Initiated by Chinese Zhang Yichun , Provides a lot of high quality third-party modules .OpenResty Is a powerful Web application server ,Web Developers can use Lua Scripting language mobilization Nginx Various supported C as well as Lua modular , More importantly, in terms of performance ,OpenResty It can be quickly constructed to be competent 10K The ultra-high performance of the above concurrent connection response Web Application system .360,UPYUN, Alibaba cloud , Sina , Tencent network , Where to go , Cool dog music, etc OpenResty Deep users .

（2）Lua

        Lua Is a lightweight and compact scripting language , Using standard C Language written and open in source , It's designed to be embedded in the application , This provides the application with flexible extension and customization capabilities .Lua It's a research group at the Catholic University of Rio de Janeiro in Brazil 1993 Year development .

adopt Lua Write current limit 、 Permission authentication 、 Black and white list and other functions

Lua characteristic ：

         Lightweight : It uses standard C Language written and open in source , After compiling, it's only over 100 K, It can be easily embedded in other programs .

         Scalable : Lua Provides a very easy to use extension interface and mechanism ： By the host language （ Usually C or C++） Provide these functions ,Lua They can be used , It's like a built-in feature .

5、 Implementation of dynamic blacklist

（1） install OpenResty

#  download 
wget https://openresty.org/download/ngx_openresty-1.9.7.1.tar.gz
#  decompression 
tar xzvf ngx_openresty-1.9.7.1.tar.gz
cd ngx_openresty-1.9.7.1/
#  To configure 
./configure
#  compile 
make
#  install 
make install
#  To configure  nginx profile PATH
PATH=/usr/local/openresty/nginx/sbin:$PATH
export PATH
#  Specify the configuration 
nginx -c /usr/local/openresty/nginx/conf/nginx.conf

（2） To configure

use OpenResty And the following redis Components , To configure redis Database information and blacklist strategy

set $redis_service "127.0.0.1";
set $redis_port 6380;
set $redis_db 0;
# 1 second 50 query
set $black_count 50;
set $black_rule_unit_time 1;
set $black_ttl 3600;
set $auto_blacklist_key blackkey;

redis_service： redis The server ip Address

redis_port： redis Server port

redis_db： What is used redis db

black_count： The maximum number of visits to pull the black limit

black_rule_unit_time： Save time of blackout limit times , That is, the number of saved visits kv Of ttl

black_ttl： The survival time of the blacklist

auto_blacklist_key： kv Part of key

Focus on control black_count and black_rule_unit_time

（3）lua Script

ip_blacklist.lua, from ip And token（ Access credentials ） Start to control

local redis_service = ngx.var.redis_service
local redis_port = tonumber(ngx.var.redis_port)
local redis_db = tonumber(ngx.var.redis_db)
local black_count = tonumber(ngx.var.black_count)
local black_rule_unit_time =tonumber(ngx.var.black_rule_unit_time)
local cache_ttl = tonumber(ngx.var.black_ttl)
local remote_ip = ngx.var.remote_addr

--  Count 
function my_count(redis, status_key, count_key)
    local key = status_key
    local key_connect_count = count_key
    local Status = redis:get(key)
    local count = redis:get(key_connect_count)
    if Status ~= ngx.null then
        --  Status as connect  And  count Not empty   And  count <=  Blackout times 
        if (Status == "Connect" and count ~= ngx.null and tonumber(count) <= black_count) then
            --  Read it again 
            count = redis:incr(key_connect_count)
            ngx.log(ngx.ERR, "count:", count)
            if count ~= ngx.null then
                if tonumber(count) > black_count then
                    redis:del(key_connect_count)
                    redis:set(key,"Black")
                    --  Permanent ban 
                    -- Redis:expire(key,cache_ttl)
                else
                    redis:expire(key_connect_count,black_rule_unit_time)
                end
            end
        else
            ngx.log(ngx.ERR,"The visit is blocked by the blacklist because it is too frequent. Please visit later.")
            return ngx.exit(ngx.HTTP_FORBIDDEN)
        end
    else
        local count = redis:get(key)
        if count == ngx.null then
            redis:del(key_connect_count)
        end
        redis:set(key,"Connect")
        redis:set(key_connect_count,1)
        redis:expire(key,black_rule_unit_time)
        redis:expire(key_connect_count,black_rule_unit_time)
    end
end

--  Read token
local token
local header = ngx.req.get_headers()["Authorization"]
if header ~= nil then
    token = string.match(header, 'token (%x+)')
end
local redis_connect_timeout = 60
local redis = require "resty.redis"
local Redis = redis:new()
local auto_blacklist_key = ngx.var.auto_blacklist_key

Redis:set_timeout(redis_connect_timeout)

        
local RedisConnectOk,ReidsConnectErr = Redis:connect(redis_service,redis_port)

local res = Redis:auth("password");

if not RedisConnectOk then
    ngx.log(ngx.ERR,"ip_blacklist connect Redis Error :" .. ReidsConnectErr)

else
    --  Successful connection 
    Redis:select(redis_db)
    local key = auto_blacklist_key..":"..remote_ip
    local key_connect_count = auto_blacklist_key..":key_connect_count:"..remote_ip
    my_count(Redis, key, key_connect_count)

    if token ~= nil then
        local token_key, token_key_connect_count
        token_key = auto_blacklist_key..":"..token
        token_key_connect_count = auto_blacklist_key..":key_connect_count:"..token
        my_count(Redis, token_key, token_key_connect_count)
    end
end

         As for those added to the blacklist ip And token, How to do the next step , This is for the specific application under the server , I won't elaborate here .

（4） Configuration to nginx Of conf When

server {
    listen 80;
    server_name edu.lagou.com;
    root /~/public;
    #  Load profile 
    include /etc/nginx/conf.d/blacklist_params;
    #  Specify what needs to be executed in the request  lua  Script 
    access_by_lua_file /etc/nginx/conf.d/ip_blacklist.lua;
    location / {
        }
    error_log /etc/nginx/conf.d/log/error.log;
    access_log /etc/nginx/conf.d/log/access.log;
}

         Configuration is complete , stay console Restart in nginx nginx -s reload , You can realize the need of dynamically adding blacklists . As for those added to the blacklist ip And token, How to do the next step , This is for the specific application under the server , I won't elaborate here .

        API gateway Kong, be based on OpenResty, Open source from 2015 year , Its core value lies in its high performance and cross site . From the global 500 Strong organizational statistics ,Kong It is now maintained 、 The most widely used gateway in production environments .Plugin IP Restriction By setting IP White list and blacklist , According to client IP To intercept and protect some requests .