Information security - threat detection - detailed design of NAT log access threat detection platform

Let's take a look first NAT Several models of ,nat Include nat、snat、dnat Three , The three are an abstract inheritance relationship , Let's take a look at these foundations first ;

One 、linux Store in dns The server ip Address profile

[[email protected] ~]# vim /etc/resolv.conf --> Deposit dns The server ip Address of the , When we do domain name query , Just look in this file dns Server address
# Generated by NetworkManager
nameserver 114.114.114.114

Two 、NAT

public ip： On the Internet
private ip： Use in LAN

NAT The concept of ：Network Address Translation, Network address translation 

NAT Problem solved

 It's solved ip The address is not enough 

 Ensure that users can access the Internet 

 It also plays a safety role 

NAT The working process of the implementation method ：

NAT The advantages of

 Saving is legal  IP  Address 
 Processing address   overlap 
 Increase flexibility 
 Security 
NAT  The shortcomings of 
 Delay increases 
 The complexity of configuration and maintenance 
 Some applications are not supported  ,  It can be done by static  NAT  Mapping to avoid 

linux Inside NAT

SNAT  ----------- Modification source ip

 

DNAT  ----------- Purpose of revision ip

3、 ... and 、SNAT

SNAT Typical application environment of policy

    LAN hosts share a single public network IP Address access Internet

SNAT The principle of strategy

    Source address translation ,Source Network Address Translation
    Modify the source of the packet IP Address

linux Gateway server ： It is understood as a router
NAT The function is realized through netfilter Module to achieve ,iptables to netfilter Pass parameters

SANT The principle of strategy

 vmware The network card of the virtual machine 3 A model

VMware Network Adapter VMnet1 --》hostonly Pattern -- Host only mode ： Virtual machines can only communicate with our real machines 

VMware Network Adapter VMnet8 --》NAT Pattern 

 Bridging mode ：  Virtual machines can communicate with the local , It can also communicate with other real machines in the LAN or virtual machines in the bridge mode 
     In the bridge mode, the status of virtual machine is the same as that of real machine , All access the Internet through the external router , And in the same network segment as the real machine 

To configure SNAT Strategy , Turn on snat function （iptables） analysis

iptables  -t nat  -A POSTROUTING -s 192.168.66.0/24  -o ens33  -j SNAT --to-source 192.168.0.133


-t nat  Specified in the nat Inside and outside operation 
-A POSTROUTING  stay POSTROUTING Add a rule to the position  append 
-s 192.168.66.0/24   Specify the source in the packet passing through the firewall ip The address is from  192.168.66.0/24  Network segment 
-o ens33  heavy ens33 Interface out   out-interface
 -j SNAT   take SNAT Strategy , Source of progress ip Address translation 
 --to-source 192.168.0.133  take ip The source in the bag ip Change the address to 192.168.0.133

iptables Is a firewall tool

iptables -F   Clear firewall rules

 iptables -L   View firewall rules

nmcli Use of commands （nmcli Detailed command _ GUI Anjun @KylinOS-CSDN Blog _nmcli Detailed command ）

[[email protected] ~]# nmcli  n   Check whether the network is enabled 
enabled
[[email protected] ~]# nmcli  n off   Turn off network 
 
[[email protected] ~]# nmcli  n on   Open the network 
[[email protected] ~]# nmcli  c   Check the connection of all networks 
NAME        UUID                                  TYPE      DEVICE 
ens33       c96bc909-188e-ec64-3a96-6a90982b08ad  ethernet  ens33  
ens37       4a5516a4-dfa4-24af-b1c4-e843e312e2fd  ethernet  ens37  
 Wired connection  1  a0953d48-3a24-36d3-b388-fe27b3f12b93  ethernet  --     
[[email protected] ~]# 
[[email protected] ~]# nmcli d show
GENERAL.DEVICE:                         ens33
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         00:0C:29:79:61:8A
GENERAL.MTU:                            1500
GENERAL.STATE:                          100（ Connected ）
GENERAL.CONNECTION:                     ens33
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/6
WIRED-PROPERTIES.CARRIER:                open 
IP4.ADDRESS[1]:                         192.168.0.127/24
IP4.GATEWAY:                            192.168.0.1
IP4.ROUTE[1]:                           dst = 192.168.0.0/24, nh = 0.0.0.0, mt = 104
IP4.ROUTE[2]:                           dst = 0.0.0.0/0, nh = 192.168.0.1, mt = 104
IP4.DNS[1]:                             114.114.114.114

3、 ... and 、DNAT

DNAT Typical application environment of policy
    stay Internet Publish servers located in the enterprise LAN
    stay Internet Intranet publishing server
    Release ： Go out in public

    Publish the server in one network so that other networks can access

    Release ： Behind it is the control of internal resources

DNAT The principle of strategy
    Target address translation ,Destination Network Address Translation
    Modify the destination of the packet IP Address

SNAT： LAN --> Extranet --> The problem of going out

DNAT： Extranet --> LAN --> Incoming questions

Springboard machine （ Fortress machine ）： The outside machine logs in to the springboard machine first , Then log in to the server on the springboard

The process is shown in the figure below ：  

DANT The principle of strategy

  To configure DNAT Strategy , Turn on dnat function （iptables） analysis

 iptables -t nat -A PREROUTING -i eth0 -d 218.29.30.31 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.6

-t nat   Appoint nat Inside and outside operation
 -A PREROUTING   stay PREROUTING Add a rule to the position  append

 -i eth0 from eth0 Interface into the system  in-interface

 -d 218.29.30.31 Purpose ip The address is 218.29.30.31
 -p tcp --dport 80   The transport layer adopts tcp agreement , At the same time, the destination port is 80  Destination port

-j DNAT   take DNAT Strategy
--to-destination 192.168.1.6 Purpose of revision ip by 192.168.1.6

DNAT How to publish different servers in the intranet ?

Use different port numbers to correspond to different ports in the intranet ip Address server
    iptables -t nat -A PREROUTING   -d 192.168.0.127 -p tcp --dport 80  -i ens33 -j DNAT  --to-destination 192.168.88.27
    iptables -t nat -A PREROUTING   -d 192.168.0.127 -p tcp --dport 80  -i ens33 -j DNAT  --to-destination 192.168.88.27:80

    iptables -t nat -A PREROUTING   -d 192.168.0.127 -p tcp --dport 8080  -i ens33 -j DNAT  --to-destination 192.168.88.88:8080
    iptables -t nat -A PREROUTING   -d 192.168.0.127 -p tcp --dport 3306  -i ens33 -j DNAT  --to-destination 192.168.88.136:3306

    dnat modify ip The purpose of the package ip Address , You can also modify tcp perhaps udp Destination port number in  

good , Here we are right nat Have some preliminary understanding of the model , Let's start with nat Analyze how the log is accessed to the threat detection platform ;

Here you can see nat The overall processing flow of the log server is as follows ：

1. We have a lot of nat Server user SNAT and DNAT Parsing ;
2. Each station nat The server will record its own parsing logs to local files ;
3. By means of nat Deployed on the server rsync client , take nat Upload the parsing log file to nat Log file server ;
4. stay nat Deploy ourselves on the log file server java Compiling agent;
5. java Of agent take nat Read all the parsing logs on the file server , And write MQ;
6. MQ Our consumers will nat Parsing log write ES colony 、Hadoop colony ;

Processing nat The log file on the log file server has the following difficulties ：

1.nat The server room is increased ;

2.nat Under the machine room, it is used for nat The number of parsing machines increases ;

3. Under the machine nat The resolution log file of the server is increased ;

4. High concurrency ： With the increasing traffic nat Log files are growing 、 The number of log lines of a single file keeps rising ;

Response plan ;

1. stay nat Build the file directory structure on the log file server ：~/ Computer room / machine name / date / file name + Time stamp .txt

2. A single thread monitors the machine room and machine , Find a new machine room or machine and start a new thread monitoring file ;

3. Each machine starts a thread to monitor file growth ;

4. Use thread pool to process file content , Each file is read by an independent thread and sent to MQ;

5. Sent to the MQ Of producers adopt kafka Bulk delivery , Set the batch size to 0.5m, Reduce sending times ;

The processing model of a single file is as follows

Processing logic under a thread ：

private void run() {
        String timeDescription = file.getName().substring(8, 22);// Get the timestamp , Return to 20201516_193047
        AtomicReference<Integer> sendCount = new AtomicReference<>(0);
        if (noExist()) {
            long start = System.currentTimeMillis();// Start time of current file processing 
            try (Stream<String> stream = Files.lines(Paths.get(file.getPath()))) {

                stream.forEach((s) ->
                {
                    recordContentCount();
                    log.debug("orimsg" + s);
                    String[] splits = s.split("\\s+");
                    if (splits.length > 6) {
                        try {
                            Natlog nat = new Natlog(splits, timeDescription, jifang);
                            String res = nat.toString();
                            kafkaProducerService.write(res);//kafka By modifying the size of the batch sent by the client batch.size To realize batch sending 
                            sendCount.getAndSet(sendCount.get() + 1);
                        } catch (Exception e) {
                            log.error("record err: {}", file.getPath() + "," + s);
                        }

                    } else {
                        log.error("record err: {}", file.getPath() + "," + s);
                    }
                });
            } catch (Exception e) {
                log.error("record err: {}", file.getPath() + "," + e.toString());
            }
            setFileExist();// The settings file has been read , cache 7 God , It's important 【 The log file retention days are 5 God ,7 After days, you can't retrieve this file again 】
            long cost = System.currentTimeMillis() - start;
            log.info(" Processing statistics   Current machine room ：{},  Current file :{}, Document processing quantity :{}, Time consuming ：{}, File last modified ：{} "
                    , jifang
                    , file.getPath()
                    , sendCount.get()
                    , cost
                    , new DateTime(file.lastModified()).toString("yyyy-MM-dd HH:mm:ss"));// Last modification time of the document 
        }
    }

Reference resources ：NAT、SNAT、DNAT_ Yiqin Shang's blog -CSDN Blog _nat and snat