0X01 Preface

Please indicate the source of the reprint ：https://www.cnblogs.com/huim/

They need to have good service ability to the outside world , Transparent to internal processes , Log 、 Troubleshooting is simple .

The service capability here refers to the system level services , The ability to encapsulate the scanner as a business service for the business is beyond the scope of this article

0X02 Simple scanning

High end vulnerabilities often use the most simple scanning methods

The simplest scanning requirements , Just read the data from the database , Just run through all the rules regularly .



A script updates the asset , A script reads data regularly 、 Scan with rules 、 And type the results into the database , A script reads the results regularly and sends an email , This is enough SRC The need for automated vulnerability digging , And the effect is good .

0X03 Distributed scanning

As more assets are scanned , The turtle speed scanning of a single machine is worrying , So the step of running rules plus distributed , That is, the task is sent to the queue (redis/MQ/kafka etc. ), Then run the scan rule by multiple nodes 、 Output vulnerability results

0X04 Several data source scans

This makes it easy to scan for host vulnerabilities

Further back , I don't want to just scan the host for vulnerabilities , Also want to scan the injection /XSS/SSRF/XXE Based on url A loophole in the , With url Type data .

Some vulnerabilities are even found to be targeted at domain names ( pure IP+ Port requests cannot reach load balancing ), Have a domain Data of type .

0X05 Multi task scanning

At this time, the production module can cope , That is, read various types of data 、 Bind various types of plug-ins .

But sometimes new rules are added , Want to simply scan the rule of scanning all data , Need another script and a temporary producer .

Sometimes new assets are added , Want to scan all the rules for this batch of assets alone , You need to write a producer script temporarily .

The code becomes jumbled , The operation becomes tedious , So we have the concept of task .

Tasks are used to bind data to rules , A task is a unit that produces scanning subtasks .

This incremental rule scans the full amount of data , Add a task to bind the rule and the corresponding data ; Incremental asset scan full rule , Add a task to bind these assets and the corresponding rules .

It is not convenient to operate tasks and rules from the database , So a visualization platform is added , Can be found in web End publishing scan task 、 Add and modify rules .

0X06 Multi data source scanning

Within Party A , As more and more data sources are accessed ,url Data has mirrored traffic 、 Reptile traffic 、 Agent traffic 、nginx Flow, etc ,host The data are hids agent Traffic 、 Black box asset detection traffic 、cmdb/IT Equal flow ,domain There is domain name explosion traffic 、 Traffic obtained by internal operation and maintenance system, etc .

Every additional data source , Have to add a piece of code logic : " When the data source is a When , Where to get traffic data ".

When the number of data sources exceeds ten , The data source acquisition code of the task module becomes very miscellaneous , And hard coded horizontal ( Where to get data )、 Logic is not universal (a The data of should be paged through from the interface 、b The data needs to be from redis read 、c The data of kafka、d Data from the database ).

Some data do not go through a certain section of filter matching logic in the middle , So we need to add another field is_xxx identification , In the engine if is_xxx=True, Low code versatility 、 Highly coupled , encounter bug The cost of troubleshooting is very high , For example, how can there be such output results when encountering this traffic 、 How can you report such problems incorrectly , It often takes half a day to track traffic .

Therefore, it is necessary to transform the data source , Unified data source input format , There are several types of data sources ,url/host/domain, Each type has a fixed format , External input in this format .

When there are too many data sources , There are too many external input codes , It can be abstracted to form a data input module .

Like defining redis Where to get type data 、 How does the interface get data through paging 、 How to read the database iteratively , Once again, configure the data format conversion mode . In this way, the traffic types that need to be added , The code to be added is a reusable data acquisition method .

0X07 Inter system service capability

But there is another problem , When the scanning capability needs to be invoked across departments or projects , Very inconvenient , You need to configure the data source on the input , You also need to scan developers to add such data , The scanning results need to be obtained from the database , Some have no results. I don't know whether they are not scanned or not .

For business parties , Demand addition and modification 、 Service invocation is inconvenient .

Therefore, it is necessary to improve the ability of service provision , For the caller , The scan is a black box , Just pass in the data 、 Start the task 、 To get the results , What is provided to the caller is the scanning service capability .

For scan engine developers , Externally encapsulate the engine capability , Services are separated from upstream and downstream services , Low coupling is also achieved 、 High maintainability 、 Extensible and easy to expand , The engine code will not be changed frequently due to the increase of requirements 、 This leads to code redundancy 、 The cost of development and maintenance increases .

Realization way ：

Data access , The caller registers the data tag in the management platform , And mark the data label when transferring data ( Abstract data configuration steps );

When the result is output , The caller registers the callback interface ( Data is sent to the callback interface ), The scanning results are flawed / No loopholes / No such scanning , The callback interface selects the received result type ; Or register the disposal result label , The scan result is called to the message bus .

The callback method does not know the status of the opposite interface settings , Maybe the interface reported an error message and did not call it correctly , The interface may return 200 Of status: false But I can't tell if it was a failure , Simply put, data consistency cannot be guaranteed , There is an error in the scan result, but the interface does not have this result because of an error . So try to use message bus , The consumer records the consumption failure data 、 Check and re consume , Ensure that the interface receiving the results will not lose data .

Then the registrant operates the task , Bind the label of the traffic to be scanned , Rules to scan , The method of disposal is to call a callback or label a result .

Realization effect ：

This encapsulates the engine , It can basically ensure that there are not too many data sources in the engine , And East and West , There are a lot of code read for different data sources .

The engine itself only guarantees data reading 、 Select the scanning rule according to the specified task 、 Print the scanned results to the result queue or call back to the caller .

0X08 Whole process log

But there's another problem , The cost of troubleshooting is relatively high .

The scanner engine logic is more complex than some products , It mainly involves the survival detection 、 Cluster judgment 、 White list restrictions 、QPS control 、 Task scheduling and other functions , Sometimes, traffic is lost or a field is wrong, which leads to false positives 、 There is a problem with the content of the request before the plug-in runs, resulting in traffic that is judged to be not alive and thus missing reports .

These situations are in the form of redis In the engine for the queue , It is troublesome to check .

Therefore, the whole process log is required ： It is best to know what the intermediate results of several key steps are , It is convenient to troubleshoot problems . The amount of data scanned by the scanner in the middle process after de duplication is less than IDS Big ( Processing results of tens of billions per day ), Probably tens of millions , It can all be recorded , Resource constraints can be recorded only for a period of time .

About the types of logs ： Generally, the intermediate results we need for tracing and troubleshooting are data sources 、 Scan subtasks 、 Scan results .

About log implementation ：redis pop Then the data is gone , It is troublesome to double write the engine after reading .

So choose a message queue that you can subscribe to , such as kafka, The engine uses a group Consume information , Start another service with another group For this batch topic To store the data of , conversant ELK structure .

DAST Black box vulnerability scanner Chapter five ： More articles on vulnerability scanning engine and service capabilities

  1. web Write a network vulnerability scanner

    I've seen a lot these two days web Article written by vulnerability scanner , such as W12scan And its predecessor W8scan, And the past in the context of pig man's automated attacks . Now and in the future , And many excellent scanners and blogs on the Internet , Except for the Jinghu Lake, which was written in part before ABC Segment scan ...

  2. qqzoneQQ Design of space vulnerability scanner attilax summary

    qqzoneQQ Design of space vulnerability scanner attilax summary 1.1. Get each other qq( The third party , And other mechanisms )1 1.2. QQ Spatial html technological process 1 1.3. Judge if there is authority 1 1.4. 2015 year Web clothing ...

  3. Web Analysis of common logic vulnerabilities in security testing （ Actual combat ）

    Web Analysis of common logic vulnerabilities in security testing ( Actual combat ) Briefly : Ultra vires vulnerability is a common type of vulnerability , Ultra vires loopholes can be understood as , A normal user A Usually, you can only add, delete, modify and check some of your own information , But because of the programmer's negligence , Add, delete and modify information ...

  4. 6. Vulnerability scanners （ Vulnerability scanner 11 individual ）

    Nessus It is one of the most popular and capable vulnerability scanners , In particular UNIX System . It was originally free and open source , But they are 2005 The source code was shut down in 1996 , And in 2008 Deleted the free “ register Feed” edition . Now it costs... Every year 2,19 ...

  5. 5. Web vulnerability scanners （ Web vulnerability scanner 20 individual ）

    5. Web vulnerability scanners ( Web vulnerability scanner 20 individual ) Burp Suite It's an attack Web Application integration platform . It contains various tools , There are many interfaces between them , Designed to facilitate and speed up attack applications ...

  6. [ original ]AndroBugs_Framework Android Introduction to vulnerability scanner

    [ original ]AndroBugs_Framework Android Introduction to vulnerability scanner 1 AndroBugs_Framework Android Introduction to vulnerability scanner An efficient Android Vulnerability scanner , It can help drive ...

  7. Acunetix Web Vulnerability Scanner(WVS)(Acunetix Network vulnerability scanner )

    Acunetix Network vulnerability scanning software is a security testing tool to detect the security of your network Acunetix Web Vulnerability Scanner(WVS) (Acunetix Network vulnerability scanner ) technology Network application security scanning technology ...

  8. Xiao Chen WEB Vulnerability scanner V2.0

    Xiao Chen WEB Vulnerability scanner V2.0 Xiao Chen WEB Vulnerability scanner V2.0 https://pan.baidu.com/s/1NSmFCyxowEa3YlOuhvtwwQ

  9. Vulnerability scanner -AWVS

    Catalog Introduce Vulnerability scanning Web crawler == Vulnerability analysis . verification The host found Subdomain detection SQL Inject HTTP Header editing HTTP monitor Introduce AWVS by Acunetix Web Vulnarability Scanner Of ...

  10. Penetration testing learning twenty 、 Summary of other vulnerabilities PHP Related loopholes

    The outline : PHP Related loopholes JSP Related loopholes Summary of other vulnerabilities PHP Related loopholes The file contains a vulnerability php://input And so on Code Execution Vulnerability Variable coverage vulnerability The file contains a vulnerability Program developers usually write reusable functions to ...

Random recommendation

  1. java.lang.Boolean-> Introduce

    Introduce : public class Boolean extends Object implements Serializable, Comparable<Boolean> Boolean Class is ...

  2. grep Query text ： Ask a simple question shell problem , take grep The output of is assigned to a variable

    Ask a simple question shell problem , take grep The output of is assigned to a variable use grep The output obtained by the command was assigned to a variable unsuccessfully . grep The order is as follows : Code : $ grep -c '^abc' file.txt Output is 22, surface ...

  3. ubuntu10.04.4 Lower installation JDK

    First step :http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html Download installation package ( It could have passed ...

  4. RDD One of the common methods subtract&amp;intersection&amp;cartesian

    subtract Return an RDD with the elements from `this` that are not in `other` .     def subtract(othe ...

  5. The great gods in the group communicate with you WEB Experience Industry experts give advice on employment skills

    I know you are a person with ideal and ambition , Not content to be a junior front-end development engineer . In the next stage , We will embark on the advanced road of front-end development , Raise your ability to a higher level . Again , Wages will also go up one level ! however , If you are one ...

  6. GAT2.0 Using document （ Single interface development ）

    3   Start writing use cases 3.1 Interface use case development 1)         preparation l  The first step is from github(https://github.com/GeneralAutomationTesting/GAT2.0 ...

  7. jqury ajax standard

    Parameter name type describe url String ( Default : Current page address ) Address to send request . type String ( Default : "GET") Request mode ("POST" or ...

  8. be based on SSH The tool is called sshfs. sshfs It allows you to mount the file system of the remote host directly locally

    Another great one is based on SSH The tool is called sshfs. sshfs It allows you to mount the file system of the remote host directly locally . $ sshfs -o idmap=user [email protected]:/home/user ...

  9. The foundation of network socket

    socket We know IP Layer of ip The address can uniquely identify the host , and TCP Layer protocol and port number can uniquely identify a process of the host , So we can use ip Address + agreement + The port number uniquely identifies a process in the network . After being able to uniquely identify the processes in the network , it ...

  10. canvas example _ The clock

    design sketch : It's a moving clock One . The composition of the clock 1. Clock dial ( Blue )  2. scale ( black )  3. Hour hand ( black )  4. minute hand ( black )  5. second hand ( Red ) To be beautified     Two . The main applied technology Canvas Draw line Canv ...