Xray linkage crawlergo automatic scanning pit climbing record

xray brief introduction

xray Is a powerful security assessment tool , It is made up of many experienced front-line safety practitioners , The main features are :

• Fast detection speed , Fast packet speed ; The vulnerability detection algorithm is efficient .
• Support a wide range of , Big to OWASP Top 10 General vulnerability detection , As small as all kinds of CMS frame POC, Can support .
• High code quality , The quality of the people who write the code is high , adopt Code Review、 unit testing 、 Multi layer verification such as integration testing to improve code reliability .
• High quality can be customized , Various parameters of the engine are exposed through the configuration file , By modifying the configuration file, you can greatly customize the function .
• There is no threat to safety ,xray Positioning as a safety assessment tool , Instead of attacking tools , All built in payload and poc All are harmless tests .

Currently supported vulnerability detection types include :

• XSS Vulnerability detection (key: xss)
• SQL Injection detection (key: sqldet)
• command / Code injection detection (key: cmd-injection)
• Directory enumeration (key: dirscan)
• Path crossing detection (key: path-traversal)
• XML Solid injection detection (key: xxe)
• File upload detection (key: upload)
• Weak password detection (key: brute-force)
• jsonp testing (key: jsonp)
• ssrf testing (key: ssrf)
• Baseline check (key: baseline)
• Arbitrary jump detection (key: redirect)
• CRLF Inject (key: crlf-injection)
• Struts2 Serial vulnerability detection ( premium ,key: struts)
• Thinkphp Serial vulnerability detection ( premium ,key: thinkphp)
• POC frame (key: phantasm)

among POC The frame is built-in by default Github On the contribution of poc, Users can also build on demand poc And run .

crawlergo brief introduction

crawlergo It's a use chrome headless Model for URL Collected browser crawlers .

It's key to the whole web page DOM The rendering phase proceeds HOOK, Automatically fill in the form and submit , With intelligent JS Events trigger , Collect as many entries as you can .

built-in URL De duplication module , A large number of pseudo statics are filtered out URL, For large websites, it still maintains a fast parsing and crawling speed , Finally, we get a high-quality set of request results .

crawlergo The following features are currently supported ：

• Native browser environment , Scheduling tasks in the process pool
• Form intelligent filling 、 Automated submission
• complete DOM Event collection , Automatic trigger
• intelligence URL duplicate removal , Remove most repeat requests
• Comprehensive analysis and collection , Include javascript The contents of the document 、 Page notes 、robots.txt Files and common paths Fuzz
• Support Host binding , Automatic addition Referer
• Support request agent , Support the active push of crawler results

precondition

• Public network VPS a
  • Windows、Linux Fine
• xray1.8.2 premium （ end 20220113 The latest version ）
  • Advanced version supports Winddows、Linux、Mac
  • Official account Xiaosheng observation room And the reply xray premium You can download it.
• crawlergo Source code https://github.com/Qianlitp/crawlergo/releases
• Google browser （ Download and install by yourself ）
• Go Environmental Science 、Python Environmental Science （ Download and install by yourself ）

compile crawlergo

take crawlergo The source code will go back to the local , And compile it into an executable file

Get into crawlergo-0.4.2\cmd\crawlergo Directory to compile , perform go build crawlergo_cmd.go sentence

In the brand new Go When this statement is executed for the first time in the environment, the following error messages will appear ：

At this point, just execute the following statement and wait a moment before recompiling

go mod tidy

Re executing go build crawlergo_cmd.go when , The following errors will still be reported ：

open C:\Users\ Xiaosheng observation room \AppData\Local\Temp\go-build2905132708\b001\exe\a.out.exe:
The process cannot access the file because it is being used by another process.
go: failed to remove work dir: remove C:\Users\ Xiaosheng observation room \AppData\Local\Temp\go-build2905132708\b001\exe\a.out.exe:
The process cannot access the file because it is being used by another process.

resolvent ：

 find  C:\Users\ Xiaosheng observation room \AppData\Local,  Right click and click  Temp  Folder  >  attribute  >  Security  >  senior 

 add to  >  The main body ：Everyone >  Open all permissions  >  application  >  Definite exit 

And then re execute go build crawlergo_cmd.go Statement will be generated in the current directory crawlergo.exe file

Automate project deployment

Project address

https://github.com/timwhitez/crawlergo_x_XRAY

The whole project clone To local , And will xray_windows_amd64.exe and crawlergo.exe Place items separately xray and crawlergo Under the table of contents

To configure xray

After executing the following statement, you will be in xray Generate... In the current directory config.yaml Configuration files and CA certificate

xray_windows_amd64.exe webscan --listen 127.0.0.1:7777 --html-output Xiaosheng observation room .html

modify launcher_new file

take launcher_new.py Medium chrome The browser path is changed to native chrmoe Browser path

And put the... In the path \ Replace with /

Actual combat recurrence

1. Save the target to be tested to targets.txt
2. It will be revised launcher_new.py and targets.txt Copied to the crawlergo Under the table of contents
3. xray Set to listening status
4. stay crawlergo Execute under directory python3 launcher_new.py

• python The following libraries need to be installed in the environment fake_useragent、simplejson

Batch testing ：

crawlergo Generated in the directory sub_domains.txt Subdomains crawled for crawlers , crawl_result.txt For the reptiles url

When a vulnerability is discovered, it will be found in xray Automatically generated under the directory Xiaosheng observation room _html The report

Other needs _xray Reverse connection platform

Modify the configuration file _ Server side

xray The default configuration file will be generated after execution cofig.yaml

You need to modify the parameters of the reverse platform in the configuration file ：

Will modify the cofing.yaml And xray Put other documents together Server side

stay Server side perform xray_linux_amd64 reverse

The normal opening of the page through the browser indicates that there is no exception

The test results

Select... In the page Generate a URL And test

Modify the configuration file _ Local end

Modify local xray in config.yaml Profile's remote_server And add Server side Of http Address

• When the configuration is complete , When detected SSRF And other related vulnerabilities will feed back the results to the server
• The anti connection platform can only be used in the advanced version
• If there is a security group, you need to release the corresponding port , The open port of firewall must be UDP No TCP, The new security group defaults to TCP
• If you still cannot access the corresponding http service , Consider disabling the firewall on the server side ufw disable