How to prevent malicious crawling of information by one-to-one live broadcast source server

In the process of running the one-to-one live broadcast source code , The generated data will be stored on the server , For any platform , Data security is the problem that needs to be solved most in data storage . Except for malicious access 、 Domain name hijacking and other data problems , Some crawlers will crawl website information maliciously , At this time, it is necessary to intercept , Prevent data leakage .
The one-to-one live broadcast source code can be configured on demand nignx The environment and Apache Environmental Science , Achieve interception , The specific operation steps are as follows ：
One 、Nginx Environment ：
1. create profile
For example, enter nginx Install under directory conf Catalog , establish ： test…conf
The contents are as follows ：

# prohibit Scrapy Wait for the tool to grab  if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {
    
     return 403;
}
# Prohibit designation UA And UA Empty visit 
if ($http_user_agent ~ "WinHttp|WebZIP|FetchURL|node-superagent|java/|
FeedDemon|Jullo|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|
CrawlDaddy|Java|Feedly|Apache-HttpAsyncClient|UniversalFeedParser|ApacheBench|
Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|
lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|BOT/0.1|
YandexBot|FlightDeckReports|Linguee Bot|^$" ) {
    
     return 403;             
}

2. Import the configuration file created above into the website configuration file
Include test.conf;

3. restart nginx Services take effect

systemctl restart nginx

Two 、apache Environment
1、 modify httpd.conf The configuration file
Add the following code , restart Apache take effect

Shell
DocumentRoot /home/wwwroot/xxx
SetEnvIfNoCase User–Agent “.*(FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms)” BADBOT
Orderallow,deny
Allow fromall
deny fromenv=BADBOT

3、 ... and 、 test
1、 Use curl -A Simulate grabbing , such as ：

curl -I -A 'Spider' www.test888.cn

give the result as follows ：

[[email protected]]# curl -I -A 'Spider' www.test888.cn
HTTP/1.1 403 Forbidden
Server: nginx/1.12.0
Date: Wed, 24 Apr 2021 11:35:21 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive

above , It is the one-to-one live source code that intercepts the content of information crawled by crawlers , Preventing malicious crawling is just one of the security strategies to ensure data security , To learn more about security policies , You can pay attention to the following content .
Statement ： The above content is original by cloudleopard technology , Without the author's consent , Prohibited reproduced , Otherwise, relevant legal liability will be investigated .