Anti crawling strategy (IP proxy, setting random sleep time, bilbili video information crawling, obtaining real URLs, processing special characters, processing timestamp, and multithreading)

Common anti - Crawler strategies

1. adopt Headers The crawler

• By identifying the... Requested by the user Headers Anti - crawling is the most commonly used anti - crawling strategy for websites . Many websites will be right HTTP Asking for the head User-Agent To test （ Determine whether it is accessed by the browser ）; There are some websites that are right Refer To test （ Anti theft links to some resource websites ）; And some will be right Cookie To test （ You need to log in to get more data ）.

2. Anti crawler based on user behavior

• It is also a common anti - Crawler strategy to determine whether a request is from a crawler by detecting user behavior . for example , same IP The address can be accessed many times in a short time , Or the same account can be operated many times in a short time , It is possible for the website to take anti - Crawler measures .

3. Anti - crawling with dynamically loaded data

• There are some web pages through JavaScript Dynamically generated , Unable to directly crawl the current web page to get the required data , This makes it difficult for the crawler to crawl directly .

Anti - reptile measures

1. Using agents IP

• For website detection IP Anti crawler policy for access , You can use agents IP. agent IP It is a way to obtain network information on behalf of users IP Address , It can help crawlers hide their true identities , breakthrough IP Restrictions on access , Hide the truth of the crawler IP, So as to avoid being prohibited by the anti - Crawler program of the website .
• requests Library implementations use agents IP Very convenient , Just construct a proxy IP Dictionary , And then send HTTP When asked , Use proxies Parameter add proxy IP Just use your dictionary . If you need to use multiple agents IP, All agents can be IP Dictionaries make up a list , Then randomly select the proxy from the list IP.