How is crawler data collected and organized?

Some users have been curious about how crawlers identify how the crawler data on the website is organized, and today we will come to youUncover how crawler data is collected and organized.

Get the rDNS method by Query IP Address

We can reverse lookup rDNS through the crawler's IP address, for example: we use Reverse DNS lookup toolFind this IP: 116.179.32.160 , rDNS is: baiduspider-116-179-32-160.crawl.baidu.com

It can be roughly judged from the above that it should be Baidu search engine spider.Since the Hostname can be faked, we only have a reverse lookup, which is still inaccurate.We also need to search forward. We use the ping command to find whether baiduspider-116-179-32-160.crawl.baidu.com can be parsed as: 116.179.32.160, we can see baiduspider-116-179-32 from the following figure-160.crawl.baidu.com is resolved to the IP address of 116.179.32.160, the description is Baidu search engine crawler for sure.

Find by ASN related information

Not all crawlers abide by the above regulations. Most crawlers have no results in reverse search. We need to query the ASN information of the IP address to determine whether the crawler information is correct.

Example: This IP is 74.119.118.20, we pass Query IP Information, you can see that this IP address is the IP address of Sunnyvale, California, USA.

From the ASN information we can see that he is the IP of Criteo Corp.

The above screenshot shows the record information of critieo crawler through log records.The yellow part is its User-agent, followed by its IP, there is nothing wrong with this record (this IP is indeed the IP address of CriteoBot).

IP address range published through the official documentation of the crawler

Some crawlers will publish IP address segments, and we will save the official crawlers IP address segments directly to the database, which is a simple and fast method.

By public log

We can often view public logs on the Internet. For example, the following image is the public log record I found:

We can parse the log records and judge which ones are crawlers and which ones are visitors according to the User-agent, which greatly enriches our crawler record database.

Summary

Through the above four methods, it explains in detail how the crawler identification website collects and organizes crawler data, and how to ensure the accuracy and reliability of crawler data. Of course, in the actual operation process, not only the above four methods, but all of them are used.It is relatively small, so I will not introduce it here.