Robots protocol

When we use crawlers to crawl data , Some websites will not block , But some websites will not easily give you the data to crawl , So there it is Anti climbing mechanism . Then we want to know what to do with data with anti crawl mechanism , So there was Anti-crawl strategy .

Anti climbing mechanism

Portal websites can formulate corresponding strategies and technical means , Prevent crawlers from crawling website data .

Anti-crawl strategy

The crawler program develops corresponding strategies and technical means , Crack the anti crawling mechanism in portal website , So you can get the data of the website .

Their relationship is like spear and shield , After that, we will be exposed to many anti climbing mechanisms , You will also learn a lot of anti climbing strategies . Now let's learn the simplest anti crawl mechanism robots Agreement is also called Gentleman agreement .

robots.txt agreement

It stipulates which data in the website can be crawled and which data cannot be crawled . But no defensive measures were taken , Whether you comply or not depends on yourself , So it is also called Gentleman agreement .

Take Baidu as an example to open https://www.baidu.com/robots.txt  You can see Baidu robots agreement .