How to view and explain robots protocol

The first thing to learn about reptiles is to know what can't be crawled . So let's get to know robots Make an agreement .

robots Where is the agreement ？
Add... Directly after the target site /robots.txt You can see , With csdn For example
visit https://www.csdn.net/robots.txt, Get the following information ：

User-agent: *
Disallow: /scripts
Disallow: /public
Disallow: /css/
Disallow: /images/
Disallow: /content/
Disallow: /ui/
Disallow: /js/
Disallow: /scripts/
Disallow: /article_preview.html*
Disallow: /tag/
Disallow: /?
Disallow: /link/
Sitemap: https://www.csdn.net/sitemap-aggpage-index.xml
Sitemap: https://www.csdn.net/article/sitemap.txt

User-agent: * It means that all reptiles should follow the following rules
disallow： Is not allowed to crawl the page
Yes, there will be Allow
Other websites may also have various interesting agreements , Go and see for yourself ！