Reptile: from introduction to imprisonment (I) -- Concept

What is a reptile ？

I believe you have heard of it for a long time , But I don't know what it is , Let's next define ！

Web crawler technology ：

By programming , Analog browser internet , Then let the program automatically grab data on the Internet according to the set rules

The value of reptiles ：

You can grab a lot of useful data from the Internet , For my use , Reuse data analysis , Generate commercialized or productized value  

Employment Perspective ：

With the advent of the big data era , High salary and large personnel gap  

Demonstration of the legitimacy of reptiles  

Crawler technology is a tool , It's neutral , Therefore, it is not prohibited in the law

The behavior with illegal risk is illegal

So reptiles are divided into ： Friendly reptiles and malicious reptiles

Malicious reptiles ：

1. The crawler interferes with the normal operation of the visited website

2. The act of crawling through legally protected data

Fooling around is easy to enter “ a mandarin orange ”！！！！

Pay attention to optimizing your code or logic at any time , Avoid interfering with the operation of the website as the website is updated

Please review the data you crawl in time , If the data violates （ privacy , Business sensitivity or other things that cannot be said ）, Be sure to delete ！！！

Classification of reptiles in use scenarios
    - Universal crawler ：
        Grab an important part of the system . Grabbing the whole page data .
    - Focus on reptiles ：
        It's based on a universal reptile . What we grab is the specific partial content of the page .
    - Incremental reptiles ：
        Detect data updates in the website . It will only grab the latest updated data in the website .

There must be a reptile reaction mechanism ！

Portal site , Can pass Formulate corresponding strategies or technical means , Prevent crawlers from crawling the website data

  Of course , The law is strong , And anti Anti climbing technology , Ha ha ha （ Always competing ）

Anti-crawl strategy ：
    Crawler programs can be developed through relevant strategies or technical means , Crack the anti crawling mechanism in portal website , So you can get the relevant data in the portal website .