Internet worm

Catalog

What is a reptile

The role of reptiles

Classification of reptiles

Reptile process

---

What is a reptile

Crawlers can also be called web spiders , Web robot . Can simulate the client , Send web page request , Receive request response . It's a rule of thumb , A program that automatically captures Internet information .

The role of reptiles

• Data collection
• software test
• Network security
• Internet voting, etc

Classification of reptiles

• Universal crawler
  • The common search engine is the general crawler
• Focus on reptiles
  • It is used to specially grab a certain （ A certain category ） Web site data

Depending on whether it is for the purpose of obtaining data , Can be divided into ：

• Functional crawler

• Data incremental crawler

according to url Whether the page content corresponding to the address changes , Data incremental crawlers can be divided into :

• be based on url Address change , The content will also change with the data increment crawler

  • The new data

• url The address remains the same , Data incremental crawler with changing content

  • The data section changes

Reptile process

• Get one url

• towards url Send a request , And get the response ( need http agreement )

• If extracted from the response url, Then continue to send the request to get the response

• If you extract data from the response , Save the data