Experience in developing large crawlers in requests Library

Order of magnitude ：100 m html page

1. In the early , If you can't judge by yourself, catch more exceptions , There are also mistakes in some inconspicuous places .

2. Determine the magnitude in the early stage , Take protective measures against the interruption of reptiles for various reasons , Reduce the phenomenon of multiple crawls of a page

3. In the early stage, you can use manual multi process debugging （ See if there is a problem with the account ）, Turn to multithreading after stabilization

4. Program development nonsense ： High cohesion , Low coupling , Think clearly before writing , Nonsense is really important