Crawl IP

1. User agent tools

1).fake-useragent install

pip install fake-useragent

2). Usage method

from fake_useragent import UserAgent # Import package 
ua = UserAgent() # Instantiation 

ua.ie  #  Random generation IE Browser's UserAgent

ua.firefox  # Random generation firefox Browser's UserAgent

ua.chrome  # Random generation chrome Browser's UserAgent

ua.safari  # Random generation safari Browser's UserAgent

ua.google  # Random generation google Browser's UserAgent

 ua.random  # Random generation of various browsers UserAgent

3). Example of camouflage head

from fake_useragent import UserAgent
ua = UserAgent()
headers = {
    'User-Agent':ua.random}

4).

If you do not want to cache the database or do not need a writable file system .

ua = UserAgent(cache=False)

5).

If you don't want to use a hosted cache server , You can disable the server cache

ua = UserAgent(use_cache_server=False)

6).

If the above methods report errors , perform

ua = UserAgent(verify_ssl=False)

because fake-useragent Library maintenance user-agent The list is stored on the online web page
List pages that are too low version dependent may report 403

7). to update

ua.update()

8). Check all user-agent

ua.data_browsers

In [20]: len(reduce(lambda a, b: a+b, ua.data_browsers.values()))
Out[20]: 250

# Get random values 
In [26]: ua.random
Out[26]: 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/31.0.1650.16 Safari/537.36'

9). see useragent The contents of the library

You can visit the website ：
https://fake-useragent.herokuapp.com/browsers/ Version number

Reference website

python Reptiles —— Build agents ip pool
python Crawler settings proxy ip pool —— Method （ One ）
Python Reptiles ——requests ip agent