Crawler notes (1) - urllib

Concept

1. Through a program , according to url Crawl web pages , Get useful information
2. Using programs and browsers , To send a request to the server , Get response information

Reptile core

1. Crawl to the web
2. Parsing data
3. difficulty ： The game between reptiles and anti reptiles

Reptile uses

1. Data analysis / Manual data sets
2. Social software cold start
3. Public opinion monitoring
4. Competitor monitoring

Reptile classification

Universal crawler ：
1、 example ：360、 Baidu 、google Wait for the search engine
2、 function ： Access page 、 Fetching the data 、 data storage 、 Data processing 、 Provide retrieval services
3、robots agreement ： A conventional agreement , add to robots.txt file , Explain what content on this website can't be captured , Not limiting
4、 Website ranking ：
（1） according to pagerank Algorithm values are ranked
（2） Baidu bidding ranking
5. shortcoming
（1） Most of the captured data is useless
（2） Unable to accurately obtain data according to the needs of users

Focus on reptiles ：
1、 function ： According to the demand , Implement crawler , Grab the data you need
2、 Design thinking ：
（1） Make sure you want to climb url -> How to get url
（2） Simulate browser through http Agreement to access url, Get the information returned by the server html Code
（3） analysis html character string

Anti climbing means

1.user-Agent
The user agent , abbreviation UA, Is a special string header , Enables the server to identify the operating system and version used by the client 、CPU type 、 Browsers and versions 、 Browser rendering engine 、 Browser language 、 Browser plug-ins, etc

2. agent IP
Xici agency
Come on, agent
（1） Use transparent proxy , The other server can know that you are using the proxy , And I know your truth IP.
（2） Use anonymous proxy , The other server can know that you are using the proxy , But I don't know your truth IP.
（3） Use high anonymous proxy , The server doesn't know you're using a proxy , I don't know your truth IP.

3. Verification code access
Code platform 、 Cloud coding platform 、 Super

4. Loading web pages dynamically
The website returns js data , Not the real data of the web page ,selenium Drive real browsers to send requests

5. Data encryption
analysis js Code

urllib Library usage

python Bring their own

1 A type of 6 A way

print(type(response))
#HTTPResponse type 

content = response.read()
content = response.read(5)# return 5 Bytes 
content = response.readline()# Read a line 
content = response.readlines()# Read line by line , Until the end of reading 
print(response.getcode())# Return status code , If it is 200 It proves correct 
print(response.geturl())# return url Address 
print(response.getheaders())# Get status information 

download

urllib.request.urlretrieve(url,filename)
Picture copy address
The video is found in the web page check vedio Of src

Request object customization

url form
http 80/https 443
agreement host Port number route s Parameters wd
mysql 3306、oracle 1521、redis 6379、mongodb 27017

# Request object customization , because urlopen Cannot store dictionary in method , therefore headers Can't pass it out 
#  Parameter order problem , Add the variable name 
request = urllib.request.Request(url=url,headers=headers)

response = urllib.request.urlopen(request)

codec

get：
quote Method can change Chinese into ASCII code ,urllib.parse.quote(‘ Jay Chou ’)

urlencode Method ： When there are multiple parameters , Turn the dictionary into ASCII code , Add... Between two &

post：

#post The requested parameters must be encoded 
data = urllib.parse.urlencode(data).encode('utf-8')

#post The requested parameters are not spliced in url The back 
request = urllib.request.Request(url=url,data=data,headers=headers)

response = urllib.request.urlopen(request)
content = response.read().decode('utf-8')
print(content)

1.post Parameters for request mode You have to code urlencode
2. After coding Must call encode Method
3. Parameters are placed in the method customized by the request object

Pictured , Baidu translation is a detailed translation .

cookie Very important .

ajax Of get request

Top of the movie rankings 10 page

ajax Of post request

See the picture below Ajax request

Exception in crawler

URLError

HTTPError yes URLError Subclasses of

cookie land

After logging in cookie You can go to any page

handler The processor basically uses

agent

Agent pool

proxies_pool=[
    {
    'http':'118.24.219.151:1681711111'},
    {
    'http':'118.24.219.151:1681722222'},
]