Selenium: detailed explanation of browser crawler use (I)

brief introduction ：

Direct online search Selenium You'll know . In short Selenium Realize the automatic operation of the browser by simulating the operation of the browser . The difference between it and other reptiles is , Other crawlers simulate the browser to send requests , and Selenium It is the simulator that operates the browser .

Realization Selenium Things needed

1、 install Selenium

Use it directly pip Command to install , It is suggested to add domestic sources faster .

pip install selenium -i https://pypi.tuna.tsinghua.edu.cn/simple

You may encounter many fucking problems during installation （ Baidu will solve it then ）

such as ：

selenium 4.3.0 requires urllib3[secure,socks]~=1.26, but you have urllib3 1.24.3 which is incompatible

requests 2.21.0 requires urllib3<1.25,>=1.21.1, but you have urllib3 1.26.10 which is incompatible

Then I installed the version

requests==2.28.1

urllib3==1.26.10

selenium==4.3.0

2、 Install browser driver

View your browser version , Find the driver of your browser version on the Internet and download .

Take Google browser for example ：

Find the download address of the corresponding browser driver on the Internet ：

Google ： http://chromedriver.storage.googleapis.com/index.html

Be careful ： There is no identical version to choose from , No, 64 Bits can be selected 32 Bit ,64 Bit compatible 32 position .

 

Unzip the driver into a folder , Then add the storage path to the Path variable .

Add the running startup software path of Google browser to the system environment variable Path variable .

It is recommended to restart the computer after modifying the environment variables , Otherwise, the setting may not take effect .

  Install the test

from selenium import webdriver
import time

#  Create browser driven objects ,Chrome() The representative is the driver of Google browser 
driver = webdriver.Chrome()
#  Visit website 
driver.get('https://blog.csdn.net/m0_72501048?type=blog')
# Program pause 10 second 
time.sleep(10)
# Output page title
print(driver.title)
#  Close the current page 
driver.close()
#  Close the browser driver object 
driver.quit()