Selenium runs slowly - speed up by setting selenium load policy

One 、 Preface

There are many fans , Even some friends with some experience in the industry say selenium Poor performance 、 Very slowly , It takes half a day to open a web page . Novice fans ask this question, I can understand , I'm a little surprised to say that after years of working experience in the industry . It seems that many little friends still don't have the habit of reading official documents , Here's a brief explanation of why you feel selenium Slow and how to solve this problem ：

When we go through webdriver（ For example, start Google browser ：webdriver.Chrome()） When you start the browser , Will start one without any cache 、cookie Browser . This time through driver.get(“https://xxx”) It's naturally slow to visit the page , Because it needs to load the resources of the page , If its picture 、 style 、js When the document is too large , This time it will become slower .

It's the same reason that we usually clear the browser cache and then access it , This is not Selnium Its performance is defective .

But at this time, the little friends must have doubts , Even if the browser cache is cleared, access again , It didn't pass Selnium Come and run so slowly ！ All the buttons you want to click have appeared for half a day , It doesn't even click , Isn't that slow ？

This is actually the same as Selenium Page loading policy .

Selnium Page loading strategy （pageLoadStrategy）

1. normal： Wait for the entire page to load before starting the operation
2. eager： Waiting for the whole dom Tree loading complete , namely DOMContentLoaded This event is complete , As long as HTML After full loading and parsing, the operation begins . Give up waiting for pictures 、 style 、 Subframe loading .
3. none： wait for html Download complete , Even if the operation starts before parsing .
   By default , When Selenium WebDriver
   When loading the page , It follows normal Load policy , So it will cause the page to load too slowly , Especially in pictures 、 When files such as styles are too large , The slow is especially obvious .

So we can adjust according to the actual situation Selenium Page loading strategy to shorten the waiting time , Improve execution speed .

The loading policy is set to eager

The configuration code is as follows ：

chrome_options = Options()
chrome_options.page_load_strategy = 'eager'
driver = webdriver.Chrome(options=chrome_options)

The loading policy is set to none

The loading policy is set to none, And introduce retry Try again （ The purpose is to prevent error reporting , Of course, setting implicit waiting can also , But no retry Err on the side of ） It can be used only 2s It can be executed around , The complete code is as follows ：

import datetime
from retrying import retry  #  Third party library required , Need to be pip Installation 
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

@retry(wait_fixed=10, stop_max_attempt_number=1)
def click(path):
    driver.find_element(By.XPATH, path).click()
    
chrome_options = Options()
chrome_options.page_load_strategy = 'none'
driver = webdriver.Chrome(options=chrome_options)
start_time = datetime.datetime.now()
driver.get('https://www.iqiyi.com/')
click('//*[@id="block-C"]/div/div/div/div[1]/div[1]/div/div[1]/div/div/a/span[2]')
end_time = datetime.datetime.now()
print(end_time - start_time)

3、 ... and 、 summary
therefore selenium Not slow , It's just that you don't choose the right loading strategy and mistakenly think it's slow .

When we encounter problems that are difficult to understand or confused , You should look at the official documents as soon as possible , Maybe it already has a solution .

————————————————
Link to the original text ：https://blog.csdn.net/m0_67393619/article/details/123321768