Simple use of robobrowser

robobrowser Brief introduction ：

Simply speaking ,robobrowser Is a lightweight browser , Automatic test library , and selenium similar , But the operation ratio selenium More hidden , Because it won't open the browser page .RoboBrowser,Your friendly neighborhood web scraper！ From pure Python To write ;
Project address ： open

install

Recommended pip Way to install , Simple ！
robobrowser Libraries depend on other libraries ,bs4,lxml, So both libraries should be installed ;
Open the command line window , Enter the code ：

pip install lxml
pip install bs4
pip install robobrowser

Wait for the installation to succeed ;

Check it out ：
Open the command line window , Input python Go to the interpreter window ;
Enter the code ：import robobrowser

If the following error is prompted

File “D:\python3.8\lib\site-packages\robobrowser\browser.py”, line 8, in <module>
from werkzeug import cached_property
ImportError: cannot import name ‘cached_property’ from ‘werkzeug’ (D:\python3.8\lib\site-packages\werkzeug_init_.py)

You need to open the file ：
D:\python3.8\lib\site-packages\werkzeug_init_.py
Input from werkzeug.utils import cached_property You can fix it

from .serving import run_simple as run_simplefrom .test import Client as Clientfrom .wrappers import Request as Requestfrom .wrappers import Response as Response# Show the introduction from werkzeug.utils import cached_property #  This is the package introduced  __version__ = "2.1.2"

Easy to use ;

Simulating Baidu search , Submit Form ：

home_url = 'http://www.baidu.com' #  parser:  Parser ,HTML parser; used by BeautifulSoup#   The official recommendation ：lxmlrb = RoboBrowser(history=True, parser='lxml') #  Open the target website rb.open(home_url)#print(rb.parsed())#  Get form object bd_form = rb.get_form()print(bd_form)bd_form['wd'].value = "robobrowser"#  Submit Form , Simulate a search rb.submit_form(bd_form)#print(rb.parsed())sleep(1)#  View results result_elements = rb.select(".result")print(result_elements)

The results obtained above , have access to

from bs4 import BeautifulSoup

To analyze ;
For example, get the title , Links, etc. ;
for index, element in enumerate(result_elements):
title = element.find(“a”).text
href = element.find(“a”)[‘href’]

Other operating

 Jump Links rb.follow_link(first_href) Get history print(rb.url)

More operations , Please refer to the official documents ;