Crawler case 05 - parsing websites using XPath

Use in this case Xpath Parsing the source code , To get what we want

1. Get web source

import requests
from lxml import etree
# Enter keywords and click search , The URL after jump is our target URL 
url = 'https://beijing.zbj.com/search/f/?kw=python%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90'
headers = {
    
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0"
}
resp = requests.get(url,headers = headers)
print(resp.text)

2. take HTML Preload the source code

html = etree.HTML(resp.text) # take HTML Preload the source code 

3. Get content

3.1 Find the location of what you need

3.2 Copy path

Get the following path ：
/html/body/div[6]/div/div/div[2]/div[5]/div[1]/div

3.3 Use Xpath Expression to get the content

# Get every service provider's div
divs = html.xpath("/html/body/div[6]/div/div/div[2]/div[5]/div[1]/div")
for div in divs:
	  # The full path 
    #prices = div.xpath("./div/div/a[2]/div[2]/div[1]/span[1]/text()")
    # according to div Of class Property to quickly locate the location 
    price = div.xpath(".//div[@class='service-price clearfix']/span[1]/text()")[0].strip("¥") # Remove the... In front of the price ￥ Symbol 
    title = "".join(div.xpath(".//div[@class='service-title']/p//text()")) 
    company_name = div.xpath("./div/div/a[1]/div/p/text()")[1].strip() # There is a newline in the list , Take the second item in the list , And remove line breaks 
    loc = div.xpath("./div/div/a[1]/div/div/span/text()")
    print(price)
    print(title)
    print(title)
    print(company_name)