Crawler career from scratch (II): crawl the photos of my little sister ② (the website has been disabled)

Preface

Start with this article , We will crawl through several articles in a row （url ：https://imoemei.com/） All the pictures of my little sister . With this example, let's learn simple python Reptiles .
See related articles

A reptilian career from scratch （ One ）： Crawling for a picture of my little sister ①

A reptilian career from scratch （ 3、 ... and ）： Crawling for a picture of my little sister ③

---

Thought analysis

1. Page source analysis

Because last time we have climbed down all the pictures of the little sister on a page , So now we just need to get the of each page url, Then climb every page again OK 了

Do as you say ！
First, let's check the source code of the page

Find a url, Go in and have a look

The result is just the cover ... Look again , I found it on the cover just now ！ wuhu ！

Observe the code of the whole page
Of all pages url Just put it here li In block
Then we just need to take out each page's url It's done ！

2. Reptilian thinking

Direct use request Get the whole page , Reuse BeatutifulSoup Parse web pages , Take out all page links , Then traverse the link , Save the picture according to the method in the previous article .

The crawler code

1. development environment

development environment ：win10 python3.6.8
Using tools ：pycharm
Using third party libraries ：requests、os、BeatutifulSoup

2. Code decomposition

（1）. Import and stock in

import requests
import os
from bs4 import BeautifulSoup

（2） Get the address of each page

target_url = "https://imoemei.com/zipai/"
r = requests.get(url=target_url)
html = BeautifulSoup(r.text, 'html5lib')
b2_gap = html.find('ul', class_='b2_gap')
print(str(1) + "page is OK")
img_main = b2_gap.find_all('a', class_='thumb-link')
img_main_urls = []
for img in img_main:
    img_main_url = img.get('href')
    img_main_urls.append(img_main_url)

（3）. Get the address of each picture

for j in range(len(img_main_urls) + 1):
    print(img_main_urls[j])
    r = requests.get(url=img_main_urls[j])

    html = BeautifulSoup(r.text, 'html5lib')
    entry_content = html.find('div', class_='entry-content')
    img_list = entry_content.find_all('img')
    img_urls = []
    num = 0
    name = html.find('h1').text
    print(name)
     for img in img_list:
        img_url = img.get('src')
        result = requests.get(img_url).content

（4）. Save the picture to the specified folder

		path = ' picture '
        if not os.path.exists(path):
            os.mkdir(path)

        f = open(path + '/' + name + str(num) + '.jpg', 'wb')
        f.write(result)
        num += 1
        print(' Downloading {} The first {} A picture '.format(name, num))

3. The overall code

import requests
import os
from bs4 import BeautifulSoup

target_url = "https://imoemei.com/zipai/"
r = requests.get(url=target_url)
html = BeautifulSoup(r.text, 'html5lib')
b2_gap = html.find('ul', class_='b2_gap')
print(str(1) + "page is OK")
img_main = b2_gap.find_all('a', class_='thumb-link')
img_main_urls = []

for img in img_main:
    img_main_url = img.get('href')
    img_main_urls.append(img_main_url)

for j in range(len(img_main_urls) + 1):
    print(img_main_urls[j])
    r = requests.get(url=img_main_urls[j])

    html = BeautifulSoup(r.text, 'html5lib')
    entry_content = html.find('div', class_='entry-content')
    img_list = entry_content.find_all('img')
    img_urls = []
    num = 0
    name = html.find('h1').text
    print(name)

    for img in img_list:
        img_url = img.get('src')
        result = requests.get(img_url).content

        path = ' picture '
        if not os.path.exists(path):
            os.mkdir(path)

        f = open(path + '/' + name + str(num) + '.jpg', 'wb')
        f.write(result)
        num += 1
        print(' Downloading {} The first {} A picture '.format(name, num))

Crawling results

This time, I only climbed the first page under the selfie item , In the next article, we will crawl all pages of all sub items , Coming soon .