Crawler career from scratch (IV): climb the bullet curtain of station B through API

Preface

In this article, we will use b Stations api To crawl B The barrage of station video , This will be the fourth article in this series .

---

Display the page that needs to be crawled

Thought analysis

Reptilian thinking

There is nothing to analyze , Is to call the barrage pool interface found on the Internet , and bv No. 1 to the interface of barrage pool number .

The crawler code

1. development environment

development environment ：win10 python3.6.8
Using tools ：pycharm
Using third party libraries ：requests、os、BeatutifulSoup

2. Code decomposition

（1）. Import and stock in

import requests
import json
from bs4 import BeautifulSoup
import re

（2）. obtain cid（ Barrage pool number ）

def bvid2cid(bvid):  #  Get video cid
    url = "https://api.bilibili.com/x/player/pagelist?bvid=" + str(bvid) + "&jsonp=jsonp"
    r = requests.get(url)
    dirt = json.loads(r.text)
    cid = dirt['data'][0]['cid']
    return cid

（3）. Bullet screen comments

def cid2data(cid):
    url = 'https://api.bilibili.com/x/v1/dm/list.so?oid=' + str(cid)
    r = requests.get(url=url)
    r.encoding = 'utf-8'
    html = BeautifulSoup(r.text, 'html5lib')
    ds = html.find_all('d')
    said = '.*">(.*)</d>.*'
    for d in ds:
        with open(str(cid) + '.txt', 'a', encoding='utf-8') as f:
            f.write(re.findall(said, str(d))[0] + '\n')

3. The overall code

import requests
import json
from bs4 import BeautifulSoup
import re


def bvid2cid(bvid):  #  Get video cid
    url = "https://api.bilibili.com/x/player/pagelist?bvid=" + str(bvid) + "&jsonp=jsonp"
    r = requests.get(url)
    dirt = json.loads(r.text)
    cid = dirt['data'][0]['cid']
    return cid


def cid2data(cid):
    url = 'https://api.bilibili.com/x/v1/dm/list.so?oid=' + str(cid)
    r = requests.get(url=url)
    r.encoding = 'utf-8'
    html = BeautifulSoup(r.text, 'html5lib')
    ds = html.find_all('d')
    said = '.*">(.*)</d>.*'
    for d in ds:
        with open(str(cid) + '.txt', 'a', encoding='utf-8') as f:
            f.write(re.findall(said, str(d))[0] + '\n')


cid = bvid2cid('BV1gp4y1e7cE')
cid2data(cid)

Crawling results

You can see , This climb was very successful