Python：pycharm爬微博博主博文及转赞评数量_用pycharm爬取微博转赞评

萌新们刚开始爬微博数据一定要用微博手机端的网页https://m.weibo.cn/
用谷歌浏览器，先分析网站结构
进入博主主界面，右键，然后点击检查，点击Network

点击Headers container和type在这里找
然后preview看网站结构
data，cards，mblog以及要爬取的数据都在这里

import requests
import pprint
from urllib.parse import urlencode
from pyquery import PyQuery
import json


base_url='https://m.weibo.cn/api/container/getIndex?'

#代码块 请求的方法
#根据页数获取数据
def get_page(page):

    prames ={
        'containerid':'1076033937348351',
        'value':'3937348351',
        'page':page
    }
    response = requests.get(base_url+urlencode(prames))
    return response.json()

#解析数据
def prase_data(res_json):
    if res_json.get('data'):
        for node in res_json['data']['cards']:
            item = dict()
            item['text'] = PyQuery(node['mblog']['text']).text()
            item['id'] = node['mblog']['id']
            item['screen_name'] = node['mblog']['user']['screen_name']
            item['attitudes_count'] = node['mblog']['attitudes_count']
            item['comments'] = node['mblog']['comments_count']
            item['reposts_count'] = node['mblog']['reposts_count']
            print(item)



def main():
    for page  in range(1,20):
        res_jsom = get_page(page)
        prase_data(res_jsom)
if __name__ == '__main__':
    main()

#get_page()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

爬取结果，下篇介绍如何导入Excel中