devbox
木道寻08
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Python 爬虫总结——案例代码_python爬虫案例代码

作者：木道寻08 | 2024-07-22 17:51:22

赞

踩

python爬虫案例代码

目录

request的基本使用

urllib使用

图片爬取

获取动态数据

session和cokkie的处理

使用xpath解析

使用正则解析

BeautifulSoup使用

seleium自动化爬虫

其他自动化操作

实现无界面

自动化处理iframe标签

基于selenium的12306用户登录

代理的使用

验证码解析

协程的使用

同步爬虫

多线程异步爬虫的使用

 线程池

异步协程

aiohttp实现任务异步协程

分布式爬虫

 简单练手项目

肯德基破解 

爬取简历模板

百度AI实现爬虫

好久之前做的python非框架爬虫全集笔记一直没整理，今天有空整理了一番，方便以后查看。

request的基本使用

案例一

  1. # -*- coding: utf-8 -*-
  2. import requests
  3.  
  4. if __name__ == "__main__":
  5.     # step 1:指定url
  6.     url = 'https://www.sogou.com/'
  7.     # step 2:发起请求
  8.     response = requests.get(url=url)
  9.     # step 3:获取响应数据.text返回的是字符串形式的响应数据
  10.     page_text = response.text
  11.     print(page_text)
  12.     # step_4:持久化存储
  13.     with open('./sogou.html', 'w', encoding='utf-8') as fp:
  14.         fp.write(page_text)
  15.     print('爬取数据结束！！！')

案例二：

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. import json
  4.  
  5. if __name__ == "__main__":
  6.     url = 'https://movie.douban.com/j/search_subjects'
  7.     param = {
  8.         'type': 'movie',
  9.         'tag': "喜剧",
  10.         'sort': 'recommend',
  11.         'page_limit': 20,  # 一次取出的个数
  12.         'page_start': 20,  # 从库中的第几部电影去取
  13.     }
  14.     headers = {
  15.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.75'
  16.     }
  17.     response = requests.get(url=url, params=param, headers=headers)
  18.  
  19.     list_data = response.json()
  20.  
  21.     fp = open('./douban.json', 'w', encoding='utf-8')
  22.     json.dump(list_data, fp=fp, ensure_ascii=False)
  23.  
  24.     print('over!!!')

案例三

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. import json
  4.  
  5. if __name__ == "__main__":
  6.     # 1.指定url
  7.     post_url = 'https://fanyi.baidu.com/sug'
  8.     # 2.进行UA伪装
  9.     headers = {
  10.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.75'
  11.     }
  12.     # 3.请求参数处理（同get请求一致）
  13.     word = input('enter a word:')
  14.     data = {
  15.         'kw': word
  16.     }
  17.     # 4.请求发送
  18.     response = requests.post(url=post_url, data=data, headers=headers)
  19.     # 5.获取响应数据:json()方法返回的是obj_(如果确认响应数据是json类型的，才可以使用json())
  20.     dic_obj = response.json()
  21.  
  22.     # 持久化存储
  23.     fileName = word + '.json'
  24.     fp = open(fileName, 'w', encoding='utf-8')
  25.     json.dump(dic_obj, fp=fp, ensure_ascii=False)
  26.  
  27.     print('over!!!')

案例四

  1. # -*- coding: utf-8 -*-
  2. # 每次爬取需要进行UA伪装，伪装成某款浏览器
  3. # User-Agent(请求载体生份标识)
  4. import requests
  5.  
  6. if __name__ == "__main__":
  7.     # UA伪装：将对应的User-Agent封装到一个字典中
  8.     headers = {
  9.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.75'
  10.     }
  11.     url = 'https://www.sogou.com/web'
  12.     # 处理url携带的参数：封装到字典中
  13.     kw = input('enter a word:')
  14.     param = {
  15.         'query': kw
  16.     }
  17.     # 对指定的url发起的请求对应的url是携带参数的，并且请求过程中处理了参数
  18.     response = requests.get(url=url, params=param, headers=headers)
  19.  
  20.     page_text = response.text
  21.     fileName = kw + '.html'
  22.     with open(fileName, 'w', encoding='utf-8') as fp:
  23.         fp.write(page_text)
  24.     print(fileName, '保存成功！！！')

urllib使用

  1. import requests
  2. import re
  3. import os
  4. import urllib
  5.  
  6. dirName = "imgLab"
  7. if not os.path.exists(dirName):
  8.     os.mkdir(dirName)
  9. url = "https://www.baidu.com/s?wd=%E7%8B%97&tn=98012088_5_dg&ch=11"
  10. headers = {
  11.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36 Edg/91.0.864.37',
  12. }
  13. response = requests.get(url=url, headers=headers)
  14. page_text = response.text
  15. ex = '<div class="op-img-address-divide-high">.*?<img src="(.*?)" class=.*?</div>'
  16. img_src_list = re.findall(ex, page_text, re.S)
  17. for src in img_src_list:
  18.     imgPath = dirName + "/" + src.split('/')[-1]
  19.     src = src + '&fm=26'
  20.     urllib.request.urlretrieve(src, imgPath)
  21.     print(imgPath, '下载成功！')

图片爬取

案例一

  1. from lxml import etree
  2. import requests
  3. import os
  4. import urllib
  5.  
  6. fileName = "图片"
  7. if not os.path.exists(fileName):
  8.     os.mkdir(fileName)
  9.  
  10. url = "https://pic.netbian.com/"
  11. headers = {
  12.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36'
  13. }
  14. response = requests.get(url=url, headers=headers)
  15. page_text = response.text
  16. tree = etree.HTML(page_text)
  17. li_list = tree.xpath('//*[@id="main"]/div[3]/ul/li')
  18. arr = []
  19. for li in li_list:
  20.     href = 'https://pic.netbian.com' + li.xpath(' ./a/span/img/@src')[0]
  21.     arr.append(href)
  22.  
  23. for ar in arr:
  24.     filePath = fileName + '/' + ar.split('/')[-1]
  25.     urllib.request.urlretrieve(ar, filePath)
  26.  
  27. print("爬取结束！！！")

案例二

  1. # -*- coding: utf-8 -*-
  2. import requests
  3.  
  4. if __name__ == "__main__":
  5.     # 如何爬取图片数据
  6.     url = 'https://th.bing.com/th/id/R6706ad2e7a68edabddbc1b5644707c4f?rik=u8uR%2bWe5bxIosA&riu=http%3a%2f%2fpic.lvmama.com%2fuploads%2fpc%2fplace2%2f2016-09-14%2f9aab9bb7-2593-4ca6-8c5a-31355443aebc.jpg&ehk=HpOwqU6w6%2fssF4CJQMbTOshMh4lIXJONXU%2btYNsAKSI%3d&risl=1&pid=ImgRaw'
  7.     # content返回的是二进制形式的图片数据
  8.     # text(字符串) content(二进制) json() (对象)
  9.     img_data = requests.get(url=url).content
  10.  
  11.     with open('./qiutu.jpg', 'wb') as fp:
  12.         fp.write(img_data)

获取动态数据

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. import json
  4.  
  5. if __name__ == "__main__":
  6.     headers = {
  7.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.75'
  8.     }
  9.     # 批量获取不同企业的id值
  10.     url = ''
  11.     # 参数的封装
  12.     id_list = []  # 存储企业的id
  13.     all_data_list = []  # 存储所有的企业详情数据
  14.     for page in range(1, 6):
  15.         page = str(page)
  16.         data = {
  17.         }
  18.         json_ids = requests.post(url=url, headers=headers, data=data).json()
  19.         for dic in json_ids['list']:
  20.             id_list.append(dic['ID'])
  21.  
  22.     # 获取所有的企业详情数据
  23.     post_url = ''
  24.     for id in id_list:
  25.         data = {
  26.             'id': id
  27.         }
  28.         detail_json = requests.post(url=url, headers=headers, data=data).json()
  29.         all_data_list.append(detail_json)
  30.     # 持久化存储all_data_list
  31.     fp = open('./allData.json', 'w', encoding='utf-8')
  32.     json.dump(all_data_list, fp=fp, ensure_ascii=False)
  33.     print('over!!!')

session和cokkie的处理

使用xpath解析

 

案例一

  1. # -*- coding: utf-8 -*-
  2. from lxml import etree
  3.  
  4. if __name__ == '__main__':
  5.     # 实例化好了一个etree对象，且将被解析的源码加载到了该对象中
  6.     tree = etree.parse('r.html')
  7.     # r=tree.xpath('/html/body/div')
  8.     # r=tree.xpath('/html//div')
  9.     # r=tree.xpath('//div')
  10.     # r=tree.xpath('//div[@class="song"]')
  11.     # r=tree.xpath('//div[@class="tang"]//li[5]/a/text()')[0]
  12.     # r=tree.xpath('//li[7]//text()')
  13.     # r=tree.xpath('//div[@class="tang"]//text()')
  14.     r = tree.xpath('//div[@class="song"]/img/@src')
  15.  
  16.     print(r)

 案例二

  1. # -*- coding: utf-8 -*-
  2. # 需求： 爬取58二手房的房源信息
  3. import requests
  4. from lxml import etree
  5.  
  6. if __name__ == "__main__":
  7.     headers = {
  8.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/53'
  9.     }
  10.     # 爬取到页面源码数据
  11.     url = 'https://www.58.com/ershoufang/'
  12.     page_text = requests.get(url=url, headers=headers).text
  13.  
  14.     # 数据解析
  15.     tree = etree.HTML(page_text)
  16.     # 存储的就是标签对象
  17.     td_list = tree.xpath('//td[@class="t"]')
  18.     fp = open('58.txt', 'w', encoding='utf-8')
  19.     for td in td_list:
  20.         title = td.xpath('./a/text()')[0]
  21.         print(title)
  22.         fp.write(title + '\n')
  23.     fp.close()

案例三

  1. # -*- coding: utf-8 -*-
  2. # 需求： 解析下载图片数据
  3. import requests
  4. from lxml import etree
  5. import os
  6.  
  7. if __name__ == "__main__":
  8.     headers = {
  9.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/53'
  10.     }
  11.     # 爬取到页面源码数据
  12.     url = 'https://pic.netbian.com/4kyouxi/'
  13.     response = requests.get(url=url, headers=headers)
  14.     # 手动设定响应数据编码格式
  15.     # response.encoding='utf-8'
  16.     page_text = response.text
  17.  
  18.     # 数据解析
  19.     tree = etree.HTML(page_text)
  20.     li_list = tree.xpath('//div[@class="slist"]/ul/li')
  21.  
  22.     # 创建一个文件夹
  23.     if not os.path.exists('./picLibs'):
  24.         os.mkdir('./picLibs')
  25.     for li in li_list:
  26.         img_src = 'http://pic.netbian.com' + li.xpath('./a/img/@src')[0]
  27.         img_name = li.xpath('./a/img/@alt')[0] + '.jpg'
  28.         # 通用解决中文乱码方案
  29.         img_name = img_name.encode('iso-8859-1').decode('gbk')
  30.  
  31.         # print(img_name,img_src)
  32.         # 请求图片进行持久化存储
  33.         img_data = requests.get(url=img_src, headers=headers).content
  34.         img_path = 'picLibs/' + img_name
  35.         with open(img_path, 'wb') as fp:
  36.             fp.write(img_data)
  37.             print(img_name, '下载成功！！！')

案例四

  1. # -*- coding: utf-8 -*-
  2. # 需求： 解析出所有城市名称
  3. import requests
  4. from lxml import etree
  5.  
  6. if __name__ == "__main__":
  7.     '''headers={
  8.         'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/53'
  9.     }
  10.     #爬取到页面源码数据
  11.     url='https://www.aqistudy.cn/historydata/'
  12.     response=requests.get(url=url,headers=headers)
  13.     #手动设定响应数据编码格式
  14.     #response.encoding='utf-8'
  15.     page_text=response.text
  16.     
  17.     #数据解析
  18.     tree=etree.HTML(page_text)
  19.     host_li_list=tree.xpath('//div[@class="bottom"]/ul/li')
  20.     all_city_names=[]
  21.     #解析到热门城市的城市名称
  22.     for li in host_li_list:
  23.         hot_city_name=li.xpath('./a/text()')[0]
  24.         all_city_names.append(hot_city_name)
  25.     
  26.     #解析全部城市的名称
  27.     city_names_list=tree.xpath('div[@class="bottom"]/ul/div[2]/li')
  28.     for li in city_names_list:
  29.         city_name=li.xpath('./a/text()')[0]
  30.         all_city_names.append(city_name)
  31.     
  32.     print(all_city_names,len(all_city_names))'''
  33.     headers = {
  34.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/53'
  35.     }
  36.     # 爬取到页面源码数据
  37.     url = 'https://www.aqistudy.cn/historydata/'
  38.     response = requests.get(url=url, headers=headers)
  39.     # 手动设定响应数据编码格式
  40.     # response.encoding='utf-8'
  41.     page_text = response.text
  42.  
  43.     # 数据解析
  44.     tree = etree.HTML(page_text)
  45.     a_list = tree.xpath('//div[@class="bottom"]/ul/li/a | //div[@class="bottom"]/div[2]/li/a')
  46.     all_city_names = []
  47.     for a in a_list:
  48.         city_name = a.xpath('./text()')[0]
  49.         all_city_names.append(city_name)
  50.     print(all_city_names, len(all_city_names))

使用正则解析

案例一

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. import re
  4. import os
  5.  
  6. # 需求： 爬取糗事百科中糗图板块下所有的糗图图片
  7. if __name__ == '__main__':
  8.     # 创建一个文件夹，保存所有的图片
  9.     if not os.path.exists('./qiutuLibs'):
  10.         os.mkdir('./qiutuLibs')
  11.         
  12.     url = 'https://www.qiushibaike.com/imgrank/'
  13.     headers = {
  14.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.75'
  15.     }
  16.     # 使用通用爬虫对url对应的一整张页面进行爬取
  17.     page_text = requests.get(url=url, headers=headers).text
  18.  
  19.     # 使用聚焦爬虫将页面中所有的糗图进行爬取
  20.     ex = '<div class="thumb">.*?<img src="(.*?)" alt.*?</div>'
  21.     img_src_list = re.findall(ex, page_text, re.S)
  22.     # print(img_src_list)
  23.     for src in img_src_list:
  24.         # 拼接一个完整的图片url
  25.         src = 'https:' + src
  26.         # 请求到了图片的二进制数据
  27.         img_data = requests.get(url=src, headers=headers).content
  28.         # 生成图片名称
  29.         img_name = src.split('/')[-1]
  30.         # 图片最终存储的路径
  31.         imgPath = './qiutuLibs/' + img_name
  32.         with open(imgPath, 'wb') as fp:
  33.             fp.write(img_data)
  34.             print(img_name, '下载成功!!!')

 案例二

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. import re
  4. import os
  5.  
  6. # 需求： 爬取糗事百科中糗图板块下所有的糗图图片
  7. if __name__ == '__main__':
  8.     headers = {
  9.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.75'
  10.     }
  11.     # 创建一个文件夹，保存所有的图片
  12.     if not os.path.exists('./qiutuLibs'):
  13.         os.mkdir('./qiutuLibs')
  14.         # 设置一个通用的url模板
  15.     url = 'https://www.qiushibaike.com/imgrank/page/%d/'
  16.  
  17.     for pageNum in range(1, 3):
  18.         # 对应页面的url
  19.         new_url = format(url % pageNum)
  20.  
  21.         page_text = requests.get(url=new_url, headers=headers).text
  22.  
  23.         # 使用聚焦爬虫将页面中所有的糗图进行爬取
  24.         ex = '<div class="thumb">.*?<img src="(.*?)" alt.*?</div>'
  25.         img_src_list = re.findall(ex, page_text, re.S)
  26.         # print(img_src_list)
  27.         for src in img_src_list:
  28.             # 拼接一个完整的图片url
  29.             src = 'https:' + src
  30.             # 请求到了图片的二进制数据
  31.             img_data = requests.get(url=src, headers=headers).content
  32.             # 生成图片名称
  33.             img_name = src.split('/')[-1]
  34.             # 图片最终存储的路径
  35.             imgPath = './qiutuLibs/' + img_name
  36.             with open(imgPath, 'wb') as fp:
  37.                 fp.write(img_data)
  38.                 print(img_name, '下载成功!!!')

BeautifulSoup使用

案例一

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. from bs4 import BeautifulSoup
  4. if __name__=='__main__':
  5.     #对首页的页面数据进行爬取
  6.     headers={
  7.         'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.76'
  8.     }
  9.     url='https://www.shicimingju.com/book/sanguoyanyi.html'
  10.     page_text=requests.get(url=url,headers=headers)
  11.     page_text.encoding='utf-8'
  12.     page_text=page_text.text
  13.     
  14.     #在首页中解析出章节的标题和详情页的url
  15.     #1.实例化BeautifulSoup对象，需要将页面源码数据加载到该对象中
  16.     soup=BeautifulSoup(page_text,'lxml')
  17.     #解析章节标题和详情的url
  18.     li_list=soup.select('.book-mulu>ul>li')
  19.     fp=open('./sanguo.txt','w',encoding='utf-8')
  20.     for li in li_list:
  21.         title=li.a.string
  22.         detail_url='https://www.shicimingju.com'+li.a['href']
  23.         #对详情页发起请求，解析出章节内容
  24.         detail_page_text=requests.get(url=detail_url,headers=headers)
  25.         detail_page_text.encoding='utf-8'
  26.         detail_page_text=detail_page_text.text
  27.         #解析出详情页中的相关的章节内容
  28.         detail_soup=BeautifulSoup(detail_page_text,'lxml')
  29.         div_tag=detail_soup.find('div',class_='chapter_content')
  30.         #解析到了章节的内容
  31.         content=div_tag.text
  32.         fp.write(title+':'+content+'\n')
  33.         print(title,'爬取成功!!!')

案例二

  1. from bs4 import BeautifulSoup
  2. import requests
  3. import os
  4.  
  5. fileName = 'novel'
  6. if not os.path.exists(fileName):
  7.     os.mkdir(fileName)
  8. headers = {
  9.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36 Edg/91.0.864.37',
  10.     'Connection': 'close'
  11. }
  12. url = "https://www.shicimingju.com/book/sanguoyanyi.html"
  13. response = requests.get(url=url, headers=headers)
  14. response.encoding = 'utf-8'
  15. soup = BeautifulSoup(response.text, 'lxml')
  16. title = soup.select('.book-mulu > ul > li > a')
  17. cnt = 0
  18. for t in title:
  19.     href = 'https://www.shicimingju.com' + t['href']
  20.     response = requests.get(url=href, headers=headers)
  21.     response.encoding = 'utf-8'
  22.     page_text = response.text
  23.     soup = BeautifulSoup(page_text, 'lxml')
  24.     div = soup.find('div', class_='card bookmark-list')
  25.  
  26.     filePath = fileName + '/' + t.string + '.txt'
  27.     pageTxt = div.text
  28.  
  29.     with open(filePath, 'w', encoding='utf-8') as fp:
  30.         fp.write(pageTxt)
  31.  
  32.     print('爬取成功！！！')
  33.     cnt += 1
  34.  
  35.     if cnt == 10:
  36.         break

seleium自动化爬虫

 解决iframe问题

 案例一

  1. # -*- coding: utf-8 -*-
  2. # 需求：模拟登录
  3. from selenium import webdriver
  4. from time import sleep
  5.  
  6. bro = webdriver.Chrome(executable_path='./chromedriver')
  7.  
  8. bro.get('https://qzone.qq.com/')
  9.  
  10. bro.switch_to.frame('login_frame')
  11.  
  12. a_tag = bro.find_element_by_id("switcher_plogin")
  13. a_tag.click()
  14.  
  15. userName_tag = bro.find_element_by_id('u')
  16. password_tag = bro.find_element_by_id('p')
  17. sleep(1)
  18. userName_tag.send_keys('1292161328')
  19. sleep(1)
  20. password_tag.send_keys('1234567890')
  21. sleep(1)
  22. btn = bro.find_element_by_id('login_button')
  23. btn.click()
  24.  
  25. sleep(3)
  26.  
  27. bro.quit()

案例二

  1. # -*- coding: utf-8 -*-
  2. from selenium import webdriver
  3. from lxml import etree
  4. from time import sleep
  5.  
  6. # 实例化一个浏览器对象（传入的驱动程序）
  7. bro = webdriver.Chrome(executable_path='./chromedriver')
  8. bro.add_argument('-kiosk')
  9. # 让浏览器发起一个指定url对应请求
  10. bro.get('http://scxk.nmpa.gov.cn:81/xk/')
  11.  
  12. # 获取浏览器当前页面的页面源码数据
  13. page_text = bro.page_source
  14.  
  15. # 解析企业名称
  16. tree = etree.HTML(page_text)
  17. li_list = tree.xpath('//ul[@id="gzlist"]/li')
  18. for li in li_list:
  19.     name = li.xpath('./dl/@title')[0]
  20.     print(name)
  21.  
  22. sleep(5)
  23. bro.quit()

案例三

其他自动化操作

  1. from selenium import webdriver
  2. from time import sleep
  3.  
  4. bro = webdriver.Chrome(executable_path='./chromedriver')
  5.  
  6. bro.get('http://www.taobao.com/')
  7.  
  8. # 实现标签定位
  9. search_input = bro.find_element_by_id('q')
  10. # 标签交互
  11. search_input.send_keys('Iphone')
  12.  
  13. # 执行一组js程序
  14. bro.execute_script('window.scrollTo(0,document.body.scrollHeight)')
  15. sleep(2)
  16.  
  17. # 点击搜索按钮
  18. btn = bro.find_element_by_css_selector('.btn-search')
  19. btn.click()
  20.  
  21. bro.get('https://www.baidu.com')
  22. sleep(2)
  23. # 回退
  24. bro.back()
  25. sleep(2)
  26. # 前进
  27. bro.forward()
  28.  
  29. sleep(5)
  30. bro.quit()

案例四

实现无界面

  1. # -*- coding: utf-8 -*-
  2. from selenium import webdriver
  3. from time import sleep
  4. # 实现无可视化界面
  5. from selenium.webdriver.chrome.options import Options
  6. # 实现规避检测
  7. from selenium.webdriver import ChromeOptions
  8.  
  9. # 实现无可视化界面的操作
  10. chrome_options = Options()
  11. chrome_options.add_argument('--headless')
  12. chrome_options.add_argument('--disable-gpu')
  13.  
  14. # 实现规避检测
  15. option = ChromeOptions()
  16. option.add_experimental_option('excludeSwitches', ['enable-automation'])
  17.  
  18. # 如何实现selenium的规避风险
  19. bro = webdriver.Chrome(executable_path='./chromedriver', chrome_options=chrome_options, options=option)
  20.  
  21. # 无可视化界面（无头浏览器） phantomJs
  22. bro.get('https://www.baidu.com')
  23.  
  24. print(bro.page_source)
  25. sleep(2)
  26. bro.quit()

案例五

  1. from selenium import webdriver
  2. from time import sleep
  3. # 后面是你的浏览器驱动位置，记得前面加r'','r'是防止字符转义的
  4. driver = webdriver.Chrome(r'./chromedriver')
  5. # 用get打开百度页面
  6. driver.get("http://www.baidu.com")
  7. # 查找页面的“设置”选项，并进行点击
  8. # driver.find_elements_by_link_text('设置')[0].click()
  9. # sleep(2)
  10. # # # 打开设置后找到“搜索设置”选项，设置为每页显示50条
  11. # driver.find_elements_by_link_text('搜索设置')[0].click()
  12. # sleep(2)
  13. # #选中每页显示50条
  14. # m = driver.find_element_by_id('nr')
  15. # sleep(2)
  16. # m.find_element_by_xpath('//*[@id="nr"]/option[3]').click()
  17. # m.find_element_by_xpath('.//option[3]').click()
  18. # sleep(2)
  19. # # 点击保存设置
  20. # driver.find_elements_by_class_name("prefpanelgo")[0].click()
  21. # sleep(2)
  22. # # 处理弹出的警告页面   确定accept() 和 取消dismiss()
  23. # driver.switch_to_alert().accept()
  24. # sleep(2)
  25. # 找到百度的输入框，并输入 美女
  26. driver.find_element_by_id('kw').send_keys('美女')
  27. sleep(2)
  28. # 点击搜索按钮
  29. driver.find_element_by_id('su').click()
  30. sleep(2)
  31. # 在打开的页面中找到“Selenium - 开源中国社区”，并打开这个页面
  32. driver.find_elements_by_link_text('美女_海量精选高清图片_百度图片')[0].click()
  33. sleep(3)
  34. # 关闭浏览器
  35. driver.quit()

案例六

自动化处理iframe标签

  1. # -*- coding: utf-8 -*-
  2. from selenium import webdriver
  3. from time import sleep
  4. # 导入动作链对应的类
  5. from selenium.webdriver import ActionChains
  6.  
  7. bro = webdriver.Chrome(executable_path='./chromedriver')
  8.  
  9. bro.get('https://www.runoob.com/try/try.php?filename=jqueryui-api-droppable')
  10.  
  11. # 如果定位的标签是存在于iframe标签之中则必须通过如下操作在进行标签定位
  12. bro.switch_to.frame('iframeResult')  # 切换浏览器标签定位的作用域
  13. div = bro.find_element_by_id('draggable')
  14.  
  15. # 动作链
  16. action = ActionChains(bro)
  17. # 点击长按的标签
  18. action.click_and_hold(div)
  19.  
  20. for i in range(5):
  21.     # perform()立即执行动作链操作
  22.     # move_by_offset(x,y)：x水平方向，y竖直方向
  23.     action.move_by_offset(17, 0).perform()
  24.     sleep(0.3)
  25.  
  26. # 释放地址链
  27. action.release()
  28.  
  29. print(div)

案例七

基于selenium的12306用户登录

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. from hashlib import md5
  4.  
  5.  
  6. class Chaojiying_Client(object):
  7.  
  8.     def __init__(self, username, password, soft_id):
  9.         self.username = username
  10.         password = password.encode('utf8')
  11.         self.password = md5(password).hexdigest()
  12.         self.soft_id = soft_id
  13.         self.base_params = {
  14.             'user': self.username,
  15.             'pass2': self.password,
  16.             'softid': self.soft_id,
  17.         }
  18.         self.headers = {
  19.             'Connection': 'Keep-Alive',
  20.             'User-Agent': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)',
  21.         }
  22.  
  23.     def PostPic(self, im, codetype):
  24.         """
  25.         im: 图片字节
  26.         codetype: 题目类型 参考 http://www.chaojiying.com/price.html
  27.         """
  28.         params = {
  29.             'codetype': codetype,
  30.         }
  31.         params.update(self.base_params)
  32.         files = {'userfile': ('ccc.jpg', im)}
  33.         r = requests.post('http://upload.chaojiying.net/Upload/Processing.php', data=params, files=files,
  34.                           headers=self.headers)
  35.         return r.json()
  36.  
  37.     def ReportError(self, im_id):
  38.         """
  39.         im_id:报错题目的图片ID
  40.         """
  41.         params = {
  42.             'id': im_id,
  43.         }
  44.         params.update(self.base_params)
  45.         r = requests.post('http://upload.chaojiying.net/Upload/ReportError.php', data=params, headers=self.headers)
  46.         return r.json()
  47.  
  48.  
  49. # 使用elenium打开登录页面
  50. from selenium import webdriver
  51. import time
  52. from PIL import Image
  53. from selenium.webdriver import ActionChains
  54.  
  55. bro = webdriver.Chrome(executable_path='./chromedriver')
  56. bro.maximize_window()  # 全屏
  57. bro.get('https://kyfw.12306.cn/otn/resources/login.html')
  58. time.sleep(1)
  59.  
  60. # 点击账号登录
  61. bro.find_elements_by_link_text('账号登录')[0].click()
  62. time.sleep(1)
  63.  
  64. # save_screenshot将当前页面进行截图并保存
  65. bro.save_screenshot('aa.png')
  66.  
  67. # 确定验证码图片对应的左上角和右下角的坐标（裁剪区域确定）
  68. code_img_ele = bro.find_element_by_css_selector('#J-loginImg')
  69. location = code_img_ele.location  # 验证码图片左上角的坐标
  70. print('location:', location)
  71. size = code_img_ele.size  # 验证码标签对应的长和宽
  72. print('size:', size)
  73. # 左上角和右下角坐标
  74. rangle = (
  75.     int(location['x']), int(location['y']), int(location['x'] + size['width']), int(location['y'] + size['height']))
  76. # 至此验证码图片区域就确定下来了
  77.  
  78. i = Image.open('./aa.png')
  79. code_img_name = './code.png'
  80. # crop根据指定区域进行图片裁剪
  81. frame = i.crop(rangle)
  82. frame.save(code_img_name)
  83.  
  84. # 将验证码图片交给超级鹰进行识别
  85. chaojiying = Chaojiying_Client('1292161328', 'wuxiangnong', '915445')  # 用户中心>>软件ID 生成一个替换 96001
  86. im = open('code.png', 'rb').read()  # 本地图片文件路径 来替换 a.jpg 有时WIN系统须要//
  87. print(chaojiying.PostPic(im, 9004)['pic_str'])
  88. result = chaojiying.PostPic(im, 9004)['pic_str']
  89. all_list = []  # 要存储即将被点击的点的坐标[x1,y1][x2,y2]
  90. if '|' in result:
  91.     list_1 = result.split('|')
  92.     count_1 = len(list_1)
  93.     for i in range(count_1):
  94.         xy_list = []
  95.         x = int(list_1[i].split(',')[0])
  96.         y = int(list_1[i].split(',')[1])
  97.         xy_list.append(x)
  98.         xy_list.append(y)
  99.         all_list.append(xy_list)
  100.  
  101. else:
  102.     x = int(result.split(',')[0])
  103.     y = int(result.split(',')[1])
  104.     xy_list = []
  105.     xy_list.append(x)
  106.     xy_list.append(y)
  107.     all_list.append(xy_list)
  108. print(all_list)
  109. # 遍历列表，使用动作链对每一个列表元素对应的x,y指定的位置进行点击操作
  110. for l in all_list:
  111.     x = l[0]
  112.     y = l[1]
  113.     ActionChains(bro).move_to_element_with_offset(code_img_ele, x, y).click().perform()
  114.     time.sleep(0.5)
  115.  
  116. bro.find_element_by_id('J-userName').send_keys('19828430139')
  117. time.sleep(2)
  118. bro.find_element_by_id('J-password').send_keys('wuxiangnong9595')
  119. time.sleep(2)
  120. bro.find_element_by_id('J-login').click()
  121. time.sleep(3)
  122. bro.quit()

代理的使用

 

 案例一

  1. # -*- coding: utf-8 -*-
  2. import requests
  3.  
  4. url = 'https://www.baidu.com/s?wd=ip'
  5. headers = {
  6.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.76'
  7. }
  8.  
  9. page_text = requests.get(url=url, headers=headers, proxies={"https": "222.110.147.50:3128"})
  10.  
  11. with open('ip.html', 'w', encoding='utf-8') as fp:
  12.     fp.write(page_text)

验证码解析

  1. # -*- coding: utf-8 -*-
  2.  
  3. import requests
  4. from lxml import etree
  5.  
  6. '''导入一个打码类'''
  7.  
  8.  
  9. # 封装识别验证码图片下载到本地的函数
  10.  
  11. def getCodeText(imgPath, codeType):
  12.     # 普通用户用户名
  13.     username = 'bobo328410948'
  14.  
  15.     # 普通用户密码
  16.     password = 'bobo328410948'
  17.  
  18.     # 软件ＩＤ，开发者分成必要参数。登录开发者后台【我的软件】获得！
  19.     appid = 6003
  20.  
  21.     # 软件密钥，开发者分成必要参数。登录开发者后台【我的软件】获得！
  22.     appkey = '1f4b564483ae5c907a1d34f8e2f2776c'
  23.  
  24.     # 图片文件：即将被识别的验证码图片的路径
  25.     filename = imgPath
  26.  
  27.     # 验证码类型，# 例：1004表示4位字母数字，不同类型收费不同。请准确填写，否则影响识别率。在此查询所有类型 http://www.yundama.com/price.html
  28.     codetype = codeType
  29.  
  30.     # 超时时间，秒
  31.     timeout = 20
  32.     result = None
  33.     # 检查
  34.     if (username == 'username'):
  35.         print('请设置好相关参数再测试')
  36.     else:
  37.         # 初始化
  38.         yundama = YDMHttp(username, password, appid, appkey)
  39.  
  40.         # 登陆云打码
  41.         uid = yundama.login();
  42.         print('uid: %s' % uid)
  43.  
  44.         # 查询余额
  45.         balance = yundama.balance();
  46.         print('balance: %s' % balance)
  47.  
  48.         # 开始识别，图片路径，验证码类型ID，超时时间（秒），识别结果
  49.         cid, result = yundama.decode(filename, codetype, timeout);
  50.         print('cid: %s, result: %s' % (cid, result))
  51.     return result
  52.  
  53.  
  54. if __name__ == "__main__":
  55.     headers = {
  56.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.76'
  57.     }
  58.     url = 'https://so.gushiwen.org/user/login.aspx'
  59.     page_text = requests.get(url=url, headers=headers).text
  60.     # 解析验证码图片img中的属性值
  61.     tree = etree.HTML(page_text)
  62.     code_img_src = 'https://so.gushiwen.org' + tree.xpath('//*[@id="imgCode"]/@src')
  63.     img_data = requests.get(url=code_img_src, headers=headers).content
  64.     # 验证码图片保存到了本地
  65.     with open('./code.jpg', 'wb') as fp:
  66.         fp.write(img_data)
  67.  
  68.     # 打码咯

协程的使用

 案例一

  1. # -*- coding: utf-8 -*-
  2. import asyncio
  3.  
  4.  
  5. async def request(url):
  6.     print('正在请求的url是', url)
  7.     print('请求成功', url)
  8.     return url
  9.  
  10.  
  11. # async修饰的函数，调用之后返回的一个协程对象
  12. c = request('www.baidu.com')
  13.  
  14. # 创建一个事件循环对象
  15. # loop=asyncio.get_event_loop()
  16.  
  17. # #将协程对象注册到loop中，然后启动loop
  18. # loop.run_until_complete(c)
  19.  
  20. # task的使用
  21. loop = asyncio.get_event_loop()
  22. # 基于loop创建一个task对象
  23. task = loop.create_task(c)
  24. print(task)
  25.  
  26.  
  27. # future的使用
  28. # loop=asyncio.get_event_loop()
  29. # task=asyncio.ensure_future(c)
  30. # print(task)
  31. # loop.run_until_complete(task)
  32. # print(task)
  33.  
  34. def callback_func(task):
  35.     # result返回的就是任务对象中封装对象对应函数的返回值
  36.     print(task.result)
  37.  
  38.  
  39. # 绑定回调
  40. loop = asyncio.get_event_loop()
  41. task = asyncio.ensure_future(c)
  42. # 将回调函数绑定到任务对象中
  43. task.add_done_callback(callback_func)
  44. loop.run_until_complete(task)

同步爬虫

  1. import requests
  2.  
  3. headers = {
  4.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.76'
  5. }
  6. urls = {
  7.     'http://xmdx.sc.chinaz.net/Files/DownLoad/jianli/201904/jianli10231.rar',
  8.     'http://zjlt.sc.chinaz.net/Files/DownLoad/jianli/201904/jianli10229.rar',
  9.     'http://xmdx.sc.chinaz.net/Files/DownLoad/jianli/201904/jianli10231.rar'
  10. }
  11.  
  12.  
  13. def get_content(url):
  14.     print('正在爬取:', url)
  15.     response = requests.get(url=url, headers=headers)
  16.     if response.status_code == 200:
  17.         return response.content
  18.  
  19.  
  20. def parse_content(content):
  21.     print('响应数据的长度为：', len(content))
  22.  
  23.  
  24. for url in urls:
  25.     content = get_content(url)
  26.     parse_content(content)

多线程异步爬虫的使用

 案例一

  1. # -*- coding: utf-8 -*-
  2. import asyncio
  3. import time
  4.  
  5.  
  6. async def request(url):
  7.     print('正在下载', url)
  8.     # 在异步协程中如果出现了同步模块相关代码，那么就无法实现异步
  9.     # time.sleep(2)
  10.     # 当在asyncio中遇到阻塞操作必须进行手动挂起
  11.     await asyncio.sleep(2)
  12.     print('下载完毕', url)
  13.  
  14.  
  15. start = time.time()
  16. urls = {
  17.     'www.baidu.com',
  18.     'www.sogou.com',
  19.     'www.goubanjia.com'
  20. }
  21.  
  22. # 任务对象：存放多个任务对象
  23. stasks = []
  24. for url in urls:
  25.     c = request(url)
  26.     task = asyncio.ensure_future(c)
  27.     stasks.append(task)
  28.  
  29. loop = asyncio.get_event_loop()
  30. # 需要将任务列表封装到wait中
  31. loop.run_until_complete(asyncio.wait(stasks))
  32.  
  33. print(time.time() - start)
'
运行

案例二

  1. # -*- coding: utf-8 -*-
  2. import requests
  3. import asyncio
  4. import time
  5.  
  6. start = time.time()
  7. urls = [
  8.     'http://127.0.0.1:5000/bobo',
  9.     'http://127.0.0.1:5000/jay',
  10.     'http://127.0.0.1:5000/tom'
  11. ]
  12.  
  13.  
  14. async def get_page(url):
  15.     print('正在下载', url)
  16.     # requests.get发起的请求是基于同步，必须基于异步的网络请求模块进行指定的url
  17.     # aiohttp:基于异步网络请求的模块
  18.     response = requests.get(url=url)
  19.     print(response.text)
  20.  
  21.  
  22. tasks = []
  23.  
  24. for url in urls:
  25.     c = get_page(url)
  26.     task = asyncio.ensure_future(c)
  27.     tasks.append(task)
  28.  
  29. loop = asyncio.get_event_loop()
  30. loop.run_until_complete(asyncio.wait(tasks))
  31.  
  32. end = time.time()
  33.  
  34. print('耗时', end - start)
'
运行

 线程池

 案例三

  1. # -*- coding: utf-8 -*-
  2. import time
  3.  
  4.  
  5. # 使用单线程串行方式的执行
  6.  
  7. def get_page(str):
  8.     print("正在下载:", str)
  9.     time.sleep(2)
  10.     print('下载成功:', str)
  11.     
  12.  
  13. name_list = ['xiaozi', 'aa', 'bb', 'cc']
  14.  
  15. start_time = time.time()
  16.  
  17. for i in range(len(name_list)):
  18.     get_page(name_list[i])
  19.  
  20. end_time = time.time()
  21. print('%d second' % (end_time - start_time))
'
运行

案例四

  1. # -*- coding: utf-8 -*-
  2. import time
  3. # 导入线程池对应模块的类
  4. from multiprocessing.dummy import Pool
  5.  
  6. # 使用线程池方式执行
  7. start_time = time.time()
  8.  
  9.  
  10. def get_page(str):
  11.     print("正在下载:", str)
  12.     time.sleep(2)
  13.     print('下载成功:', str)
  14.  
  15.  
  16. name_list = ['xiaozi', 'aa', 'bb', 'cc']
  17.  
  18. # 实例化一个线程池
  19. pool = Pool(4)
  20. # 将列表每一个元素传递给get_page进行处理
  21. pool.map(get_page, name_list)
  22.  
  23. end_time = time.time()
  24. print(end_time - start_time)
'
运行

异步协程

 案例五

aiohttp实现任务异步协程

  1. # -*- coding: utf-8 -*-
  2. import time
  3. import asyncio
  4. import aiohttp
  5.  
  6. start = time.time()
  7. urls = [
  8.     'http://127.0.0.1:5000/bobo',
  9.     'http://127.0.0.1:5000/jay',
  10.     'http://127.0.0.1:5000/tom'
  11. ]
  12.  
  13.  
  14. async def get_page(url):
  15.     async with aiohttp.ClientSession() as session:
  16.         # get()、post()：
  17.         # headers,params/data,proxy='http://ip:port'
  18.         async with session.get(url) as response:
  19.             # text()可以返回字符串形式的响应数据
  20.             # read()返回二进制形式的响应数据
  21.             # json()返回的就是json对象
  22.             # 注意：获取响应数据操作之前一定要使用await进行手动挂起
  23.             page_text = await response.text()
  24.             print(page_text)
  25.  
  26.  
  27. tasks = []
  28.  
  29. for url in urls:
  30.     c = get_page(url)
  31.     task = asyncio.ensure_future(c)
  32.     tasks.append(task)
  33.  
  34. loop = asyncio.get_event_loop()
  35. loop.run_until_complete(asyncio.wait(tasks))
  36.  
  37. end = time.time()
  38.  
  39. print('耗时', end - start)

分布式爬虫

 简单练手项目

 

肯德基破解 

  1. # -*- coding: utf-8 -*-
  2. import requests
  3.  
  4. if __name__ == "__main__":
  5.     url = 'http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=keyword'
  6.     param = {
  7.         'cname': '',
  8.         'pid': '',
  9.         'keyword': '北京',
  10.         'pageIndex': '1',
  11.         'pageSize': '10',
  12.     }
  13.     headers = {
  14.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.75'
  15.     }
  16.     response = requests.get(url=url, params=param, headers=headers)
  17.  
  18.     list_data = response.text
  19.  
  20.     fp = open('./KFC.text', 'w', encoding='utf-8')
  21.     fp.write(list_data)
  22.     fp.close()
  23.  
  24.     print('over!!!')

爬取简历模板

  1. import requests
  2. import os
  3. from lxml import etree
  4.  
  5. if __name__ == "__main__":
  6.     headers = {
  7.         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/53'
  8.     }
  9.     url = 'https://sc.chinaz.com/jianli/free.html'
  10.     page_text = requests.get(url=url, headers=headers).text
  11.     
  12.     # 创建文件夹
  13.     if not os.path.exists('./new'):
  14.         os.mkdir('./new')
  15.  
  16.     # 实例对象
  17.     tree = etree.HTML(page_text)
  18.     a_lists = tree.xpath('//div[@id="container"]/div/a')
  19.  
  20.     for a in a_lists:
  21.         href = a.xpath('./@href')[0]
  22.         src = 'https:' + href
  23.  
  24.         page_text_detail = requests.get(url=src, headers=headers).text
  25.  
  26.         treeDetail = etree.HTML(page_text_detail)
  27.         a_lists_products = treeDetail.xpath('//div[@class="clearfix mt20 downlist"]/ul/li')[0]
  28.         href2 = a_lists_products.xpath('./a/@href')[0]
  29.         products_name = href2[-7:]
  30.  
  31.         response = requests.get(url=href2, headers=headers)
  32.         data_products = response.content
  33.         data_path = 'new/' + products_name
  34.  
  35.         with open(data_path, 'wb') as fp:
  36.             fp.write(data_products)
  37.             fp.close()
  38.             print(products_name, "下载成功！！！")

百度AI实现爬虫

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/木道寻08/article/detail/866253
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号