您好,登錄后才能下訂單哦!
這篇文章主要講解了基于線程池提升request模塊效率的方法,內(nèi)容清晰明了,對此有興趣的小伙伴可以學(xué)習(xí)一下,相信大家閱讀完之后會有幫助。
普通方法:爬取梨視頻
import re import time import random import requests from lxml import etree start_time = time.time() url = "https://www.pearvideo.com/category_3" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" } ex = 'srcUrl="(.*?)",vdoUrl=srcUrl' def request_video(url): """ 向視頻鏈接發(fā)送請求 """ return requests.get(url=url, headers=headers).content def save_video(content): """ 將視頻的二進(jìn)制數(shù)據(jù)保存到本地 """ video_name = str(random.randint(100, 999)) + ".mp4" with open(video_name, 'wb') as f: f.write(content) # 獲取首頁源碼 page_text = requests.get(url=url, headers=headers).text tree = etree.HTML(page_text) li_list = tree.xpath('//ul[@class="listvideo-list clearfix"]/li') video_url_list = list() for li in li_list: detail_url = "https://www.pearvideo.com/" + li.xpath('./div/a/@href')[0] # 獲取該視頻頁面的源碼 detail_page_text = requests.get(url=detail_url, headers=headers).text # 正則匹配視頻的URL video_url = re.findall(ex, detail_page_text, re.S)[0] video_url_list.append(video_url) content = request_video(video_url) save_video(content) print("執(zhí)行耗時: ", time.time() - start_time)
執(zhí)行耗時: 147.22410440444946
使用線程池:爬取梨視頻
# 使用線程池爬去梨視頻的 import re import time import random import requests from lxml import etree from multiprocessing.dummy import Pool start_time = time.time() url = "https://www.pearvideo.com/category_3" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" } ex = 'srcUrl="(.*?)",vdoUrl=srcUrl' def request_video(url): """ 向視頻鏈接發(fā)送請求 """ return requests.get(url=url, headers=headers).content def save_video(content): """ 將視頻的二進(jìn)制數(shù)據(jù)保存到本地 """ video_name = str(random.randint(100, 999)) + ".mp4" with open(video_name, 'wb') as f: f.write(content) # 獲取首頁源碼 page_text = requests.get(url=url, headers=headers).text tree = etree.HTML(page_text) li_list = tree.xpath('//ul[@class="listvideo-list clearfix"]/li') video_url_list = list() for li in li_list: detail_url = "https://www.pearvideo.com/" + li.xpath('./div/a/@href')[0] # 獲取該視頻頁面的源碼 detail_page_text = requests.get(url=detail_url, headers=headers).text # 正則匹配視頻的URL video_url = re.findall(ex, detail_page_text, re.S)[0] video_url_list.append(video_url) pool = Pool(4) #使用線程池將視頻的二進(jìn)制數(shù)據(jù)下載下來 content_list = pool.map(request_video, video_url_list) # 使用線程池將視頻的二進(jìn)制數(shù)據(jù)保存到本地 pool.map(save_video, content_list) print("執(zhí)行耗時: ", time.time() - start_time)
看完上述內(nèi)容,是不是對基于線程池提升request模塊效率的方法有進(jìn)一步的了解,如果還想學(xué)習(xí)更多內(nèi)容,歡迎關(guān)注億速云行業(yè)資訊頻道。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報,并提供相關(guān)證據(jù),一經(jīng)查實,將立刻刪除涉嫌侵權(quán)內(nèi)容。