您好,登錄后才能下訂單哦!
一、說(shuō)明
本文主要講述采集貓眼電影用戶評(píng)論進(jìn)行分析,相關(guān)爬蟲(chóng)采集程序可以爬取多個(gè)電影評(píng)論。
運(yùn)行環(huán)境:Win10/Python3.5。
分析工具:jieba、wordcloud、pyecharts、matplotlib。
基本流程:下載內(nèi)容 ---> 分析獲取關(guān)鍵數(shù)據(jù) ---> 保存本地文件 ---> 分析本地文件制作圖表
注意:本文所有圖文和源碼僅供學(xué)習(xí),請(qǐng)勿他用,轉(zhuǎn)發(fā)請(qǐng)注明出處!
本文主要參考:https://mp.weixin.qq.com/s/mTxxkwRZPgBiKC3Sv-jo3g
二、開(kāi)始采集
2.1、分析數(shù)據(jù)接口:
為了健全數(shù)據(jù)樣本,數(shù)據(jù)直接從移動(dòng)端接口進(jìn)行采集,連接如下,其中橙色部分為貓眼電影ID,修改即可爬取其他電影。
鏈接地址:http://m.maoyan.com/mmdb/comments/movie/1208282.json?v=yes&offset=15&startTime=
接口返回的數(shù)據(jù)如下,主要采集(昵稱、城市、評(píng)論、評(píng)分和時(shí)間),用戶評(píng)論在 json['cmts'] 中:
2.2、爬蟲(chóng)程序核心內(nèi)容(詳細(xì)可以看后面源代碼):
>啟動(dòng)腳本需要的參數(shù)如下(腳本名+貓眼電影ID+上映日期+數(shù)據(jù)保存的文件名):.\myMovieComment.py 1208282 2016-11-16 myCmts2.txt
>下載html內(nèi)容:download(self, url),通過(guò)python的requests模塊進(jìn)行下載,將下載的數(shù)據(jù)轉(zhuǎn)成json格式
def download(self, url): """下載html內(nèi)容""" print("正在下載URL: "+url) # 下載html內(nèi)容 response = requests.get(url, headers=self.headers) # 轉(zhuǎn)成json格式數(shù)據(jù) if response.status_code == 200: return response.json() else: # print(html.status_code) print('下載數(shù)據(jù)為空!') return ""
>然后就是對(duì)已下載的內(nèi)容進(jìn)行分析,就是取出我們需要的數(shù)據(jù):
def parse(self, content): """分析數(shù)據(jù)""" comments = [] try: for item in content['cmts']: comment = { 'nickName': item['nickName'], # 昵稱 'cityName': item['cityName'], # 城市 'content': item['content'], # 評(píng)論內(nèi)容 'score': item['score'], # 評(píng)分 'startTime': item['startTime'], # 時(shí)間 } comments.append(comment) except Exception as e: print(e) finally: return comments
>將分析出來(lái)的數(shù)據(jù),進(jìn)行本地保存,方便后續(xù)的分析工作:
def save(self, data): """寫(xiě)入文件""" print("保存數(shù)據(jù),寫(xiě)入文件中...") self.save_file.write(data)
> 爬蟲(chóng)的核心控制也即爬蟲(chóng)的程序啟動(dòng)入口,管理上面幾個(gè)方法的有序執(zhí)行:
def start(self): """啟動(dòng)控制方法""" print("爬蟲(chóng)開(kāi)始...\r\n") start_time = self.start_time end_time = self.end_time num = 1 while start_time > end_time: print("執(zhí)行次數(shù):", num) # 1、下載html content = self.download(self.target_url + str(start_time)) # 2、分析獲取關(guān)鍵數(shù)據(jù) comments = '' if content != "": comments = self.parse(content) if len(comments) <= 0: print("本次數(shù)據(jù)量為:0,退出爬取!\r\n") break # 3、寫(xiě)入文件 res = '' for cmt in comments: res += "%s###%s###%s###%s###%s\n" % (cmt['nickName'], cmt['cityName'], cmt['content'], cmt['score'], cmt['startTime']) self.save(res) print("本次數(shù)據(jù)量:%s\r\n" % len(comments)) # 獲取最后一條數(shù)據(jù)的時(shí)間 ,然后減去一秒 start_time = datetime.strptime(comments[len(comments) - 1]['startTime'], "%Y-%m-%d %H:%M:%S") + timedelta(seconds=-1) # start_time = datetime.strptime(start_time, "%Y-%m-%d %H:%M:%S") # 休眠3s num += 1 time.sleep(3) self.save_file.close() print("爬蟲(chóng)結(jié)束...")
2.3 數(shù)據(jù)樣本,最終爬取將近2萬(wàn)條數(shù)據(jù),每條記錄的每個(gè)數(shù)據(jù)使用 ### 進(jìn)行分割:
三、圖形化分析數(shù)據(jù)
3.1、制作觀眾城市分布熱點(diǎn)圖,(pyecharts-geo):
從圖表可以輕松看出,用戶主要分布地區(qū),主要以沿海一些發(fā)達(dá)城市群為主:
def createCharts(self): """生成圖表""" # 讀取數(shù)據(jù),格式:[{"北京", 10}, {"上海",10}] data = self.readCityNum() # 1 熱點(diǎn)圖 geo1 = Geo("《無(wú)名之輩》觀眾位置分布熱點(diǎn)圖", "數(shù)據(jù)來(lái)源:貓眼,F(xiàn)ly采集", title_color="#FFF", title_pos="center", width="100%", height=600, background_color="#404A59") attr1, value1 = geo1.cast(data) geo1.add("", attr1, value1, type="heatmap", visual_range=[0, 1000], visual_text_color="#FFF", symbol_size=15, is_visualmap=True, is_piecewise=False, visual_split_number=10) geo1.render("files/無(wú)名之輩-觀眾位置熱點(diǎn)圖.html") # 2 位置圖 geo2 = Geo("《無(wú)名之輩》觀眾位置分布", "數(shù)據(jù)來(lái)源:貓眼,F(xiàn)ly采集", title_color="#FFF", title_pos="center", width="100%", height=600, background_color="#404A59") attr2, value2 = geo1.cast(data) geo2.add("", attr2, value2, visual_range=[0, 1000], visual_text_color="#FFF", symbol_size=15, is_visualmap=True, is_piecewise=False, visual_split_number=10) geo2.render("files/無(wú)名之輩-觀眾位置圖.html") # 3、top20 柱狀圖 data_top20 = data[:20] bar = Bar("《無(wú)名之輩》觀眾來(lái)源排行 TOP20", "數(shù)據(jù)來(lái)源:貓眼,F(xiàn)ly采集", title_pos="center", width="100%", height=600) attr, value = bar.cast(data_top20) bar.add('', attr, value, is_visualmap=True, visual_range=[0, 3500], visual_text_color="#FFF", is_more_utils=True, is_label_show=True) bar.render("files/無(wú)名之輩-觀眾來(lái)源top20.html") print("圖表生成完成")
3.2、制作觀眾人數(shù)TOP20的柱形圖,(pyecharts-bar):
3.3、制作評(píng)論詞云,(jieba、wordcloud):
生成詞云核心代碼:
def createWordCloud(self): """生成評(píng)論詞云""" comments = self.readAllComments() # 19185 # 使用 jieba 分詞 commens_split = jieba.cut(str(comments), cut_all=False) words = ''.join(commens_split) # 給詞庫(kù)添加停止詞 stopwords = STOPWORDS.copy() stopwords.add("電影") stopwords.add("一部") stopwords.add("無(wú)名之輩") stopwords.add("一部") stopwords.add("一個(gè)") stopwords.add("有點(diǎn)") stopwords.add("覺(jué)得") # 加載背景圖片 bg_image = plt.imread("files/2048_bg.png") # 初始化 WordCloud wc = WordCloud(width=1200, height=600, background_color='#FFF', mask=bg_image, font_path='C:/Windows/Fonts/STFANGSO.ttf', stopwords=stopwords, max_font_size=400, random_state=50) # 生成,顯示圖片 wc.generate_from_text(words) plt.imshow(wc) plt.axis('off') plt.show()
四、修改pyecharts源碼
4.1、樣本數(shù)據(jù)的城市簡(jiǎn)稱與數(shù)據(jù)集完整城市名匹配不上:
使用位置熱點(diǎn)圖時(shí)候,由于采集數(shù)據(jù)城市是一些簡(jiǎn)稱,與pyecharts的已存在數(shù)據(jù)的城市名對(duì)不上,所以對(duì)源碼進(jìn)行一些修改,方便匹配一些簡(jiǎn)稱。
黔南 =>黔南布依族苗族自治州
模塊自帶的全國(guó)主要市縣經(jīng)緯度在:[python安裝路徑]\Lib\site-packages\pyecharts\datasets\city_coordinates.json
由于默認(rèn)情況下,一旦城市名不能完全匹配就會(huì)報(bào)異常,程序會(huì)停止,所以對(duì)源碼修改如下(報(bào)錯(cuò)方法為 Geo.add()),其中添加注析為個(gè)人修改部分:
def get_coordinate(self, name, region="中國(guó)", raise_exception=False): """ Return coordinate for the city name. :param name: City name or any custom name string. :param raise_exception: Whether to raise exception if not exist. :return: A list like [longitude, latitude] or None """ if name in self._coordinates: return self._coordinates[name] coordinate = get_coordinate(name, region=region) # [ 20181204 添加 # print(name, coordinate) if coordinate is None: # 如果字典key匹配不上,嘗試進(jìn)行模糊查詢 search_res = search_coordinates_by_region_and_keyword(region, name) # print("###",search_res) if search_res: coordinate = sorted(search_res.values())[0] # 20181204 添加 ] if coordinate is None and raise_exception: raise ValueError("No coordinate is specified for {}".format(name)) return coordinate
相應(yīng)的需要對(duì) __add()方法進(jìn)行如下修改:
五、附錄-源碼
*說(shuō)明:源碼為本人所寫(xiě),數(shù)據(jù)來(lái)源為貓眼,全部?jī)?nèi)容僅供學(xué)習(xí),拒絕其他用途!轉(zhuǎn)發(fā)請(qǐng)注明出處!
5.1 采集源碼
# -*- coding:utf-8 -*- import requests from datetime import datetime, timedelta import os import time import sys class MaoyanFilmReviewSpider: """貓眼影評(píng)爬蟲(chóng)""" def __init__(self, url, end_time, filename): # 頭部 self.headers = { 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1' } # 目標(biāo)URL self.target_url = url # 數(shù)據(jù)獲取時(shí)間段,start_time:截止日期,end_time:上映時(shí)間 now = datetime.now() # 獲取當(dāng)天的 零點(diǎn) self.start_time = now + timedelta(hours=-now.hour, minutes=-now.minute, seconds=-now.second) self.start_time = self.start_time.replace(microsecond=0) self.end_time = datetime.strptime(end_time, "%Y-%m-%d %H:%M:%S") # 打開(kāi)寫(xiě)入文件, 創(chuàng)建目錄 self.save_path = "files/" if not os.path.exists(self.save_path): os.makedirs(self.save_path) self.save_file = open(self.save_path + filename, "a", encoding="utf-8") def download(self, url): """下載html內(nèi)容""" print("正在下載URL: "+url) # 下載html內(nèi)容 response = requests.get(url, headers=self.headers) # 轉(zhuǎn)成json格式數(shù)據(jù) if response.status_code == 200: return response.json() else: # print(html.status_code) print('下載數(shù)據(jù)為空!') return "" def parse(self, content): """分析數(shù)據(jù)""" comments = [] try: for item in content['cmts']: comment = { 'nickName': item['nickName'], # 昵稱 'cityName': item['cityName'], # 城市 'content': item['content'], # 評(píng)論內(nèi)容 'score': item['score'], # 評(píng)分 'startTime': item['startTime'], # 時(shí)間 } comments.append(comment) except Exception as e: print(e) finally: return comments def save(self, data): """寫(xiě)入文件""" print("保存數(shù)據(jù),寫(xiě)入文件中...") self.save_file.write(data) def start(self): """啟動(dòng)控制方法""" print("爬蟲(chóng)開(kāi)始...\r\n") start_time = self.start_time end_time = self.end_time num = 1 while start_time > end_time: print("執(zhí)行次數(shù):", num) # 1、下載html content = self.download(self.target_url + str(start_time)) # 2、分析獲取關(guān)鍵數(shù)據(jù) comments = '' if content != "": comments = self.parse(content) if len(comments) <= 0: print("本次數(shù)據(jù)量為:0,退出爬取!\r\n") break # 3、寫(xiě)入文件 res = '' for cmt in comments: res += "%s###%s###%s###%s###%s\n" % (cmt['nickName'], cmt['cityName'], cmt['content'], cmt['score'], cmt['startTime']) self.save(res) print("本次數(shù)據(jù)量:%s\r\n" % len(comments)) # 獲取最后一條數(shù)據(jù)的時(shí)間 ,然后減去一秒 start_time = datetime.strptime(comments[len(comments) - 1]['startTime'], "%Y-%m-%d %H:%M:%S") + timedelta(seconds=-1) # start_time = datetime.strptime(start_time, "%Y-%m-%d %H:%M:%S") # 休眠3s num += 1 time.sleep(3) self.save_file.close() print("爬蟲(chóng)結(jié)束...") if __name__ == "__main__": # 確保輸入?yún)?shù) if len(sys.argv) != 4: print("請(qǐng)輸入相關(guān)參數(shù):[moveid]、[上映日期]和[保存文件名],如:xxx.py 42962 2018-11-09 text.txt") exit() # 貓眼電影ID mid = sys.argv[1] # "1208282" # "42964" # 電影上映日期 end_time = sys.argv[2] # "2018-11-16" # "2018-11-09" # 每次爬取條數(shù) offset = 15 # 保存文件名 filename = sys.argv[3] spider = MaoyanFilmReviewSpider(url="http://m.maoyan.com/mmdb/comments/movie/%s.json?v=yes&offset=%d&startTime=" % (mid, offset), end_time="%s 00:00:00" % end_time, filename=filename) # spider.start() spider.start() # t1 = "2018-11-09 23:56:23" # t2 = "2018-11-25" # # res = datetime.strptime(t1, "%Y-%m-%d %H:%M:%S") + timedelta(days=-1) # print(type(res)) MaoyanFilmReviewSpider.py
5.2 分析制圖源碼
# -*- coding:utf-8 -*- from pyecharts import Geo, Bar, Bar3D import jieba from wordcloud import STOPWORDS, WordCloud import matplotlib.pyplot as plt class ACoolFishAnalysis: """無(wú)名之輩 --- 數(shù)據(jù)分析""" def __init__(self): pass def readCityNum(self): """讀取觀眾城市分布數(shù)量""" d = {} with open("files/myCmts2.txt", "r", encoding="utf-8") as f: row = f.readline() while row != "": arr = row.split('###') # 確保每條記錄長(zhǎng)度為 5 while len(arr) < 5: row += f.readline() arr = row.split('###') # 記錄每個(gè)城市的人數(shù) if arr[1] in d: d[arr[1]] += 1 else: d[arr[1]] = 1 # 首次加入字典,為 1 row = f.readline() # print(len(comments)) # print(d) # 字典 轉(zhuǎn) 元組數(shù)組 res = [] for ks in d.keys(): if ks == "": continue tmp = (ks, d[ks]) res.append(tmp) # 按地點(diǎn)人數(shù)降序 res = sorted(res, key=lambda x: (x[1]),reverse=True) return res def readAllComments(self): """讀取所有評(píng)論""" comments = [] # 打開(kāi)文件讀取數(shù)據(jù) with open("files/myCmts2.txt", "r", encoding="utf-8") as f: row = f.readline() while row != "": arr = row.split('###') # 每天記錄長(zhǎng)度為 5 while len(arr) < 5: row += f.readline() arr = row.split('###') if len(arr) == 5: comments.append(arr[2]) # if len(comments) > 20: # break row = f.readline() return comments def createCharts(self): """生成圖表""" # 讀取數(shù)據(jù),格式:[{"北京", 10}, {"上海",10}] data = self.readCityNum() # 1 熱點(diǎn)圖 geo1 = Geo("《無(wú)名之輩》觀眾位置分布熱點(diǎn)圖", "數(shù)據(jù)來(lái)源:貓眼,F(xiàn)ly采集", title_color="#FFF", title_pos="center", width="100%", height=600, background_color="#404A59") attr1, value1 = geo1.cast(data) geo1.add("", attr1, value1, type="heatmap", visual_range=[0, 1000], visual_text_color="#FFF", symbol_size=15, is_visualmap=True, is_piecewise=False, visual_split_number=10) geo1.render("files/無(wú)名之輩-觀眾位置熱點(diǎn)圖.html") # 2 位置圖 geo2 = Geo("《無(wú)名之輩》觀眾位置分布", "數(shù)據(jù)來(lái)源:貓眼,F(xiàn)ly采集", title_color="#FFF", title_pos="center", width="100%", height=600, background_color="#404A59") attr2, value2 = geo1.cast(data) geo2.add("", attr2, value2, visual_range=[0, 1000], visual_text_color="#FFF", symbol_size=15, is_visualmap=True, is_piecewise=False, visual_split_number=10) geo2.render("files/無(wú)名之輩-觀眾位置圖.html") # 3、top20 柱狀圖 data_top20 = data[:20] bar = Bar("《無(wú)名之輩》觀眾來(lái)源排行 TOP20", "數(shù)據(jù)來(lái)源:貓眼,F(xiàn)ly采集", title_pos="center", width="100%", height=600) attr, value = bar.cast(data_top20) bar.add('', attr, value, is_visualmap=True, visual_range=[0, 3500], visual_text_color="#FFF", is_more_utils=True, is_label_show=True) bar.render("files/無(wú)名之輩-觀眾來(lái)源top20.html") print("圖表生成完成") def createWordCloud(self): """生成評(píng)論詞云""" comments = self.readAllComments() # 19185 # 使用 jieba 分詞 commens_split = jieba.cut(str(comments), cut_all=False) words = ''.join(commens_split) # 給詞庫(kù)添加停止詞 stopwords = STOPWORDS.copy() stopwords.add("電影") stopwords.add("一部") stopwords.add("無(wú)名之輩") stopwords.add("一部") stopwords.add("一個(gè)") stopwords.add("有點(diǎn)") stopwords.add("覺(jué)得") # 加載背景圖片 bg_image = plt.imread("files/2048_bg.png") # 初始化 WordCloud wc = WordCloud(width=1200, height=600, background_color='#FFF', mask=bg_image, font_path='C:/Windows/Fonts/STFANGSO.ttf', stopwords=stopwords, max_font_size=400, random_state=50) # 生成,顯示圖片 wc.generate_from_text(words) plt.imshow(wc) plt.axis('off') plt.show() if __name__ == "__main__": demo = ACoolFishAnalysis() demo.createWordCloud()
總結(jié)
以上就是這篇文章的全部?jī)?nèi)容了,希望本文的內(nèi)容對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,如果有疑問(wèn)大家可以留言交流,謝謝大家對(duì)億速云的支持。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。