您好,登錄后才能下訂單哦!
Python中怎么分析網(wǎng)站日志數(shù)據(jù),針對(duì)這個(gè)問(wèn)題,這篇文章詳細(xì)介紹了相對(duì)應(yīng)的分析和解答,希望可以幫助更多想解決這個(gè)問(wèn)題的小伙伴找到更簡(jiǎn)單易行的方法。
import numpy as np import pandas as pd import matplotlib.pyplot as plt import apache_log_parser # 首先通過(guò) pip install apache_log_parser 安裝庫(kù) %matplotlib inline
fformat = '%V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T' # 創(chuàng)建解析器 p = apache_log_parser.make_parser(fformat)
sample_string = 'koldunov.net 85.26.235.202 - - [16/Mar/2013:00:19:43 +0400] "GET /?p=364 HTTP/1.0" 200 65237 "http://koldunov.net/?p=364" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11" 0' data = p(sample_string) #解析后的數(shù)據(jù)為字典結(jié)構(gòu) data
datas = open(r'H:\python數(shù)據(jù)分析\數(shù)據(jù)\apache_access_log').readlines() #逐行讀取log數(shù)據(jù) log_list = [] # 逐行讀取并解析為字典 for line in datas: data = p(line) data['time_received'] = data['time_received'][1:12]+' '+data['time_received'][13:21]+' '+data['time_received'][22:27] #時(shí)間數(shù)據(jù)整理 log_list.append(data) #傳入列表
log = pd.DataFrame(log_list) #構(gòu)造DataFrame log = log[['status','response_bytes_clf','remote_host','request_first_line','time_received']] #提取感興趣的字段 log.head() #status 狀態(tài)碼 response_bytes_clf 返回的字節(jié)數(shù)(流量)remote_host 遠(yuǎn)端主機(jī)IP地址 request_first_line 請(qǐng)求內(nèi)容t ime_received 時(shí)間數(shù)據(jù)
log['time_received'] = pd.to_datetime(log['time_received']) #把time_received字段轉(zhuǎn)換為時(shí)間數(shù)據(jù)類型,并設(shè)置為索引 log = log.set_index('time_received') log.head()
log['status'] = log['status'].astype('int') # 轉(zhuǎn)換為int類型
log['response_bytes_clf'].unique() array(['26126', '10532', '1853', ..., '66386', '47413', '48212'], dtype=object)
log[log['response_bytes_clf'] == '-'].head() #對(duì)response_bytes_clf字段進(jìn)行轉(zhuǎn)換時(shí)報(bào)錯(cuò),查找原因發(fā)現(xiàn)其中含有“-”
def dash3nan(x): # 定義轉(zhuǎn)換函數(shù),當(dāng)為“-”字符時(shí),將其替換為空格,并將字節(jié)數(shù)據(jù)轉(zhuǎn)化為M數(shù)據(jù) if x == '-': x = np.nan else: x = float(x)/1048576 return x
log['response_bytes_clf'] = log['response_bytes_clf'].map(dash3nan) log.head()
log.dtypes
流量起伏不大,但有一個(gè)極大的峰值超過(guò)了20MB。
log[log['response_bytes_clf']>20] #查看流量峰值
t_log = log['response_bytes_clf'].resample('30t').count() t_log.plot()
對(duì)時(shí)間重采樣(30min),并計(jì)數(shù) ,可看出每個(gè)時(shí)間段訪問(wèn)的次數(shù),早上8點(diǎn)訪問(wèn)次數(shù)最多,其余時(shí)間處于上下波動(dòng)中。
h_log = log['response_bytes_clf'].resample('H').count() h_log.plot()
當(dāng)繼續(xù)轉(zhuǎn)換頻率到低頻率時(shí),上下波動(dòng)不明顯。
d_log = pd.DataFrame({'count':log['response_bytes_clf'].resample('10t').count(),'sum':log['response_bytes_clf'].resample('10t').sum()}) d_log.head()
構(gòu)造訪問(wèn)次數(shù)和訪問(wèn)流量的 DataFrame。
plt.figure(figsize=(10,6)) #設(shè)置圖表大小 ax1 = plt.subplot(111) #一個(gè)subplot ax2 = ax1.twinx() #公用x軸 ax1.plot(d_log['count'],color='r',label='count') ax1.legend(loc=2) ax2.plot(d_log['sum'],label='sum') ax2.legend(loc=0)
繪制折線圖,有圖可看出,訪問(wèn)次數(shù)與訪問(wèn)流量存在相關(guān)性。
ip_count = log['remote_host'].value_counts()[0:10] #對(duì)remote_host計(jì)數(shù),并取前10位 ip_count
ip_count.plot(kind='barh') #IP前十位柱狀圖
import pygeoip # pip install pygeoip 安裝庫(kù) # 同時(shí)需要在網(wǎng)站上(http://dev.maxmind.com/geoip/legacy/geolite)下載DAT文件才能解析IP地址 gi = pygeoip.GeoIP(r'H:\python數(shù)據(jù)分析\數(shù)據(jù)\GeoLiteCity.dat', pygeoip.MEMORY_CACHE) info = gi.record_by_addr('64.233.161.99') info #解析IP地址
ips = log.groupby('remote_host')['status'].agg(['count']) # 對(duì)IP地址分組統(tǒng)計(jì) ips.head()
ips.drop('91.224.246.183',inplace=True) ips['country'] = [gi.record_by_addr(i)['country_code3'] for i in ips.index] # 將IP解析的國(guó)家和經(jīng)緯度寫入DataFrame ips['latitude'] = [gi.record_by_addr(i)['latitude'] for i in ips.index] ips['longitude'] = [gi.record_by_addr(i)['longitude'] for i in ips.index] ips.head()
country = ips.groupby('country')['count'].sum() #對(duì)country字段分組統(tǒng)計(jì) country = country.sort_values(ascending=False)[0:10] # 篩選出前10位的國(guó)家 country
country.plot(kind='bar')
俄羅斯的訪問(wèn)量最多,可推斷該網(wǎng)站來(lái)源于俄羅斯。
from mpl_toolkits.basemap import Basemap plt.style.use('ggplot') plt.figure(figsize=(10,6)) map1 = Basemap(projection='robin', lat_0=39.9, lon_0=116.3, resolution = 'l', area_thresh = 1000.0) map1.drawcoastlines() map1.drawcountries() map1.drawmapboundary() map1.drawmeridians(np.arange(0, 360, 30)) map1.drawparallels(np.arange(-90, 90, 30)) size = 0.03 for lon, lat, mag in zip(list(ips['longitude']), list(ips['latitude']), list(ips['count'])): x,y = map1(lon, lat) msize = mag * size map1.plot(x, y, 'ro', markersize=msize)
關(guān)于Python中怎么分析網(wǎng)站日志數(shù)據(jù)問(wèn)題的解答就分享到這里了,希望以上內(nèi)容可以對(duì)大家有一定的幫助,如果你還有很多疑惑沒(méi)有解開,可以關(guān)注億速云行業(yè)資訊頻道了解更多相關(guān)知識(shí)。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。