您好,登錄后才能下訂單哦!
本篇內(nèi)容介紹了“如何用python抓取鏈家網(wǎng)二手房數(shù)據(jù)”的有關(guān)知識,在實(shí)際案例的操作過程中,不少人都會遇到這樣的困境,接下來就讓小編帶領(lǐng)大家學(xué)習(xí)一下如何處理這些情況吧!希望大家仔細(xì)閱讀,能夠?qū)W有所成!
# -*- coding: utf-8
import urllib2
import urllib
import re,os
import time
#from bs4 import BeautifulSoup
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
class HomeLink:
#初始化數(shù)據(jù)
def __init__(self,base_url):
self.base_url = base_url
self.page = 1
self.out_put_file = 'D:/python/test/house.txt'
self.user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
self.headers = { 'User-Agent' : self.user_agent }
#獲取頁面內(nèi)容
def get_content(self,url):
try:
request = urllib2.Request(url,headers=self.headers)
response = urllib2.urlopen(request)
act_url = response.geturl()
print 'init url=',url,'act url=',act_url
if url == act_url:
content = response.read()
return content
else:
return None
except urllib2.URLError, e:
if hasattr(e,"reason"):
print u"連接頁面失敗,錯(cuò)誤原因",e.reason
return None
#獲取每個(gè)區(qū)的起始url鏈接地址
def get_region_url(self):
d_region_url = {}
content = self.get_content(self.base_url)
pattern = re.compile('<a href="(/ershoufang/\w+/)" title=".*?">(.*?)</a>',re.S)
result = re.findall(pattern,content)
if result:
for x in result:
d_region_url[x[1]] = x[0]
else:
pass
return d_region_url
#獲取每個(gè)區(qū)的所有頁面url地址列表
def get_region_url_list(self,region_url):
page_num = self.get_page_num(region_url)
l_url = [region_url+'pg'+str(i)+'/' for i in range(2,page_num+1)]
return l_url
#獲取總頁數(shù)
def get_page_num(self,url):
content = self.get_content(url)
pattern = re.compile('{"totalPage":(\d+),"curPage":1}',re.S)
result = re.search(pattern,content)
if result:
return int(result.group(1).strip())
else:
return None
#獲取每套房子的房價(jià)信息
def get_house_info(self,url,region):
content = self.get_content(url)
pattern = re.compile('<a href=".*?" target="_blank" data-log_index="\d+" data-el="region">' +
'(.*?)</a>(.*?)</div>.*?</a></div>.*?</div>'
+ '.*?<div class="totalPrice"><span>(\d+)(\S+)</div>',re.S)
result = re.findall(pattern,content)
if result:
for x in result:
l = x[1].split('|')
rooms,area,direct,other = l[1],l[2],l[3],l[4]
s_str = '|'.join([region,x[0],rooms,area,direct,other,x[2],x[3]])
self.writeStr2File(self.out_put_file,s_str)
else:
return None
#開始抓取鏈家網(wǎng)房價(jià)數(shù)據(jù)
def start_scrapy(self):
d_region_url = self.get_region_url()
for k in d_region_url:
region = k
region_init_url = 'http://bj.lianjia.com' + d_region_url[region]
l_region_url = self.get_region_url_list(region_init_url)
for url in l_region_url:
time.sleep(1)
url = url.strip()
self.get_house_info(url,region)
#寫文件
def writeStr2File(self,out_put_file,str1,append = 'a'):
# 去掉文件,保留路徑。比如 'a/b/c/d.txt' 經(jīng)過下面代碼會變成 'a/b/c'
subPath = out_put_file[:self.out_put_file.rfind('/')]
# 如果給定的路徑中,文件夾不存在,則創(chuàng)建
if not os.path.exists(subPath):
os.makedirs(subPath)
# 打開文件并將 str 內(nèi)容寫入給定的文件
with open(out_put_file, append) as f:
f.write(str1.strip()+'\n')
url = 'http://bj.lianjia.com/ershoufang/'
home = HomeLink(url)
home.start_scrapy()
“如何用python抓取鏈家網(wǎng)二手房數(shù)據(jù)”的內(nèi)容就介紹到這里了,感謝大家的閱讀。如果想了解更多行業(yè)相關(guān)的知識可以關(guān)注億速云網(wǎng)站,小編將為大家輸出更多高質(zhì)量的實(shí)用文章!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。