您好,登錄后才能下訂單哦!
這期內(nèi)容當(dāng)中小編將會(huì)給大家?guī)碛嘘P(guān)使用python爬蟲如何判斷網(wǎng)頁信息是否存在,文章內(nèi)容豐富且以專業(yè)的角度為大家分析和敘述,閱讀完這篇文章希望大家可以有所收獲。
在找工作的時(shí)候,我們會(huì)選擇上網(wǎng)查詢招聘的信息,或者是通過一些招聘會(huì)進(jìn)行現(xiàn)場(chǎng)面試。但由于信息更新不及時(shí),有一些崗位會(huì)出現(xiàn)下架的情況,如果我們不注意的話,可能就撲了空。在時(shí)間上耽誤了不說,面試的信息也會(huì)受到一點(diǎn)點(diǎn)打擊。今天小編就教大家python爬蟲來判斷招聘信息是否存在。
首先這里需要一個(gè)判斷某條招聘是否還掛在網(wǎng)站上的方法,這個(gè)暫時(shí)想到了還沒弄,然后對(duì)于發(fā)布時(shí)間在兩個(gè)月之前的數(shù)據(jù),就不進(jìn)行統(tǒng)計(jì)計(jì)算。
以下是完成代碼:
{ "_id" : ObjectId("5a30ad2068504386f47d9a4b"), "city" : "蘇州", "companyShortName" : "藍(lán)海彤翔", "companySize" : "100-499人", "education" : "本科", "financeStage" : "B輪", "industryField" : "互聯(lián)網(wǎng)", "level" : 3, "pid" : "11889834", "positionLables" : [ "PHP", "ThinkPHP" ], "positionName" : "php研發(fā)工程師", "salary" : { "avg" : 7500.0, "low" : 7000, "high" : 8000 }, "time" : "2017-06-06", "updated_at" : "2017-12-13 18:31:15", "workYear" : "1-3年", "detail" : "1、處理landcloud云計(jì)算相關(guān)系統(tǒng)的各類開發(fā)和調(diào)研工作;2、處理coms高性能計(jì)算的各類開發(fā)和調(diào)研工作崗位要求:1、本科學(xué)歷,兩年以上工作經(jīng)驗(yàn),熟悉PHP開發(fā),了解常用的php開發(fā)技巧和框架;2、了解C++,python及Java開發(fā);3、有一定的研發(fā)能力和鉆研精神;4、有主動(dòng)溝通能力和吃苦耐勞的精神。", "location" : "蘇州市高新區(qū)科技城錦峰路158號(hào)101park8幢"
實(shí)例擴(kuò)展:
python爬蟲爬取騰訊招聘信息 (靜態(tài)爬蟲)
import requests from bs4 import BeautifulSoup from math import ceil header = { 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'} # 獲取崗位頁數(shù) def getJobPage(url): ret = requests.get(url, headers=header) ret.encoding = "utf-8" # 解決亂碼問題 html = ret.text soup = BeautifulSoup(html, 'html.parser') # 獲取崗位總數(shù),< span class ="lightblue total" > 512 < / span > totalJob = soup.select('span[class="lightblue total"]')[0].text jobPage = ceil(int(totalJob) / 10) return jobPage def getJobOrder(url): ret = requests.get(url, headers=header) ret.encoding = "utf-8" # 解決亂碼問題 html = ret.text soup = BeautifulSoup(html, 'html.parser') # 工作職責(zé) jobRequests = soup.select('ul[class="squareli"]')[0].text # 工作要求 jobOrder = soup.select('ul[class="squareli"]')[1].text return jobRequests, jobOrder # 獲取崗位信息 def getJobInfo(url): myfile = open("tencent_job.txt", "a", encoding='gb18030', errors='ignore') # 解決亂碼問題 ret = requests.get(url, headers=header) ret.encoding = "utf-8" # 解決亂碼問題 html = ret.text soup = BeautifulSoup(html, 'html.parser') jobList = soup.find_all('tr', class_=['even', 'odd']) for job in jobList: # url jobUrl = "https://hr.tencent.com/" + job.select('td:nth-of-type(1) > a')[0]['href'] # 職位名稱 jobName = job.select('td:nth-of-type(1) > a')[0].text # 人數(shù) jobPeople = job.select('td:nth-of-type(3)')[0].text # 地點(diǎn) jobAddre = job.select('td:nth-of-type(4)')[0].text # 發(fā)布時(shí)間 jobTime = job.select('td:nth-of-type(5)')[0].text # 工作職責(zé) jobRequests = getJobOrder(jobUrl)[0] # 工作要求 jobOrder = getJobOrder(jobUrl)[1] #print(jobName, jobUrl, jobAddre, jobPeople, jobTime, jobRequests, jobOrder) tt = jobName + " " + jobUrl + " " + jobAddre + " " + jobPeople + " " + jobTime + " " + jobRequests + " " + jobOrder myfile.write(tt + "\n") if __name__ == '__main__': mainurl = 'https://hr.tencent.com/position.php?keywords=python' jobPage = getJobPage(mainurl) print(jobPage) for page in range(jobPage): pageUrl = 'https://hr.tencent.com/position.php?keywords=python&start=' + str(page * 10) + '#a' print("第" + str(page + 1) + "頁") getJobInfo(pageUrl)
上述就是小編為大家分享的使用python爬蟲如何判斷網(wǎng)頁信息是否存在了,如果剛好有類似的疑惑,不妨參照上述分析進(jìn)行理解。如果想知道更多相關(guān)知識(shí),歡迎關(guān)注億速云行業(yè)資訊頻道。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。