您好,登錄后才能下訂單哦!
Python中怎么定時執(zhí)行網(wǎng)站爬蟲,針對這個問題,這篇文章詳細(xì)介紹了相對應(yīng)的分析和解答,希望可以幫助更多想解決這個問題的小伙伴找到更簡單易行的方法。
編寫爬蟲代碼
編寫一個爬蟲程序,使用requests
與beautifulsoup4
包爬取和解析Yahoo!股市-上市成交價排行與Yahoo!股市-上柜成交價排行的資料,再利用pandas
包將解析后的展示出來。
import datetime
import requests
from bs4 import BeautifulSoup
import pandas as pd
def get_price_ranks():
current_dt = datetime.datetime.now().strftime("%Y-%m-%d %X")
current_dts = [current_dt for _ in range(200)]
stock_types = ["tse", "otc"]
price_rank_urls = ["https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e={}&n=100".format(st) for st in stock_types]
tickers = []
stocks = []
prices = []
volumes = []
mkt_values = []
ttl_steps = 10*100
each_step = 10
for pr_url in price_rank_urls:
r = requests.get(pr_url)
soup = BeautifulSoup(r.text, 'html.parser')
ticker = [i.text.split()[0] for i in soup.select(".name a")]
tickers += ticker
stock = [i.text.split()[1] for i in soup.select(".name a")]
stocks += stock
price = [float(soup.find_all("td")[2].find_all("td")[i].text) for i in range(5, 5+ttl_steps, each_step)]
prices += price
volume = [int(soup.find_all("td")[2].find_all("td")[i].text.replace(",", "")) for i in range(11, 11+ttl_steps, each_step)]
volumes += volume
mkt_value = [float(soup.find_all("td")[2].find_all("td")[i].text)*100000000 for i in range(12, 12+ttl_steps, each_step)]
mkt_values += mkt_value
types = ["上市" for _ in range(100)] + ["上柜" for _ in range(100)]
ky_registered = [True if "KY" in st else False for st in stocks]
df = pd.DataFrame()
df["scrapingTime"] = current_dts
df["type"] = types
df["kyRegistered"] = ky_registered
df["ticker"] = tickers
df["stock"] = stocks
df["price"] = prices
df["volume"] = volumes
df["mktValue"] = mkt_values
return df
price_ranks = get_price_ranks()
print(price_ranks.shape)
這個的結(jié)果展示為
## (200, 8)
接下來我們利用pandas進行前幾行展示
price_ranks.head()
price_ranks.tail()
接下來我們就開始往服務(wù)器上部署
對于服務(wù)器的選擇,環(huán)境配置不在本課的討論范圍之內(nèi),我們主要是要講一下怎么去設(shè)置定時任務(wù)。
接下來我們改造一下代碼,改造成結(jié)果有sqlite存儲。
import datetime
import requests
from bs4 import BeautifulSoup
import pandas as pd
import sqlite3
def get_price_ranks():
current_dt = datetime.datetime.now().strftime("%Y-%m-%d %X")
current_dts = [current_dt for _ in range(200)]
stock_types = ["tse", "otc"]
price_rank_urls = ["https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e={}&n=100".format(st) for st in stock_types]
tickers = []
stocks = []
prices = []
volumes = []
mkt_values = []
ttl_steps = 10*100
each_step = 10
for pr_url in price_rank_urls:
r = requests.get(pr_url)
soup = BeautifulSoup(r.text, 'html.parser')
ticker = [i.text.split()[0] for i in soup.select(".name a")]
tickers += ticker
stock = [i.text.split()[1] for i in soup.select(".name a")]
stocks += stock
price = [float(soup.find_all("td")[2].find_all("td")[i].text) for i in range(5, 5+ttl_steps, each_step)]
prices += price
volume = [int(soup.find_all("td")[2].find_all("td")[i].text.replace(",", "")) for i in range(11, 11+ttl_steps, each_step)]
volumes += volume
mkt_value = [float(soup.find_all("td")[2].find_all("td")[i].text)*100000000 for i in range(12, 12+ttl_steps, each_step)]
mkt_values += mkt_value
types = ["上市" for _ in range(100)] + ["上櫃" for _ in range(100)]
ky_registered = [True if "KY" in st else False for st in stocks]
df = pd.DataFrame()
df["scrapingTime"] = current_dts
df["type"] = types
df["kyRegistered"] = ky_registered
df["ticker"] = tickers
df["stock"] = stocks
df["price"] = prices
df["volume"] = volumes
df["mktValue"] = mkt_values
return df
price_ranks = get_price_ranks()
conn = sqlite3.connect('/home/ubuntu/yahoo_stock.db')
price_ranks.to_sql("price_ranks", conn, if_exists="append", index=False)
接下來如果我們讓他定時啟動,那么,我們需要linux的crontab命令:
如果我們要設(shè)置每天的 9:30 到 16:30 之間每小時都執(zhí)行一次
那么我們只需要先把文件命名為price_rank_scraper.py
關(guān)于Python中怎么定時執(zhí)行網(wǎng)站爬蟲問題的解答就分享到這里了,希望以上內(nèi)容可以對大家有一定的幫助,如果你還有很多疑惑沒有解開,可以關(guān)注億速云行業(yè)資訊頻道了解更多相關(guān)知識。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進行舉報,并提供相關(guān)證據(jù),一經(jīng)查實,將立刻刪除涉嫌侵權(quán)內(nèi)容。