您好,登錄后才能下訂單哦!
這篇文章將為大家詳細(xì)講解有關(guān)如何解決Python爬取網(wǎng)頁(yè)時(shí)請(qǐng)求中收到響應(yīng)400的問(wèn)題,小編覺(jué)得挺實(shí)用的,因此分享給大家做個(gè)參考,希望大家閱讀完這篇文章后可以有所收獲。
一般情況下抓取網(wǎng)頁(yè)時(shí)找到url中可以從 post 請(qǐng)求中獲取數(shù)據(jù),但得到的響應(yīng)是400而不是 200時(shí),可以嘗試從服務(wù)器獲得正確的響應(yīng),使用json=參數(shù)requests進(jìn)行嘗試可以解決。
問(wèn)題案例:
import requests headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0', 'Accept': 'application/json', 'Accept-Language': 'en-US,en;q=0.5', 'Content-Type': 'application/x-www-form-urlencoded', 'Origin': 'https://www.ketto.org', 'Sec-Fetch-Dest': 'empty', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'cross-site', 'Referer': 'https://www.ketto.org/', 'Connection': 'keep-alive', } params = ( ('x-algolia-agent', 'Algolia for JavaScript (3.35.1); Browser (lite); angular (8.2.14); angular-instantsearch (3.0.0-beta.4); instantsearch.js (3.7.0); JS Helper (2.28.1)'), ('x-algolia-application-id', 'NN2UORRIZX'), ('x-algolia-api-key', 'b2caa1b0589e8db9398d5fe2a40bbaed'), ) data = [ ('{requests:[{indexName:fundraiser_prod,params:query', ''), ('hitsPerPage', '9'), ('hitsPerPage', '1'), ('hitsPerPage', '1'), ('hitsPerPage', '1'), ('maxValuesPerFacet', '10'), ('maxValuesPerFacet', '10'), ('maxValuesPerFacet', '10'), ('maxValuesPerFacet', '10'), ('page', '1'), ('page', '0'), ('page', '0'), ('page', '0'), ('highlightPreTag', '__ais-highlight__'), ('highlightPreTag', '__ais-highlight__'), ('highlightPreTag', '__ais-highlight__'), ('highlightPreTag', '__ais-highlight__'), ('highlightPostTag', '__/ais-highlight__'), ('highlightPostTag', '__/ais-highlight__'), ('highlightPostTag', '__/ais-highlight__'), ('highlightPostTag', '__/ais-highlight__'), ('facets', '["cause.label","tags","address"]'), ('facets', '["cause.label"]'), ('facets', '["tags"]'), ('facets', '["address"]'), ('tagFilters', ''), ('tagFilters', ''), ('tagFilters', ''), ('tagFilters', ''), ('facetFilters', '[["cause.label:"],["tags:"],["address:"]]},{indexName:fundraiser_prod,params:query='), ('facetFilters', '[["tags:"],["address:"]]},{indexName:fundraiser_prod,params:query='), ('facetFilters', '[["cause.label:"],["address:"]]},{indexName:fundraiser_prod,params:query='), ('facetFilters', '[["cause.label:"],["tags:"]]}]}'), ('attributesToRetrieve', '[]'), ('attributesToRetrieve', '[]'), ('attributesToRetrieve', '[]'), ('attributesToHighlight', '[]'), ('attributesToHighlight', '[]'), ('attributesToHighlight', '[]'), ('attributesToSnippet', '[]'), ('attributesToSnippet', '[]'), ('attributesToSnippet', '[]'), ('analytics', 'false'), ('analytics', 'false'), ('analytics', 'false'), ('clickAnalytics', 'false'), ('clickAnalytics', 'false'), ('clickAnalytics', 'false'), ] response = requests.post('https://nn2uorrizx-dsn.algolia.net/1/indexes/*/queries', headers=headers, params=params, data=data) print(response)
代碼進(jìn)行優(yōu)化處理:
import requestsurl = "https://nn2uorrizx-dsn.algolia.net/1/indexes/*/queries"params = { "x-algolia-agent": "Algolia for JavaScript (3.35.1); Browser (lite); angular (8.2.14); angular-instantsearch (3.0.0-beta.4); instantsearch.js (3.7.0); JS Helper (2.28.1)", "x-algolia-application-id": "NN2UORRIZX", "x-algolia-api-key": "b2caa1b0589e8db9398d5fe2a40bbaed",}data = { "requests": [ { "indexName": "fundraiser_prod", "params": "query=&hitsPerPage=9&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&facets=%5B%22cause.label%22%2C%22tags%22%2C%22address%22%5D&tagFilters=&facetFilters=%5B%5B%22cause.label%3A%22%5D%2C%5B%22tags%3A%22%5D%2C%5B%22address%3A%22%5D%5D", }, { "indexName": "fundraiser_prod", "params": "query=&hitsPerPage=1&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=%5B%22cause.label%22%5D&facetFilters=%5B%5B%22tags%3A%22%5D%2C%5B%22address%3A%22%5D%5D", }, { "indexName": "fundraiser_prod", "params": "query=&hitsPerPage=1&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=%5B%22tags%22%5D&facetFilters=%5B%5B%22cause.label%3A%22%5D%2C%5B%22address%3A%22%5D%5D", }, { "indexName": "fundraiser_prod", "params": "query=&hitsPerPage=1&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=%5B%22address%22%5D&facetFilters=%5B%5B%22cause.label%3A%22%5D%2C%5B%22tags%3A%22%5D%5D", }, ]}data = requests.post(url, params=params, json=data).json()print(json.dumps(data, indent=4))
打印結(jié)果:
{ "results": [ { "hits": [ { "id": 180773, "title": "Feeding From Far-Ration Distribution Amid Lockdown", "end_date": "2021-09-11 23:59:59", "amount_requested": 65000000, "entity_details_id": 1505699, "creator_entity_details_id": 1463388, "address_1": "Mumbai", "creation_date": "2020-04-05 15:10:12", "parent_cause_id": 48, "event_entity_details_id": 1399904, "sucess_story_flag": 0, "custom_tag": "FeedingFromFarForCorona", "beneficiary": { "full_name": "Prayatna " }, "raised": { "campaign_id": 180773, "backers": 10616, "raised": "45094771.40", "usdraised": "644211.02", "currencies": { "INR": 45094771, "USD": 644211, "GBP": 501053, "EUR": 536843, "AED": 2254739, "SGD": 867207, "SAR": 2254739 } }, "widget": { "media_type": "image", "file_name": "wid60a133a89542c.jpg", "path": "/media/campaign/180000/180773/image/", "cdn_path": "https://d1vdjc70h9nzd9.cloudfront.net/media/campaign/180000/180773/image/wid60a133a89542c.jpg" }, "cause": { "info_1": "Food & Hunger", "info_3": "Orange", "label": "Food & Hunger" }, "campaigner": { "id": 1463388, "fname": "Pooja Reddy", "lname": null, "full_name": "Pooja Reddy ", "disable_foreign_donation": 0, "no_80g": false, "user_details_id": 1470493, "entity_type": "individual", "tax_benefit": false, "avtar": { "entity_type_id": 1463388, "entity_type": "individual", "media_type": "image", "file_name": "60efdb6db4758.jpg", "path": "/media/individual/1463000/1463388/image/", "cdn_path": "https://d1vdjc70h9nzd9.cloudfront.net/media/individual/1463000/1463388/image/60efdb6db4758.jpg" } }, "taxexempted": { "id": 1505699, "no_80g": true, "entity_type": "ngo", "tax_benefit": true },... and so on.}
關(guān)于“如何解決Python爬取網(wǎng)頁(yè)時(shí)請(qǐng)求中收到響應(yīng)400的問(wèn)題”這篇文章就分享到這里了,希望以上內(nèi)容可以對(duì)大家有一定的幫助,使各位可以學(xué)到更多知識(shí),如果覺(jué)得文章不錯(cuò),請(qǐng)把它分享出去讓更多的人看到。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。