爬虫post请求返回Server Error in '/' Application.

  • Z时代
  • 2024-02-17
  • 分类：IT

百度学者库的网址https://expert.cnki.net/Searc...
使用feapder发送post请求 模拟点击检索的请求爬虫post请求返回Server Error in '/' Application.
这个请求的地址是https://expert.cnki.net/Searc...
可是response返回出现这个问题爬虫post请求返回Server Error in '/' Application.
代码如下：

import feapder
from items import *
headers = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
    "Connection": "keep-alive",
    "Content-Length": "1401",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Cookie": "Ecp_ClientId=1210322215905370029; cnkiUserKey=8ce7d5ad-0de3-cd0b-40a6-aef90ee05b75; Ecp_ClientIp=222.204.55.203; Ecp_loginuserbk=nj0350; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221795576336658b-054e8e4d2478a3-d7e1739-2073600-179557633671c6%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%7D%2C%22%24device_id%22%3A%221795576336658b-054e8e4d2478a3-d7e1739-2073600-179557633671c6%22%7D; Ecp_loginuserjf=1102644615@qq.com; UM_distinctid=17b3a6c8314850-004a35a510dead-4343363-1fa400-17b3a6c8315d",
    "Host": "expert.cnki.net",
    "Origin": "https://expert.cnki.net",
    "Referer": "https://expert.cnki.net/Search/AdvFind",
    "sec-ch-ua": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Windows\"",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "same-origin",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest"
}
url = 'https://expert.cnki.net/Search/AdvFindResult'
parms1 = 'fieldParam=I138&name_pcni_select=name_0&name_0=&name_match_0=1&unit_relation_0=11&unit_0=%E5%8D%97%E6%98%8C%E5%A4%A7%E5%AD%A6&unit_match_0=1&name_relation_1=11&name_pcni_select=name_1&name_1=&name_match_1=1&unit_relation_1=11&unit_1=&unit_match_1=1&name_relation_2=11&name_pcni_select=name_2&name_2=&name_match_2=1&unit_relation_2=11&unit_2=&unit_match_2=1&name_relation_3=11&name_pcni_select=name_3&name_3=&name_match_3=1&unit_relation_3=11&unit_3=&unit_match_3=1&name_relation_4=11&name_pcni_select=name_4&name_4=&name_match_4=1&unit_relation_4=11&unit_4=&unit_match_4=1&keyword_0=&keyword_match_0=1&researcharea_relation_0=11&researcharea_0=&keyword_relation_1=11&keyword_1=&keyword_match_1=1&researcharea_relation_1=11&researcharea_1=&keyword_relation_2=11&keyword_2=&keyword_match_2=1&researcharea_relation_2=11&researcharea_2=&keyword_relation_3=11&keyword_3=&keyword_match_3=1&researcharea_relation_3=11&researcharea_3=&keyword_relation_4=11&keyword_4=&keyword_match_4=1&researcharea_relation_4=11&researcharea_4=&fund_match_0=1&fundcodes=&statNum_field_0=9&statNum_match_0=3&statNum_0=&statNum_relation_1=11&statNum_field_1=9&statNum_match_1=3&statNum_1=&statNum_relation_2=11&statNum_field_2=9&statNum_match_2=3&statNum_2=&statNum_relation_3=11&statNum_field_3=9&statNum_match_3=3&statNum_3=&statNum_relation_4=11&statNum_field_4=9&statNum_match_4=3&statNum_4=&X-Requested-With=XMLHttpRequest'
class ListSpider(feapder.AirSpider):
    def start_requests(self):
        yield feapder.Request(url, render=True, method='POST', headers=headers, data=parms1)
    def parse(self, request, response):
        print(response.text)
    def parse_detail(self, request, response):
        print(response.text)
if __name__ == "__main__":
    ListSpider().start()

参数都是对的 用普通的requests都是没问题可以正常请求到，但是feapder就有毛病 哎

回答：

yield feapder.Request(url, render=True, method='POST', headers=headers, data=parms1)

改成

yield feapder.Request(url, method='POST', headers=headers, data=parms1)

取消动态渲染
大功告成

以上是 爬虫post请求返回Server Error in '/' Application. 的全部内容， 来源链接： utcz.com/p/938122.html

回到顶部