动态更改IP地址?

我想经常抓取网站,但是我的IP地址在某天/限制后被阻止。

那么,如何动态地更改我的IP地址或其他任何想法?

回答:

使用Scrapy的方法将利用两个组件,RandomProxyRotateUserAgentMiddleware

修改DOWNLOADER_MIDDLEWARES如下。你将必须在中插入新组件settings.py

DOWNLOADER_MIDDLEWARES = {

'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 90,

'tutorial.randomproxy.RandomProxy': 100,

'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,

'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware' : None,

'tutorial.spiders.rotate_useragent.RotateUserAgentMiddleware' :400,

}

随机代理

你可以使用scrapy-proxies。该组件将使用列表中的随机代理处理Scrapy请求,以避免IP禁止并提高抓取速度。

你可以通过快速的互联网搜索来建立代理列表。list.txt根据请求的URL格式复制文件中的链接。

轮换用户代理

对于每个临时请求,将从你预先定义的列表中使用随机用户代理:

class RotateUserAgentMiddleware(UserAgentMiddleware):

def __init__(self, user_agent=''):

self.user_agent = user_agent

def process_request(self, request, spider):

ua = random.choice(self.user_agent_list)

if ua:

request.headers.setdefault('User-Agent', ua)

# Add desired logging message here.

spider.log(

u'User-Agent: {} {}'.format(request.headers.get('User-Agent'), request),

level=log.DEBUG

)

# the default user_agent_list composes chrome,I E,firefox,Mozilla,opera,netscape

# for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php

user_agent_list = [

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",

"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",

"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"

]

以上是 动态更改IP地址? 的全部内容, 来源链接: utcz.com/qa/421686.html

回到顶部