当前位置:网站首页>Scrapy IP agent is not responding
Scrapy IP agent is not responding
2022-07-26 11:29:00 【ha_ lee】
Scrapy ip Analysis of the reason why the agent has not responded for a long time
stay setting.py Add available in ip Agent pool :
PROXIES=[
'http://182.149.82.74:9999',
'http://121.237.25.238:3000',
'http://61.183.176.122:57210',
'http://175.43.84.29:9999',
]
In middleware middlewares.py Add the following classes :
import scrapy
from scrapy import signals
import random
class ProxyMiddleware(object):
def __init__(self, ip):
self.ip = ip
@classmethod
def from_crawler(cls, crawler):
return cls(ip=crawler.settings.get('PROXIES'))
def process_request(self, request, spider):
ip = random.choice(self.ip)
request.meta['http_proxy'] = ip
print(" At present ip by :"+ip)
stay setting.py Of documents DOWNLOADER_MIDDLEWARES Add middleware to attributes :
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddleware.useragent.UserAgentMiddleware': None,
'myproject.middlewares.MyUserAgentMiddleware': 400,
}
At the beginning, I added agents to the middleware IP Part of the code is :
request.meta['proxy'] = ip
my python Version is 3.7,Scrapy by 1.6.0, Maybe due to version problems , Setting the proxy has been unsuccessful , Change it to :
request.meta['http_proxy'] = ip
after , The problem of unsuccessful agency was successfully solved !!!
边栏推荐
猜你喜欢
随机推荐
[vscode]如何远程连接服务器
[学习进度]5月
List and dictionary instance applications (※)
Load orb dictionary
记录个人遇到的错误
Real time streaming protocol --rtsp
Synchronized and reentrantlock
27. Remove elements
3Dunity游戏项目实战——第一人称射击游戏
测试用例千万不能随便,记录由一个测试用例异常引起的思考
easyui02
JVM基本概念及内存管理模型
Data visualization - White Snake 2: black snake robbery (2)
梅科尔工作室-华为14天鸿蒙设备开发实战笔记八
MLX90640 红外热成像仪测温传感器模块开发笔记(六)
1837. Sum of digits under k-ary representation
[报错]Exception: Found duplicate column(s) in the data schema: `value`;
Toolstrip border removal
五万美元的年薪是如何花光的
Pytest fixture decorator




![[vscode]如何远程连接服务器](/img/b4/9a80ad995bd589596d8b064215b55a.png)


![[idea]如何新建一个项目](/img/33/f210d59ccd3664487f401929dac24c.png)

