当前位置:网站首页>Scrapy IP agent is not responding
Scrapy IP agent is not responding
2022-07-26 11:29:00 【ha_ lee】
Scrapy ip Analysis of the reason why the agent has not responded for a long time
stay setting.py Add available in ip Agent pool :
PROXIES=[
'http://182.149.82.74:9999',
'http://121.237.25.238:3000',
'http://61.183.176.122:57210',
'http://175.43.84.29:9999',
]
In middleware middlewares.py Add the following classes :
import scrapy
from scrapy import signals
import random
class ProxyMiddleware(object):
def __init__(self, ip):
self.ip = ip
@classmethod
def from_crawler(cls, crawler):
return cls(ip=crawler.settings.get('PROXIES'))
def process_request(self, request, spider):
ip = random.choice(self.ip)
request.meta['http_proxy'] = ip
print(" At present ip by :"+ip)
stay setting.py Of documents DOWNLOADER_MIDDLEWARES Add middleware to attributes :
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddleware.useragent.UserAgentMiddleware': None,
'myproject.middlewares.MyUserAgentMiddleware': 400,
}
At the beginning, I added agents to the middleware IP Part of the code is :
request.meta['proxy'] = ip
my python Version is 3.7,Scrapy by 1.6.0, Maybe due to version problems , Setting the proxy has been unsuccessful , Change it to :
request.meta['http_proxy'] = ip
after , The problem of unsuccessful agency was successfully solved !!!
边栏推荐
猜你喜欢

Getting started step by step using g2o to solve ICP problems - estimating the transformation relationship between two sets of 3D point sets with matching relationship

MySql基础知识汇总

MySQL transaction details

easyui03

Real time streaming protocol --rtsp

easyui02

Three properties of concurrency

QT - connect USB camera

The combination of pytest confitest.py and fixture

Leetcode-209. subarray with the smallest length (binary, prefix and, sliding window)
随机推荐
找工作4个月,面试15家,终于拿到3个offer,定级P7+
Shape matching Halcon template
Why give up NPM and turn to yarn
[报错]Exception: Found duplicate column(s) in the data schema: `value`;
Orbslam2 cmakelists File Structure Parsing
浅谈VIO之IMU预积分(还是刚入门时的想法)
Basic concepts of JVM and memory management model
[开发工具] IEDA报红
如何配置JdbcRealm数据源呢?
easyui02
Multipartfil to file
QT——LCDNumber
easyui04
QT - connect USB camera
Dictionary and int matrix
There is an unhandled exception at 0x003b66c3 in MFC: 0xc000041d: unhandled exception encountered during user callback
Several ways of communication between threads
数据中台建设(二):数据中台简单介绍
Caused by: scala.MatchError: None (of class scala.None$)
实时流式协议--RTSP