Squid proxy service +ip proxy pool

There are two services ：

One is to provide all ip And random ip, Delete those maintenance ip Pool function api service .

Reference resources ： First set up the agent pool http Provide the interface
http://www.spiderpy.cn/blog/detail/13
https://xz.aliyun.com/t/7718
https://github.com/jhao104/proxy_pool

One is the tunnel ip pool , When writing code, you can fix a tunnel ip agent .
Reference resources ：

And then through squid To provide an agent ip Forwarding switching function
https://blog.csdn.net/aaronjny/article/details/87865942
https://www.jianshu.com/p/130083146c8f
https://github.com/AaronJny/open_proxy_pool
If systemctl start squid.service Failed to start service , use squid -k parse Check the configuration format and other problems , Even newspaper ERROR: Ignoring unknown cache_peer option In fact, it can be started normally
Error starting service ：Failed to make swap directory /var/spool/squid, Just modify the permissions chown squid:squid /var/spool/squid
Request returns ：ERROR:The requested URL could not be retrieved, May be squid.conf Li agent ip Not configured or invalid
see systemctl status squid.service Start up , If appear FATAL: ERROR: cache_peer 1.1.1.1 specified twice Is to configure multiple proxy entries , identical ip Different port Report errors , stay cache_peer Then add name=proxy-90 Random . Reference resources https://www.cnblogs.com/my8100/p/7441501.html
 

https://xnathan.com/2017/03/02/squid-proxy-pool/
build ip pool , Use Squid Provided parent agent function , Automatically forward the request of the crawler server to Agency service device .Squid Automatic polling function is provided , Automatically validate and weed out unavailable agents . It reduces our redundant verification steps .

Request to get proxy , Then write directly to squid To configure ：https://github.com/xNathan/squid_proxy_pool

The actual measurement is , Take it directly redis in ip Configuration to squid, This project is used ：https://github.com/AaronJny/open_proxy_pool
 

Squid It will automatically check the validity of each agent
http://blog.itpub.net/31557738/viewspace-2679380/
squid Do access control , It's just one. ip Balanced use ？

nginx You can do the same thing ？