Python3 e-learning case 4: writing web proxy

Please go to Baidu Encyclopedia for the definition and function of proxy server ~

1. Web Proxy How to realize it

This is based on the previous article “ Write Web Server” It's written , The main logic is shown in the figure below ：

 

  What we're going to write is the middle Web Proxy part , When the client to Web Proxy Send a request for access to a web site （Request） When ,Web Proxy You will first check whether you have the requested file , If there is, go straight back to （Response）, If not ,Web Proxy It's like Web Server（ The server to access the web address ） Send a request to get the target file , And then to Client return .

2. Web Proxy Use

First , When we visit a website, in order to visit through proxy, we can't simply open the browser and enter the URL to visit （ That would be Client Direct to Web Server Transmit Reuest 了 ）, Here you can download a file called Wget Tools for , This is for Web Proxy It's like jdk For Java The same thing （ Of course, there may be other tools that can access the proxy server first , There is no discussion about ）, After downloading, you can unzip it and use it , It's like using jdk First, find the folder where the file is located in the command line window , If you do not want to input a list of directories to query each time, you can also add the path of this file to the environment variable （ How to configure self search ）.

When Web Proxy and Wget When you're ready, you can start running ：

First, execute Web Proxy The program , And then through Wget Request to use a proxy and deliver Request

（Wget command ：wget xxx.xxx.xx -e use_proxy=on -e http_proxy=127.0.0.1:8000）, among “xxx.xxx.xx” It's the URL you want to ask for

3. Web Proxy Source code

import socket


def handleReq(clientSocket):
    # recv data
    # find the fileName
    # judge if the file named "fileName" if existed
    # if not exists, send req to get it

    recvData = clientSocket.recv(1024).decode()
    fileName = recvData.split()[1].split("//")[1].replace('/', '')
    print("fileName: " + fileName)
    try:
        file = open(fileName, 'rb')
        print("File is found in proxy server.")
        responseMsg = file.readlines()
        for i in range(0, len(responseMsg)):
            clientSocket.sendall(responseMsg[i])
        print("Send, done.")
    except Exception:
        try:
            print("File is not exist.\nSend request to server...")
            proxyClientSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            serverName = fileName.split(":")[0]
            proxyClientSocket.connect((serverName, 80))
            proxyClientSocket.sendall(recvData.encode())
            responseMsg = proxyClientSocket.recv(4069)
            print("File is found in server.")
            proxyClientSocket.sendall(responseMsg)
            print("Send, done.")
            # cache
            cache = open("./" + fileName, 'w')
            cache.writelines(responseMsg.decode().replace('\r\n', '\n'))
            cache.close()
            print("Cache, done.")
        except:
            print("Connect timeout.")


def startProxy(port):
    proxyServerSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    proxyServerSocket.bind(("", port))
    proxyServerSocket.listen(0)
    while True:
        try:
            print("Proxy is waiting for connecting...")
            clientSocket, addr = proxyServerSocket.accept()
            print("Connect established")
            handleReq(clientSocket)
            clientSocket.close()
        except Exception as e:
            print("error: {0}".format(e))
            break
    proxyServerSocket.close()


if __name__ == '__main__':
    while True:
        try:
            port = int(input("choose a port number over 1024:"))
        except ValueError:
            print("Please input an integer rather than {0}".format(type(port)))
            continue
        else:
            if port <= 1024:
                print("Please input an integer greater than 1024")
                continue
            else:
                break
    startProxy(port)

4. Wget tool kit

Link to ：https://pan.baidu.com/s/1Ae2_Cq9SYbKnfhhyJ1VhpQ
Extraction code ：awsl 

&n