Asynchronous Python HTTP Requests for Humans using twisted

Overview

Asynchronous Python HTTP Requests for Humans

https://travis-ci.org/tardyp/txrequests.png?branch=master

Small add-on for the python requests http library. Makes use twisted's ThreadPool, so that the requests'API returns deferred

The additional API and changes are minimal and strives to avoid surprises.

The following synchronous code:

from requests import Session

session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)

Can be translated to make use of futures, and thus be asynchronous by creating a FuturesSession and catching the returned Future in place of Response. The Response can be retrieved by calling the result method on the Future:

from txrequests import Session
from twisted.internet import defer

@defer.inlineCallbacks
def main():
    # use with statement to cleanup session's threadpool, and connectionpool after use
    # you can also use session.close() if want to use session for long term use
    with Session() as session:
        # first request is started in background
        d1 = session.get('http://httpbin.org/get')
        # second requests is started immediately
        d2 = session.get('http://httpbin.org/get?foo=bar')
        # wait for the first request to complete, if it hasn't already
        response_one = yield d1
        print('response one status: {0}'.format(response_one.status_code))
        print(response_one.content)
        # wait for the second request to complete, if it hasn't already
        response_two = yield d2
        print('response two status: {0}'.format(response_two.status_code))
        print(response_two.content)

By default a ThreadPool is created with 4 max workers. If you would like to adjust that value or share a threadpool across multiple sessions you can provide one to the Session constructor.

from twisted.python.threadpool import ThreadPool
from txrequests import Session

session = FuturesSession(pool=ThreadPool(maxthreads=10))
# ...

As a shortcut in case of just increasing workers number you can pass minthreads and/or maxthreads straight to the Session constructor:

from txrequests import Session
session = Session(maxthreads=10)

That's it. The api of requests.Session is preserved without any modifications beyond returning a Deferred rather than Response. As with all futures exceptions are shifted to the deferred errback.

Working in the Background

There is one additional parameter to the various request functions, background_callback, which allows you to work with the Response objects in the background thread. This can be useful for shifting work out of the foreground, for a simple example take json parsing.

from pprint import pprint
from txrequests import Session
from twisted.internet import defer

@defer.inlineCallbacks
def main():
    with Session() as session:

        def bg_cb(sess, resp):
            # parse the json storing the result on the response object
            resp.data = resp.json()
            return resp

        d = session.get('http://httpbin.org/get', background_callback=bg_cb)
        # do some other stuff, send some more requests while this one works
        response = yield d
        print('response status {0}'.format(response.status_code))
        # data will have been attached to the response object in the background
        pprint(response.data)

Installation

pip install txrequests

Credits

txrequests is based on requests_future, from Ross McFarland

Owner
Pierre Tardy
Pierre Tardy
As easy as /aitch-tee-tee-pie/ 🥧 Modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more. https://twitter.com/httpie

HTTPie: human-friendly CLI HTTP client for the API era HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI in

HTTPie 25.4k Jan 01, 2023
Small, fast HTTP client library for Python. Features persistent connections, cache, and Google App Engine support. Originally written by Joe Gregorio, now supported by community.

Introduction httplib2 is a comprehensive HTTP client library, httplib2.py supports many features left out of other HTTP libraries. HTTP and HTTPS HTTP

457 Dec 10, 2022
hackhttp2 make everything easier

hackhttp2 intro This repo is inspired by hackhttp, but it's out of date already. so, I create this repo to make simulation and Network request easier.

youbowen 5 Jun 15, 2022
A minimal HTTP client. ⚙️

HTTP Core Do one thing, and do it well. The HTTP Core package provides a minimal low-level HTTP client, which does one thing only. Sending HTTP reques

Encode 306 Dec 27, 2022
💡Python package for HTTP/1.1 style headers. Parse headers to objects. Most advanced available structure for http headers.

HTTP Headers, the Complete Toolkit 🧰 Object-oriented headers. Kind of structured headers. ❓ Why No matter if you are currently dealing with code usin

TAHRI Ahmed R. 103 Dec 02, 2022
HTTP request/response parser for python in C

http-parser HTTP request/response parser for Python compatible with Python 2.x (=2.7), Python 3 and Pypy. If possible a C parser based on http-parser

Benoit Chesneau 334 Dec 24, 2022
Aiohttp-openmetrics - OpenMetrics endpoint provider for aiohttp

aiohttp-openmetrics This project contains a simple middleware and /metrics route

Jelmer Vernooij 1 Dec 15, 2022
Python package for caching HTTP response based on etag

Etag cache implementation for HTTP requests, to save request bandwidth for a non-modified response. Returns high-speed accessed dictionary data as cache.

Rakesh R 2 Apr 27, 2022
A toolbelt of useful classes and functions to be used with python-requests

The Requests Toolbelt This is just a collection of utilities for python-requests, but don't really belong in requests proper. The minimum tested reque

892 Jan 06, 2023
PycURL - Python interface to libcurl

PycURL -- A Python Interface To The cURL library PycURL is a Python interface to libcurl, the multiprotocol file transfer library. Similarly to the ur

PycURL 933 Jan 09, 2023
Python requests like API built on top of Twisted's HTTP client.

treq: High-level Twisted HTTP Client API treq is an HTTP library inspired by requests but written on top of Twisted's Agents. It provides a simple, hi

Twisted Matrix Labs 553 Dec 18, 2022
Some example code for using a raspberry pi to draw text (including emojis) and twitch emotes to a HUB75 RGB matrix via an HTTP post endpoint.

Some example code for using a raspberry pi to draw text (including emojis) and twitch emotes to a HUB75 RGB matrix via an HTTP post endpoint.

7 Nov 05, 2022
A Python obfuscator using HTTP Requests and Hastebin.

🔨 Jawbreaker 🔨 Jawbreaker is a Python obfuscator written in Python3, using double encoding in base16, base32, base64, HTTP requests and a Hastebin-l

Billy 50 Sep 28, 2022
Probe and discover HTTP pathname using brute-force methodology and filtered by specific word or 2 words at once

pathprober Probe and discover HTTP pathname using brute-force methodology and filtered by specific word or 2 words at once. Purpose Brute-forcing webs

NFA 41 Jul 06, 2022
Aiohttp simple project with Swagger and ccxt integration

crypto_finder What Where Documentation http://localhost:8899/docs Maintainer nordzisko Crypto Finder aiohttp application Application that connects to

Norbert Danisik 5 Feb 27, 2022
suite de mocks http em json

Ritchie Formula Repo Documentation Contribute to the Ritchie community This repository contains rit formulas which can be executed by the ritchie-cli.

Kaio Fábio Prates Prudêncio 1 Nov 01, 2021
Script to automate PUT HTTP method exploitation to get shell.

Script to automate PUT HTTP method exploitation to get shell.

devploit 116 Nov 10, 2022
An interactive command-line HTTP and API testing client built on top of HTTPie featuring autocomplete, syntax highlighting, and more. https://twitter.com/httpie

HTTP Prompt HTTP Prompt is an interactive command-line HTTP client featuring autocomplete and syntax highlighting, built on HTTPie and prompt_toolkit.

HTTPie 8.6k Dec 31, 2022
Python Client for the Etsy NodeJS Statsd Server

Introduction statsd is a client for Etsy's statsd server, a front end/proxy for the Graphite stats collection and graphing server. Links The source: h

Rick van Hattem 107 Jun 09, 2022
Bot que responde automáticamente as perguntas do giga unitel

Gigabot+ Bot que responde automáticamente as perguntas do giga unitel LINK DOWNLOAD: Gigabot.exe O script pode apresentar alguns erros, pois não tive

Joaquim Roque 20 Jul 16, 2021