aiometer
aiometer is a Python 3.6+ concurrency scheduling library compatible with asyncio and trio and inspired by Trimeter. It makes it easier to execute lots of tasks concurrently while controlling concurrency limits (i.e. applying backpressure) and collecting results in a predictable manner.
Content
Example
Let's use HTTPX to make web requests concurrently...
Try this code interactively using IPython.
>>> import asyncio
>>> import functools
>>> import random
>>> import aiometer
>>> import httpx
>>>
>>> client = httpx.AsyncClient()
>>>
>>> async def fetch(client, request):
... response = await client.send(request)
... # Simulate extra processing...
... await asyncio.sleep(2 * random.random())
... return response.json()["json"]
...
>>> requests = [
... httpx.Request("POST", "https://httpbin.org/anything", json={"index": index})
... for index in range(100)
... ]
...
>>> # Send requests, and process responses as they're made available:
>>> async with aiometer.amap(
... functools.partial(fetch, client),
... requests,
... max_at_once=10, # Limit maximum number of concurrently running tasks.
... max_per_second=5, # Limit request rate to not overload the server.
... ) as results:
... async for data in results:
... print(data)
...
{'index': 3}
{'index': 4}
{'index': 1}
{'index': 2}
{'index': 0}
...
>>> # Alternatively, fetch and aggregate responses into an (ordered) list...
>>> jobs = [functools.partial(fetch, client, request) for request in requests]
>>> results = await aiometer.run_all(jobs, max_at_once=10, max_per_second=5)
>>> results
[{'index': 0}, {'index': 1}, {'index': 2}, {'index': 3}, {'index': 4}, ...]
Installation
This project is in beta and maturing. Be sure to pin any dependencies to the latest minor.
pip install "aiometer==0.3.*"
Features
- Concurrency management and throttling helpers.
asyncioandtriosupport.- Fully type annotated.
- 100% test coverage.
Guide
Flow control
The key highlight of aiometer is allowing you to apply flow control strategies in order to limit the degree of concurrency of your programs.
There are two knobs you can play with to fine-tune concurrency:
max_at_once: this is used to limit the maximum number of concurrently running tasks at any given time. (If you have 100 tasks and setmax_at_once=10, thenaiometerwill ensure that no more than 10 run at the same time.)max_per_second: this option limits the number of tasks spawned per second. This is useful to not overload I/O resources, such as servers that may have a rate limiting policy in place.
Example usage:
>>> import asyncio
>>> import aiometer
>>> async def make_query(query):
... await asyncio.sleep(0.05) # Simulate a database request.
...
>>> queries = ['SELECT * from authors'] * 1000
>>> # Allow at most 5 queries to run concurrently at any given time:
>>> await aiometer.run_on_each(make_query, queries, max_at_once=5)
...
>>> # Make at most 10 queries per second:
>>> await aiometer.run_on_each(make_query, queries, max_per_second=10)
...
>>> # Run at most 10 concurrent jobs, spawning new ones at least every 5 seconds:
>>> async def job(id):
... await asyncio.sleep(10) # A very long task.
...
>>> await aiometer.run_on_each(job, range(100), max_at_once=10, max_per_second=0.2)
Running tasks
aiometer provides 4 different ways to run tasks concurrently in the form of 4 different run functions. Each function accepts all the options documented in Flow control, and runs tasks in a slightly different way, allowing to address a variety of use cases. Here's a handy table for reference:
| Entrypoint | Use case |
|---|---|
run_on_each() |
Execute async callbacks in any order. |
run_all() |
Return results as an ordered list. |
amap() |
Iterate over results as they become available. |
run_any() |
Return result of first completed function. |
To illustrate the behavior of each run function, let's first setup a hello world async program:
>>> import asyncio
>>> import random
>>> from functools import partial
>>> import aiometer
>>>
>>> async def get_greeting(name):
... await asyncio.sleep(random.random()) # Simulate I/O
... return f"Hello, {name}"
...
>>> async def greet(name):
... greeting = await get_greeting(name)
... print(greeting)
...
>>> names = ["Robert", "Carmen", "Lucas"]
Let's start with run_on_each(). It executes an async function once for each item in a list passed as argument:
>>> await aiometer.run_on_each(greet, names)
'Hello, Robert!'
'Hello, Lucas!'
'Hello, Carmen!'
If we'd like to get the list of greetings in the same order as names, in a fashion similar to Promise.all(), we can use run_all():
>>> await aiometer.run_all([partial(get_greeting, name) for name in names])
['Hello, Robert', 'Hello, Carmen!', 'Hello, Lucas!']
amap() allows us to process each greeting as it becomes available (which means maintaining order is not guaranteed):
>>> async with aiometer.amap(get_greeting, names) as greetings:
... async for greeting in greetings:
... print(greeting)
'Hello, Lucas!'
'Hello, Robert!'
'Hello, Carmen!'
Lastly, run_any() can be used to run async functions until the first one completes, similarly to Promise.any():
>>> await aiometer.run_any([partial(get_greeting, name) for name in names])
'Hello, Carmen!'
As a last fun example, let's use amap() to implement a no-threads async version of sleep sort:
>>> import asyncio
>>> from functools import partial
>>> import aiometer
>>> numbers = [0.3, 0.1, 0.6, 0.2, 0.7, 0.5, 0.5, 0.2]
>>> async def process(n):
... await asyncio.sleep(n)
... return n
...
>>> async with aiometer.amap(process, numbers) as results:
... sorted_numbers = [n async for n in results]
...
>>> sorted_numbers
[0.1, 0.2, 0.2, 0.3, 0.5, 0.5, 0.6, 0.7]
How To
Multiple parametrized values in run_on_each and amap
run_on_each and amap only accept functions that accept a single positional argument (i.e. (Any) -> Awaitable).
So if you have a function that is parametrized by multiple values, you should refactor it to match this form.
This can generally be achieved like this:
- Build a proxy container type (eg. a
namedtuple), egT. - Refactor your function so that its signature is now
(T) -> Awaitable. - Build a list of these proxy containers, and pass it to
aiometer.
For example, assuming you have a function that processes X/Y coordinates...
async def process(x: float, y: float) -> None:
pass
xs = list(range(100))
ys = list(range(100))
for x, y in zip(xs, ys):
await process(x, y)
You could use it with amap by refactoring it like this:
from typing import NamedTuple
# Proxy container type:
class Point(NamedTuple):
x: float
y: float
# Rewrite to accept a proxy as a single positional argument:
async def process(point: Point) -> None:
x = point.x
y = point.y
...
xs = list(range(100))
ys = list(range(100))
# Build a list of proxy containers:
points = [Point(x, y) for x, y in zip(x, y)]
# Use it:
async with aiometer.amap(process, points) as results:
...
License
MIT
814 Dec 26, 2022
1.3k Jan 08, 2023
44 Dec 13, 2022
1 Dec 27, 2021
4 Feb 10, 2022
2 May 16, 2022
2 Jul 07, 2022
5 Mar 18, 2022
14 Jun 29, 2022
0 Jan 15, 2022
747 Nov 09, 2022
10.9k Jan 04, 2023
29 Dec 12, 2022
23 Nov 30, 2022
3 Oct 09, 2022
0 Nov 14, 2021
1k Dec 15, 2022
10.4k Jan 02, 2023
15 Nov 04, 2022