Asynchronous, fast, pythonic DynamoDB Client

Overview

AsyncIO DynamoDB

CircleCI Code style: black Documentation Status

Asynchronous pythonic DynamoDB client; 2x faster than aiobotocore/boto3/botocore.

Quick start

With httpx

Install this library

pip install "aiodynamo[httpx]" or, for poetry users poetry add aiodynamo -E httpx

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.httpx import HTTPX
from httpx import AsyncClient

    async with AsyncClient() as h:
        client = Client(HTTPX(h), Credentials.auto(), "us-east-1")

With aiohttp

Install this library

pip install "aiodynamo[aiohttp]" or, for poetry users poetry add aiodynamo -E aiohttp

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession

    async with ClientSession() as session:
        client = Client(AIOHTTP(session), Credentials.auto(), "us-east-1")

API use

        table = client.table("my-table")

        # Create table if it doesn't exist
        if not await table.exists():
            await table.create(
                Throughput(read=10, write=10),
                KeySchema(hash_key=KeySpec("key", KeyType.string)),
            )

        # Create or override an item
        await table.put_item({"key": "my-item", "value": 1})
        # Get an item
        item = await table.get_item({"key": "my-item"})
        print(item)
        # Update an item, if it exists.
        await table.update_item(
            {"key": "my-item"}, F("value").add(1), condition=F("key").exists()
        )

Why aiodynamo

  • boto3 and botocore are synchronous. aiodynamo is built for asynchronous apps.
  • aiodynamo is fast. Two times faster than aiobotocore, botocore or boto3 for operations such as query or scan.
  • aiobotocore is very low level. aiodynamo provides a pythonic API, using modern Python features. For example, paginated APIs are automatically depaginated using asynchronous iterators.
  • Legible source code. botocore and derived libraries generate their interface at runtime, so it cannot be inspected and isn't typed. aiodynamo is hand written code you can read, inspect and understand.
  • Pluggable HTTP client. If you're already using an asynchronous HTTP client in your project, you can use it with aiodynamo and don't need to add extra dependencies or run into dependency resolution issues.

Complete documentation is here

Comments
  • question: Any chance for transaction support?

    question: Any chance for transaction support?

    I'm wondering if there are any plans to support TransactWriteItems or just Transactions in general.

    I can probably work on this if/when I have more time ๐Ÿค”

    opened by stupoid 16
  • add AWS_PROFILE env var to credentials chain

    add AWS_PROFILE env var to credentials chain

    Adds support for using AWS_PROFILE env var in the credentials chain.
    This behavior is more in line with the behavior expected by users coming from boto3 or aiobotocore. Added a unit test to cover this feature.

    Relates to: Issue #139

    opened by BTripp1986 8
  • Http wrapping issues

    Http wrapping issues

    When I was adding credentials handler to assume role with web identity which we need for our EKS setup I came across few issues why I am for now implementing it as custom code in our own project instead of providing it for the library.

    Http wrappers assume response is json, https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py but as endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs ๐Ÿ’ฉ ). This is blocker and thought that perhaps there needs to be some refactoring around http handling which takes too long for us.

    Some additional things I noticed

    1. in MetadataCredentials there is fetch_with_retry which couldn't use because it assumes GET but endpoint for assume role is POST. Was thinking should/could this retry option be in base http classes?

    2. Missing timeout in POST, would prefer to have timeout also for this credentials call. As mentioned in https://github.com/HENNGE/aiodynamo/issues/45

    3. http.post(..) requires body as bytes even when it can in aiohttp and httpx also be None. https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py#L28-L30

    opened by jarikujansuu 6
  • Add support for pay_per_request billing mode during table creation

    Add support for pay_per_request billing mode during table creation

    Currently aiodynamo creates tables using the provisioned billing mode. DynamoDB also supports pay_per_request where no throughput config is provided. This PR is backwards compatible, but also adds the support to provide a billing_mode parameter to create_table. The throughput parameter has been changed to optional as the pay_per_request billing mode doesn't use it.

    PS. I'd be very grateful if you could also add the hacktoberfest-accepted label to this PR once it is accepted so that it can be counted as a hacktoberfest PR. Thanks :)

    hacktoberfest-accepted 
    opened by TheEdgeOfRage 6
  • Handle ClientConnectorError in connector

    Handle ClientConnectorError in connector

    Exception text: ClientConnectorError: Cannot connect to host dynamodb.<snip>.amazonaws.com:443 ssl:default [None]

    Exception type: aiohttp.client_exceptions.ClientConnectorError

    Stack:

    aiohttp/connector.py in _wrap_create_connection at line 943
    aiohttp/connector.py in _create_direct_connection at line 980
    aiohttp/connector.py in _create_direct_connection at line 1004
    aiohttp/connector.py in _create_connection at line 858
    aiohttp/connector.py in connect at line 523
    aiohttp/client.py in _request at line 480
    aiohttp/client.py in __aenter__ at line 1012
    aiodynamo/http/aiohttp.py in post at line 31
    aiodynamo/client.py in send_request at line 651
    aiodynamo/client.py in get_item at line 454
    aiodynamo/client.py in get_item at line 124
    
    opened by dimaqq 6
  • Raise asyncio.TimeoutError on httpx.TimeoutException

    Raise asyncio.TimeoutError on httpx.TimeoutException

    Now asyncio.TimeoutError is handled in the client logic. But httpx wrapper raises httpx.TimeoutException. This pull request convert the exception and make the client be able to handle timeouts in httpx.

    ref.) httpx timeout exceptions https://github.com/encode/httpx/blob/master/httpx/_exceptions.py#L7-L11

    ref.) Custom HTTP Client Adaptor document https://github.com/HENNGE/aiodynamo/blob/2e6c4716a3ac9fe5669bbdcaa73e6bbe0f73cfbb/docs/advanced.rst

    opened by gunyarakun 5
  • Count limits

    Count limits

    This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

    opened by Tinche 5
  • Use 3rd-party ddbcereal library for deserialize/serialize.

    Use 3rd-party ddbcereal library for deserialize/serialize.

    Figured I'd put this out here as an option. I've been working on a serializer/deserializer lib to supplement my use of aiobotocore, but it might make sense for this project too. It should be as fast or faster than this one.

    Overall, it reduces/simplifies the code, but the initialization of Client is a tad uglier due to ddbcereal needing a pre-constructed Deserializer and Client being frozen. It could be made pretty by unfreezing it or alternatively, making a seperate client-spawning function that passes the Deserializer to the Client constructor.

    opened by JustinTArthur 5
  • Don't send `StreamSpecification: null`

    Don't send `StreamSpecification: null`

    https://github.com/HENNGE/aiodynamo/blob/81b6f98111ce4bed41aeb8d07747695af836c76a/src/aiodynamo/client.py#L41-L49

    Sends StreamSpecification=None when none is supplied by the caller. Perhaps aiodynamo should not inject this key if the value is None.

    It seems that's OK for AWS SaaS dynamo, dynamodb-local and dynalite, but not ScyllaDB:

    https://github.com/scylladb/scylla/issues/5796

    opened by dimaqq 5
  • How to update map element using a pattern?

    How to update map element using a pattern?

    Described in https://dev.to/matthewvielkind/updating-values-in-dyanmodb-map-attributes-can

    UpdateExpression="SET players.#player_id.score = :score_val",  
    ExpressionAttributeNames={"#player_id": player_id},  
    

    I'm not sure what the API is in this case... Would I need to overload F?

    opened by dimaqq 4
  • Support consistent read

    Support consistent read

    Purpose

    Support consistent read in get_item, query, scan, count and batch get operations.

    Note: Tests do not actually check anything except that DynamoDb calls do not explode if using consistent_read option. Local DynamoDb is always strongly consistent. And might want to reduce copied code in them.

    opened by jarikujansuu 4
  • Empty-set safety

    Empty-set safety

    aiodynamo used to be empty-string-safe when DynamoDB didn't support empty string values. This mercifully got fixed in DynamoDB, but DynamoDB still doesn't accept empty sets (NS, SS, BS). There's an argument to be made that aiodynamo should handle this automatically, converting F("some_set").set(set()) to F("some_set").remove(), possibly with a warning.

    opened by ojii 3
  • Shall we try goodfirstissue.dev ?

    Shall we try goodfirstissue.dev ?

    links: https://goodfirstissue.dev/language/python https://github.com/deepsourcelabs/good-first-issue#adding-a-new-project

    I think this project may qualify, and would get some exposure and helping hands

    opened by dimaqq 0
  • Support AWS Web Identity Token File authentication

    Support AWS Web Identity Token File authentication

    aiodynamo ~~doesn't support AWS Web Identity Token File authentication~~ doesn't support AWS Web Identity Token File authentication out of the box.

    It looks like a typical case. For my team, we would like to use that authentication in our services in k8s by using IAM Roles for Service Accounts(IRSA)

    Related issue: https://github.com/HENNGE/aiodynamo/issues/128 Related PR: https://github.com/HENNGE/aiodynamo/pull/127

    There was a similar issue, but I decided to create another one because I think we can describe our goal and issue more directly.

    So, as discovered in a related PR, we should implement the AssumeRoleWithWebIdentity API for this authentication.

    opened by mrkovalchuk 2
  • Auth Error With No Default Credentials

    Auth Error With No Default Credentials

    I have access to several AWS accounts via SSO and no default profile for AWS. When I sign in to AWS via SSO the .aws/credentials file is updated with a fresh Access Key, Secret Key, and Security Token.

    I specify which credentials I will use with the AWS_PROFILE env var, which is checked as part of the credentials chain lookup. If using boto3 or the aws-cli this method works. With this method I can sign in via SSO to dev and prod accounts. Then if I AWS_PROFILE=dev aws s3 ls(or similar call in boto3) it will run the command against the dev account. Then if I AWS_PROFILE=prod aws s3 ls it will run against prod.

    This doesn't seem to work with aiodynamo. In order to get it to work I need to create a default profile and then copy/paste the credentials from the desired profile into the default profile. Aiodynamo should respect the AWS_PROFILE env var.

    opened by BTripp1986 1
  • Surprising behaviour in `Client.table_exists`

    Surprising behaviour in `Client.table_exists`

    We use Table.exists as part of a health check for the containers in our application, which in turn calls Client.table_exists. Today we had a short outage when all health checks began failing. It turned out the Dynamo table had entered UPDATING status.

    The implementation does the following:

    async def table_exists(self, name: TableName) -> bool:
            try:
                description = await self.describe_table(name)
            except TableNotFound:
                return False
    
            return description.status is TableStatus.active
    

    This feels like a bug to us: UPDATING is a valid state while a variety of operations are being applied to the table. So the code should do return description.status in [TableStatus.active, TableStatus.updating].

    Thoughts?

    opened by darylweir 1
Releases(21.9)
  • 21.9(Sep 1, 2021)

  • 21.8(Aug 24, 2021)

  • 21.7(Aug 24, 2021)

  • 21.6(Aug 24, 2021)

  • 21.5(Aug 24, 2021)

  • 20.4.2(Apr 21, 2020)

    Release Date: April 15th, 2020

    • Fix comparison conditions (equals, not_equals, gt, gte, lt, lte on aiodynamo.expressions.F and aiodynamo.expressions.Size via aiodynamo.expressions.F.size() to support referencing other fields (using aiodynamo.expressions.F)
    • Fix timeout handling in aiohttp based client.
    Source code(tar.gz)
    Source code(zip)
  • 20.4.1(Apr 21, 2020)

    Release Date: April 13th, 2020

    • Fixed put_item and delete_item with a condition which does not carry any values.
    • Wrap underlying HTTP client errors, such as connection issues, so networking issues during requests are retried.
    Source code(tar.gz)
    Source code(zip)
  • 20.4(Apr 21, 2020)

    Release Date: April 3rd, 2020

    • Fixed scan with a projection but no filter_expression.
    • Fixed logs leaking session tokens (request sending) and keys (metadata fetch).
    Source code(tar.gz)
    Source code(zip)
  • 20.3(Apr 21, 2020)

    Release Date: March 31st, 2020

    • Added TTL support
    • Added support for pluggable HTTP clients. Built in support for httpx and aiohttp.
    • Added custom client implementation.
    • Added custom credentials loaders, with support for custom credential loaders.
    • Fixed a typo in delete_item
    • Improved item deserialization performance
    • Improved overall client performance, especially for query, scan and count, which are now up to twice as fast.
    • Changed condition, key condition and filter expression APIs to not rely on boto3.
    • Moved aiodynamo.models.F to aiodynamo.expressions.F.
    • Removed boto3 dependency
    • Removed botocore dependency
    • Removed aiobotocore dependency
    Source code(tar.gz)
    Source code(zip)
  • 19.9(Apr 21, 2020)

  • 19.3(Mar 6, 2019)

  • 19.1(Jan 22, 2019)

Implementing basic MySQL CRUD (Create, Read, Update, Delete) queries, using Python.

MySQL with Python Implementing basic MySQL CRUD (Create, Read, Update, Delete) queries, using Python. We can connect to a MySQL database hosted locall

MousamSingh 5 Dec 01, 2021
AWS SDK for Python

Boto3 - The AWS SDK for Python Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to wri

the boto project 7.8k Jan 04, 2023
Pure Python MySQL Client

PyMySQL Table of Contents Requirements Installation Documentation Example Resources License This package contains a pure-Python MySQL client library,

PyMySQL 7.2k Jan 09, 2023
An asyncio compatible Redis driver, written purely in Python. This is really just a pet-project for me.

asyncredis An asyncio compatible Redis driver. Just a pet-project. Information asyncredis is, like I've said above, just a pet-project for me. I reall

Vish M 1 Dec 25, 2021
Toolkit for storing files and attachments in web applications

DEPOT - File Storage Made Easy DEPOT is a framework for easily storing and serving files in web applications on Python2.6+ and Python3.2+. DEPOT suppo

Alessandro Molina 139 Dec 25, 2022
Use SQL query in a jupyter notebook!

SQL-query Use SQL query in a jupyter notebook! The table I used can be found on UN Data. Or you can just click the link and download the file undata_s

Chuqin 2 Oct 05, 2022
Official Python low-level client for Elasticsearch

Python Elasticsearch Client Official low-level client for Elasticsearch. Its goal is to provide common ground for all Elasticsearch-related code in Py

elastic 3.8k Jan 01, 2023
A Python wheel containing PostgreSQL

postgresql-wheel A Python wheel for Linux containing a complete, self-contained, locally installable PostgreSQL database server. All servers run as th

Michel Pelletier 71 Nov 09, 2022
Pure-python PostgreSQL driver

pg-purepy pg-purepy is a pure-Python PostgreSQL wrapper based on the anyio library. A lot of this library was inspired by the pg8000 library. Credits

Lura Skye 11 May 23, 2022
PyPika is a python SQL query builder that exposes the full richness of the SQL language using a syntax that reflects the resulting query. PyPika excels at all sorts of SQL queries but is especially useful for data analysis.

PyPika - Python Query Builder Abstract What is PyPika? PyPika is a Python API for building SQL queries. The motivation behind PyPika is to provide a s

KAYAK 1.9k Jan 04, 2023
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.2k Jan 02, 2023
Asynchronous Python client for InfluxDB

aioinflux Asynchronous Python client for InfluxDB. Built on top of aiohttp and asyncio. Aioinflux is an alternative to the official InfluxDB Python cl

Gustavo Bezerra 159 Dec 27, 2022
aioodbc - is a library for accessing a ODBC databases from the asyncio

aioodbc aioodbc is a Python 3.5+ module that makes it possible to access ODBC databases with asyncio. It relies on the awesome pyodbc library and pres

aio-libs 253 Dec 31, 2022
Python script to clone SQL dashboard from one workspace to another

Databricks dashboard clone Unofficial project to allow Databricks SQL dashboard copy from one workspace to another. Resource clone Setup: Create a fil

Quentin Ambard 12 Jan 01, 2023
python-bigquery Apache-2python-bigquery (๐Ÿฅˆ34 ยท โญ 3.5K ยท ๐Ÿ“ˆ) - Google BigQuery API client library. Apache-2

Python Client for Google BigQuery Querying massive datasets can be time consuming and expensive without the right hardware and infrastructure. Google

Google APIs 550 Jan 01, 2023
Application which allows you to make PostgreSQL databases with Python

Automate PostgreSQL Databases with Python Application which allows you to make PostgreSQL databases with Python I used the psycopg2 library which is u

Marc-Alistair Coffi 0 Dec 31, 2021
A fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching.

:: Description :: A fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching :: Requirements :: Requires P

ESN Social Software 549 Nov 18, 2022
Pony Object Relational Mapper

Downloads Pony Object-Relational Mapper Pony is an advanced object-relational mapper. The most interesting feature of Pony is its ability to write que

3.1k Jan 04, 2023
A supercharged SQLite library for Python

SuperSQLite: a supercharged SQLite library for Python A feature-packed Python package and for utilizing SQLite in Python by Plasticity. It is intended

Plasticity 703 Dec 30, 2022
A fast unobtrusive MongoDB ODM for Python.

MongoFrames MongoFrames is a fast unobtrusive MongoDB ODM for Python designed to fit into a workflow not dictate one. Documentation is available at Mo

getme 45 Jun 01, 2022