Bringing Async Capabilities to django ORM

Overview

Disclaimer: Don't use this module in production it's still in active development.

Django Async Orm

Django module that brings async to django ORM.

Installing

python -m pip install django-async-orm

then add django_async_orm to your INSTALLED_APPS list:

INSTALLED_APPS = [
    ...,
    'django_async_orm'
]

Usage

Django Async Orm will patch all your existing models to add async_* prefixed methods. To be

example:

class MyModel(models.Model):
    name = models.CharField(max_length=250)

you can use it as follow:

async def get_model():
    return await  MyModel.objects.async_get(name="something")

you can also iterate over a query set with async for:

async def all_models():
    all_result_set = await MyModel.objects.async_all()
    async for obj in all_result_set:
        print(obj)

Some wrappers are also available for template rendering, form validation and login/logout

Async login

from django_async_orm.wrappers import async_login

async def my_async_view(request):
    await async_login(request)
    ...

Form validation

from django_async_orm.wrappers import async_form_is_valid
async def a_view(request):
    form = MyForm(request.POST)
    is_valid_form = await async_form_is_valid(form)
    if is_valid_form:
        ...
    

Django ORM support:

This is an on going projects, not all model methods are ported.

Manager:

methods supported comments
Model.objects.async_get
Model.objects.async_create
Model.objects.async_bulk_create
Model.objects.async_bulk_update
Model.objects.async_get_or_create
Model.objects.async_update_or_create
Model.objects.async_earliest
Model.objects.async_latest
Model.objects.async_first
Model.objects.async_last
Model.objects.async_in_bulk
Model.objects.async_delete
Model.objects.async_update
Model.objects.async_exists
Model.objects.async_explain
Model.objects.async_raw
Model.objects.async_all
Model.objects.async_filter
Model.objects.async_exclude
Model.objects.async_complex_filter
Model.objects.async_union
Model.objects.async_intersection
Model.objects.async_difference
Model.objects.async_select_for_update
Model.objects.async_prefetch_related
Model.objects.async_annotate
Model.objects.async_order_by
Model.objects.async_distinct
Model.objects.async_difference
Model.objects.async_extra
Model.objects.async_reverse
Model.objects.async_defer
Model.objects.async_only
Model.objects.async_using
Model.objects.async_resolve_expression
Model.objects.async_ordered
__aiter__
__repr__
__len__
__getitem__
Model.objects.async_iterator

Model:

methods supported comments
Model.async_save
Model.async_update
Model.async_delete
...

User Model / Manager

methods supported comments
UserModel.is_authenticated
UserModel.is_super_user
UserModel.objects.async_create_user
...

Foreign object lazy loading:

Not supported

Wrappers:

methods supported comments
wrappers.async_render
wrappers.async_login
wrappers.async_logout
Comments
  • add none query feature and test for the same

    add none query feature and test for the same

    Added none query feature:

    in Django ORM, calling none() will create a queryset that never returns any objects and no query will be executed when accessing the results. A qs.none() queryset is an instance of EmptyQuerySet.

    example:

    Entry.objects.none() <QuerySet []> from django.db.models.query import EmptyQuerySet isinstance(Entry.objects.none(), EmptyQuerySet) True

    opened by rkisdp 4
  • replace the default ThreadPoolExecutor with gevent.threadpool.ThreadPoolExecutor

    replace the default ThreadPoolExecutor with gevent.threadpool.ThreadPoolExecutor

    I have no idea if this will work, but maybe it will?

    Here's a bit of my research on what I was thinking: https://github.com/rednaks/django-async-orm/discussions/9

    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabce2e9d0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabf109bb0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabf109250>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabc5654f0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabd7846d0>
    ^CGracefully stopping... (press Ctrl+C again to force)
    Stopping benchmark-django-fastapi_web_1      ... done
    Stopping benchmark-django-fastapi_postgres_1 ... done
    
    

    looks kind of close ... although something is expecting concurrent.futures.Future instead of also allowing gevent.threadpool._FutureProxy

    update

    a recursion error of some sort logged from here : https://github.com/rednaks/django-async-orm/pull/7/files#diff-142ac986547c5b13f77ae025cb60609eed65082cc41ac6e81f4b778422d50017R116

    recursion.error.txt

    opened by allen-munsch 4
  • TypeError: 'NoneType' object is not iterable

    TypeError: 'NoneType' object is not iterable

    Thanks for sharing this library. Very interesting!

    Any suggestions on why NoneType would be returned from async_all?

    web              |   File "/venv/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    web              |     await super().__call__(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    web              |     await self.middleware_stack(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    web              |     raise exc from None
    web              |   File "/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    web              |     raise exc from None
    web              |   File "/venv/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    web              |     await response(scope, receive, sender)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    web              |     await response(scope, receive, sender)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
    web              |     await route.handle(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
    web              |     await self.app(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
    web              |     response = await func(request)
    web              |   File "/venv/lib/python3.8/site-packages/fastapi/routing.py", line 201, in app
    web              |     raw_response = await run_endpoint_function(
    web              |   File "/venv/lib/python3.8/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
    web              |     return await dependant.call(**values)
    web              |   File "/app/api/generated_app/main.py", line 100, in get_buildings
    web              |     data = await get_data()
    web              |   File "/app/api/generated_app/main.py", line 88, in get_data
    web              |     async for building in data:
    web              |   File "/venv/lib/python3.8/site-packages/django_async_orm/query.py", line 71, in __aiter__
    web              |     return AsyncIter(self._result_cache)
    web              |   File "/venv/lib/python3.8/site-packages/django_async_orm/iter.py", line 6, in __init__
    web              |     self._iter = iter(iterable)
    web              | TypeError: 'NoneType' object is not iterable
    

    py3.8.10 django3.2

    async def get_data():
        data = await Building.objects.async_all()
        result = []
        async for building in data:
            print(data, dir(data))
            print(building.__dict__)
            result.append({"id": building.id, "number": building.number})
        return result
    
    @app.get("/buildings", response_model=List[BuildingSummary])
    async def get_buildings(limit: conint(le=500), skip: conint(le=500)) -> List[BuildingSummary]:
        """
        Building List
        """
        # this is a custom thing
        data = await get_data()
        return data
    
    opened by allen-munsch 3
  • PR for issue #2

    PR for issue #2

    async def get_data():
        return await Building.objects.async_all()
    
    @app.get("/buildings", response_model=List[BuildingSummary])
    async def get_buildings(
        limit: conint(le=500),
        skip: conint(le=500),
        x_pm_org: Optional[List[int]] = Header(None, alias="x-pm-org"),
        x_pm_org_id: Optional[List[int]] = Header(None, alias="x-pm-org-id"),
        x_pm_user_type: Optional[str] = Header(None, alias="x-pm-user-type"),
    ) -> List[BuildingSummary]:
        """
        Building List
        [{'ApiKeyAuth': []}, {'PmOrg': []}]
        """
    
        org.set(x_pm_org)
        pydant, item_all = paths["get_buildings"]
    
    
    
        # this works
        data = await get_data()
    
        # this also works
        data2 = await Buildings.objects.async_all()
    
    
    
        result = []
        async for item in data[skip:limit]:
            print(item.__dict__)
            result.append(pydant.from_orm(item).dict())
        return result
    

    Related to: #2

    I just tested it locally and it appears to work.

    Thanks for the quick response.

    Not sure where to ask this, possibly could enable the "discussions" section on the repo?

    But how does the ThreadPoolExecutor work with something like psycogreen/eventlet?

    opened by allen-munsch 1
  • comply with official django async orm syntax

    comply with official django async orm syntax

    when we patch models, we prefix the methods with async_ but the new official django api prefixes the methods with a ex:

    patched api : sync objects.get => async objects.async_get official api: sync objects.get => async objects.aget

    so to make it forward compatible we need to comply to django's new api.

    be aware to keep our api so we don't break the internet

    opened by rednaks 0
  • Problem with default update_or_create()

    Problem with default update_or_create()

    Hello! After installing django_async_orm facing such problem with default update_or_create() function

    image

    Do you have any ideas by a chance why is it going like this?

    python 3.8 django 3.2.9 django-async-orm latest

    opened by max4nik 0
  • Count feature, related test and import optimisation

    Count feature, related test and import optimisation

    Hi, @rednaks,

    I have added the count feature and test for the same, and with that, I have also changed some other tests accordingly and removed unused imports from the file. Please have a look and give your feedback.

    opened by rkisdp 8
  • Perhaps it is better to use channels?

    Perhaps it is better to use channels?

    Perhaps it's worth starting django-channels library for asynchronous db calls?

    There is a function there. And, as I understand it, it will be more effective than it is now. image

    opened by daveusa31 0
Releases(0.1.4)
Piccolo - A fast, user friendly ORM and query builder which supports asyncio.

A fast, user friendly ORM and query builder which supports asyncio.

919 Jan 04, 2023
SQLModel is a library for interacting with SQL databases from Python code, with Python objects.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Sebastián Ramírez 9.1k Dec 31, 2022
Python helpers for using SQLAlchemy with Tornado.

tornado-sqlalchemy Python helpers for using SQLAlchemy with Tornado. Installation $ pip install tornado-sqlalchemy In case you prefer installing from

Siddhant Goel 122 Aug 23, 2022
ORM for Python for PostgreSQL.

New generation (or genius) ORM for Python for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
Sqlalchemy-databricks - SQLAlchemy dialect for Databricks

sqlalchemy-databricks A SQLAlchemy Dialect for Databricks using the officially s

Flynn 19 Nov 03, 2022
Prisma Client Python is an auto-generated and fully type-safe database client

Prisma Client Python is an unofficial implementation of Prisma which is a next-generation ORM that comes bundled with tools, such as Prisma Migrate, which make working with databases as easy as possi

Robert Craigie 930 Jan 08, 2023
SQLAlchemy support for aiohttp.

aiohttp-sqlalchemy SQLAlchemy 1.4 / 2.0 support for AIOHTTP. The library provides the next features: initializing asynchronous sessions through a midd

Ruslan Ilyasovich Gilfanov 5 Dec 11, 2022
A single model for shaping, creating, accessing, storing data within a Database

'db' within pydantic - A single model for shaping, creating, accessing, storing data within a Database Key Features Integrated Redis Caching Support A

Joshua Jamison 178 Dec 16, 2022
Python 3.6+ Asyncio PostgreSQL query builder and model

windyquery - A non-blocking Python PostgreSQL query builder Windyquery is a non-blocking PostgreSQL query builder with Asyncio. Installation $ pip ins

67 Sep 01, 2022
A Python Library for Simple Models and Containers Persisted in Redis

Redisco Python Containers and Simple Models for Redis Description Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm

sebastien requiem 436 Nov 10, 2022
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.2k Dec 26, 2022
An async ORM. 🗃

ORM The orm package is an async ORM for Python, with support for Postgres, MySQL, and SQLite. ORM is built with: SQLAlchemy core for query building. d

Encode 1.7k Dec 28, 2022
A simple project to explore the number of GCs when doing basic ORM work.

Question: Does Python do extremely too many GCs for ORMs? YES, OMG YES. Check this out Python Default GC Settings: SQLAlchemy - 20,000 records in one

Michael Kennedy 26 Jun 05, 2022
Global base classes for Pyramid SQLAlchemy applications.

pyramid_basemodel pyramid_basemodel is a thin, low level package that provides an SQLAlchemy declarative Base and a thread local scoped Session that c

Grzegorz Śliwiński 15 Jan 03, 2023
A curated list of awesome tools for SQLAlchemy

Awesome SQLAlchemy A curated list of awesome extra libraries and resources for SQLAlchemy. Inspired by awesome-python. (See also other awesome lists!)

Hong Minhee (洪 民憙) 2.5k Dec 31, 2022
Pydantic model support for Django ORM

Pydantic model support for Django ORM

Jordan Eremieff 318 Jan 03, 2023
Redis OM Python makes it easy to model Redis data in your Python applications.

Object mapping, and more, for Redis and Python Redis OM Python makes it easy to model Redis data in your Python applications. Redis OM Python | Redis

Redis 568 Jan 02, 2023
Solrorm : A sort-of solr ORM for python

solrorm : A sort-of solr ORM for python solrpy - deprecated solrorm - currently in dev Usage Cores The first step to interact with solr using solrorm

Aj 1 Nov 21, 2021
Object mapper for Amazon's DynamoDB

Flywheel Build: Documentation: http://flywheel.readthedocs.org/ Downloads: http://pypi.python.org/pypi/flywheel Source: https://github.com/stevearc/fl

Steven Arcangeli 128 Dec 31, 2022
Twisted wrapper for asynchronous PostgreSQL connections

This is txpostgres is a library for accessing a PostgreSQL database from the Twisted framework. It builds upon asynchronous features of the Psycopg da

Jan Urbański 104 Apr 22, 2022