Hi,

Thanks for the awesome package (happy to use it with starlette). For rapid dev purpose, I'm relying on the SQLite backend and I see an odd behavior.

When I create tables with an in-memory db, those tables can never be found in sqlite_master afterwards. But when I rely on a file database, then that works fine.

Is there any gotcha for in-memory sqllite I should be aware of?

Cheers,

Sqlalchemy 1.4 is out with some breaking changes (e.g. RowProxy got removed in favor of Row or the removal of row.values() in favor of row._mapping.values())

closes #298 and closes #348

my version: 0.2.2 asyncpg version: 0.18.3 functions: database.fetch_one, database.fetch_all

dict(user_row) when user_row is a databases.backends.postgres.Record object returns an empty dict for me when using postgres through asyncpg.

dict(user_row.__dict__['_row']) works, so I know that at least in my case the record has successfully loaded and there is a dict somewhere. dict style user_row['id'] works just fine.

When using asyncpg, I am getting InterfaceError: cannot perform operation: another operation is in progress when running multiple db calls concurrently.

I am able to replicate and isolate the issue with force_rollback=True, although in my project where I encountered this bug, it is happening regardless.

Here is the isolated issue:

import asyncio
import databases
from starlette.applications import Starlette
from starlette.config import Config
from starlette.responses import JSONResponse

config = Config('.env')
DATABASE_URL = config('DATABASE_URL', default='postgresql://[email protected]:5432/postgres')

database = databases.Database(DATABASE_URL, force_rollback=True)
app = Starlette()


@app.on_event("startup")
async def startup():
    await database.connect()


@app.on_event("shutdown")
async def shutdown():
    await database.disconnect()


@app.route("/test", methods=["GET"])
async def test_view(request):
    await asyncio.gather(
        get_from_db(),
        get_from_db(),
    )
    return JSONResponse({"success": True})


async def get_from_db():
    return await database.fetch_all("SELECT pg_sleep(1)")


if __name__ == '__main__':
    from starlette.testclient import TestClient
    with TestClient(app) as test_client:
        test_client.get('/test')

Result

$ python app.py
Traceback (most recent call last):
  File "app.py", line 40, in <module>
    test_client.get('/test')
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/testclient.py", line 382, in request
    json=json,
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/testclient.py", line 211, in send
    raise exc from None
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/testclient.py", line 208, in send
    loop.run_until_complete(connection(receive, send))
  File "/Users/ryan/.pyenv/versions/3.7.2/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/middleware/errors.py", line 125, in asgi
    raise exc from None
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/middleware/errors.py", line 103, in asgi
    await asgi(receive, _send)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/exceptions.py", line 74, in app
    raise exc from None
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/exceptions.py", line 63, in app
    await instance(receive, sender)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/routing.py", line 41, in awaitable
    response = await func(request)
  File "app.py", line 28, in test_view
    get_from_db(),
  File "app.py", line 34, in get_from_db
    return await database.fetch_all("SELECT pg_sleep(1)")
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/databases/core.py", line 95, in fetch_all
    return await connection.fetch_all(query, values)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/databases/core.py", line 179, in fetch_all
    return await self._connection.fetch_all(self._build_query(query, values))
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/databases/backends/postgres.py", line 137, in fetch_all
    rows = await self._connection.fetch(query, *args)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/asyncpg/connection.py", line 421, in fetch
    return await self._execute(query, args, 0, timeout)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/asyncpg/connection.py", line 1412, in _execute
    with self._stmt_exclusive_section:
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/asyncpg/connection.py", line 1847, in __enter__
    'cannot perform operation: another operation is in progress')
asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app.py", line 40, in <module>
    test_client.get('/test')
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/testclient.py", line 415, in __exit__
    loop.run_until_complete(self.wait_shutdown())
  File "/Users/ryan/.pyenv/versions/3.7.2/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/testclient.py", line 435, in wait_shutdown
    self.task.result()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/testclient.py", line 420, in lifespan
    await inner(self.receive_queue.get, self.send_queue.put)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/routing.py", line 483, in asgi
    await self.shutdown()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/starlette/routing.py", line 468, in shutdown
    await handler()
  File "app.py", line 21, in shutdown
    await database.disconnect()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/databases/core.py", line 74, in disconnect
    await self._global_transaction.__aexit__()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/databases/core.py", line 256, in __aexit__
    await self.rollback()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/databases/core.py", line 297, in rollback
    await self._transaction.rollback()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/databases/backends/postgres.py", line 215, in rollback
    await self._transaction.rollback()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/asyncpg/transaction.py", line 219, in rollback
    await self.__rollback()
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/asyncpg/transaction.py", line 198, in __rollback
    await self._connection.execute(query)
  File "/Users/ryan/.local/share/virtualenvs/postgres-bug-tR_IhxOx/lib/python3.7/site-packages/asyncpg/connection.py", line 273, in execute
    return await self._protocol.query(query, timeout)
  File "asyncpg/protocol/protocol.pyx", line 301, in query
  File "asyncpg/protocol/protocol.pyx", line 659, in asyncpg.protocol.protocol.BaseProtocol._check_state
asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress

When running into this issue with force_rollback=False, since there isn't supposed to be a global connection and transaction, I was able to get around the issue by changing the following function:

async def get_from_db():
    return await database.fetch_all("SELECT pg_sleep(1)")

to:

async def get_from_db():
    async with Connection(database._backend) as conn:
        return await conn.fetch_all("SELECT pg_sleep(1)")

I tried using async with database.connection() as conn: but it would return the same connection that is being currently being used. I guess I was just trying to understand the logic of sharing the pool connection, instead of letting the pool allocate connections (when available), since asyncpg Connection only allows one statement at a time.

Closes #445

As discussed in the main issue i've added _mapping property to the result set interface. Both asyncpg Record and sqlalchemy Row have this property so it's safe to add it.

Maybe I am being an idiot and have forgot to install a package but I am having the following issue:

I have created an API with FastAPI and databases. When I run my files in a Conda environment with SQLAlchemy 1.4.41 installed there are no issues, but when I switch to a Python Virtual Environment with version 1.4.42, I am returned the following error:

Traceback (most recent call last):
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 404, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 75, in __call__
    raise exc
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 64, in __call__
    await self.app(scope, receive, sender)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/routing.py", line 680, in __call__
    await route.handle(scope, receive, send)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/routing.py", line 275, in handle
    await self.app(scope, receive, send)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/fastapi/routing.py", line 231, in app
    raw_response = await run_endpoint_function(
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/./app/endpoints/ingredient.py", line 33, in get_ingredient
    for id_, name in await db.fetch_all(query=SEARCH_INGREDIENT_QUERY, values={'search1': f"{query}%", 'search2': f"% {query}%"})
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/databases/core.py", line 142, in fetch_all
    return await connection.fetch_all(query, values)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/databases/core.py", line 250, in fetch_all
    return await self._connection.fetch_all(built_query)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/databases/backends/mysql.py", line 115, in fetch_all
    metadata = CursorResultMetaData(context, cursor.description)
  File "/Users/jacob/Documents/python/applications/gumbo/fastapi/venv/lib/python3.9/site-packages/sqlalchemy/engine/cursor.py", line 164, in __init__
    (
ValueError: not enough values to unpack (expected 5, got 4)

Looking at the file venv/lib/python3.9/site-packages/sqlalchemy/engine/cursor.py on line 164 I can see that:

    def __init__(self, parent, cursor_description):
        context = parent.context
        dialect = context.dialect
        self._tuplefilter = None
        self._translated_indexes = None
        self.case_sensitive = dialect.case_sensitive
        self._safe_for_cache = False

        if context.result_column_struct:   
            (
                result_columns,
                cols_are_ordered,
                textual_ordered,
                ad_hoc_textual,
                loose_column_name_matching,
            ) = context.result_column_struct      # <---- ERROR IS HERE

I decided to print context.result_column_struct and found that it returned ([], True, False, False).

I am not really sure how to go about fixing this. If it helps the modules I have installed in the venv are listed below:

aiomysql==0.1.1
anyio==3.6.1
click==8.1.3
databases==0.6.1
fastapi==0.85.1
h11==0.14.0
httptools==0.5.0
idna==3.4
protobuf==3.20.1
pydantic==1.10.2
PyMySQL==1.0.2
python-dotenv==0.21.0
PyYAML==6.0
sniffio==1.3.0
SQLAlchemy==1.4.42
starlette==0.20.4
typing_extensions==4.4.0
uvicorn==0.18.3
uvloop==0.17.0
watchfiles==0.17.0
websockets==10.3

https://github.com/encode/databases/releases/edit/untagged-d723639e4d24f39254b7

Changelog:

Added

• Support for Unix domain in connections (#423)
• Added asyncmy MySQL driver (#382)

Fixed

• Fix SQLite fetch queries with multiple parameters (#435)
• Changed Record type to Sequence (#408)

I'm deploying a FastAPI application on Google Cloud Run which connects to a Cloud SQL instance using this package. The crux of the issue is that connecting with:

db = databases.Database(url)
await db.connect()

fails whereas connecting through sqlalchemy's create_engine with

engine = create_engine(url)
engine.connect()

works.

The connection url uses unix_sock structure (docs here) rather than the regular sqlalchemy connection url, something like this:

#  all these urls work fine when connecting with sqlalchemy create_engine
"postgresql://user:pass@/db_name?host=/path/to/sock"
"postgresql+psycopg2://user:pass@/db_name?host=/path/to/sock"
"postgresql+pg8000://user:pass@/db_name?unix_sock=/path/to/sock/.s.PGSQL.5432"

I'm unsure whether this would be an issue with using async in the Google Cloud environment or something about how connection urls like the one above get translated in this package to work with sqlalchemy. I've posted on Stack Overflow about it here but thought I'd raise an issue here as well in case it was the latter.

Fixes #244 . Adding asyncmy as another mysql driver which is more up to date and faster in some areas.

• [x] Adding asyncmy
• [x] Fixing all tests (raw queries are failing)
• [x] Update the docs

Hi. I have a daemon with several asyncio tasks, each of which executes some requests. All of the tasks share the same connection.

After some time one of them begins to produce this error.

Failed code is simple :)

In main.py I have self.engine = Database(config.Config.SQLALCHEMY_DATABASE_URI) await self.engine.connect()

And in broken task await wait_for(self.engine.execute(query=statement), 10) What could be wrong?

I've been prototyping an API using starlette with SQLAlchemy and had a question about how to handle DB errors with the databases module.

# SQLAlchemy table definition
class Users(Base):
    __tablename__ = "users"
    id = Column(Integer, primary_key=True)
    username = Column(String(length=20), unique=True)
    password_hash = Column(String(length=80))

# Endpoint
@validate_schema(CREATE_USER_SCHEMA)
async def create_user(request):
    data = await request.json()
    query = Users.__table__.insert().values(username=data["username"], password_hash=hash_password(data["password"]))
    try:
        await database.execute(query)
    except exc.IntegrityError:
        return JSONResponse({"message": "User already exists."})
    return JSONResponse({"message": "Created user."})

When I try to create a user with a username that already exists, violating the UNIQUE constraint specified in the table definition, I receive a sqlite3.IntegrityError: UNIQUE constraint failed: users.username (venv/lib/python3.8/site-packages/aiosqlite/core.py:153: IntegrityError). This is not caught by the code above, which uses sqlalchemy.exc.IntegrityError.

I like to run tests in SQLite, and use Postgres in production; in other APIs I've built (with plain SQLAlchemy), sqlalchemy.exc generally does the trick regardless of the DB backend used.

Q: Is there a good way to catch errors independent of the DB used? For now, I'm catching both the SQLite and Postgres exceptions manually, but I was wondering what best practices for this case would look like using the databases module.

Thanks for creating starlette and databases, and sorry in advance if this is a duplicate.

Even though #495 mentions that the execute query returns the last inserted id, I am receiving None.

# from databases import Database
# here `db: Database`
res = await self.db.execute(
            'INSERT INTO sensor_data_filter (label, user_id, client_id, filters, "order") VALUES (:label, :user_id, :client_id, :filters, :order)',
            values=new_filter,
        )
print(res)  # returns `None`

Bumps wheel from 0.37.1 to 0.38.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I was trying to do something like this where the table name is part of the values in execute_many but it doesn't work...

        queries = []
        for name in unformatted:
            value = user_data[name]['value']
            leaderboard = name.replace("-",'_')
            queries.append(
                {
                    "table":leaderboard,
                    "uuid":uuid,
                    "user":user,
                    "prefix":prefix,
                    "suffix":suffix,
                    "value":value
                }
            )
            
        await lb_db.execute_many(query="""
            INSERT INTO :table VALUES 
            (:uuid,:user,:prefix,:suffix, :value) 
                ON CONFLICT(uuid) DO UPDATE SET 
            user = :user, prefix = :prefix, suffix = :suffix, value = :value
            """, values=queries)

asyncpg version: 0.25.0 PostgreSQL version: 12.11

We have been frequently receiving this Timeout Error

asyncio.exceptions.TimeoutError
[View similar errors](https://link.datadoghq.com/apm/error-tracking?issueId=...471-da7ad0900002)

Traceback (most recent call last):
  File /usr/local/lib/python3.9/site-packages/ddtrace/contrib/asyncpg/patch.py, line 89, in _traced_query
    return await method(*args, **kwargs)
File asyncpg/protocol/protocol.pyx, line 338, in query
asyncio.exceptions.TimeoutError

upon checking the logs, it's making an internal query SELECT pg_advisory_unlock_all ( ) CLOSE ALL UNLISTEN * RESET ALL on PostgreSQL database

Not sure why the tables are locking when we are only doing read operations from the db.

Hey everyone. After deploying my fastapi app on Deta and/or on Vercel, I got an error complaining about DatabaseBackend not running. However, the connect method is being called during the startup event of fastapi instance:

@chat_app.on_event("startup")
async def startup():
    await init_models(SQLALCHEMY_DATABASE_URL)
    await database.connect()

The weird thing is that the app is running as expected on Heroku; you can interract with the endpoints to test things out.

Here are the full logs on vercel:

[ERROR] AssertionError: DatabaseBackend is not running
Traceback (most recent call last):
  File "/var/task/vc__handler__python.py", line 293, in vc_handler
    response = asgi_cycle(__vc_module.app, body)
  File "/var/task/vc__handler__python.py", line 204, in __call__
    loop.run_until_complete(asgi_task)
  File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/var/task/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/var/task/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/task/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/var/task/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/var/task/starlette/middleware/base.py", line 68, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/var/task/app/main.py", line 75, in add_process_time_header
    response = await call_next(request)
  File "/var/task/starlette/middleware/base.py", line 46, in call_next
    raise app_exc
  File "/var/task/starlette/middleware/base.py", line 36, in coro
    await self.app(scope, request.receive, send_stream.send)
  File "/var/task/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/var/task/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/var/task/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/var/task/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/var/task/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/var/task/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/var/task/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/var/task/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/var/task/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/var/task/fastapi/routing.py", line 231, in app
    raw_response = await run_endpoint_function(
  File "/var/task/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/var/task/app/auth/router.py", line 62, in register
    results = await register_user(user)
  File "/var/task/app/auth/crud.py", line 131, in register_user
    fetched_user = await find_existed_user(user.email)
  File "/var/task/app/auth/crud.py", line 64, in find_existed_user
    return await database.fetch_one(query, values=values)
  File "/var/task/databases/core.py", line 147, in fetch_one
    async with self.connection() as connection:
  File "/var/task/databases/core.py", line 230, in __aenter__
    raise e
  File "/var/task/databases/core.py", line 227, in __aenter__
    await self._connection.acquire()
  File "/var/task/databases/backends/mysql.py", line 99, in acquire
    assert self._database._pool is not None, "DatabaseBackend is not running"

• Deta Demo
• Vercel Demo

Digging a bit deeper, I realized that the query calls were not working even though the database.connect is being called at the startup. It seems like some sort of a race condition. Not sure.