Compare to update(), ~replace()~ write_back() provide a more customizable workflow to modify documents. One can fistly search the documents and modify them by thier conditions, then input new douments with thier doc_ids to ~replace()~ write_back().

Example Usage

from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage

db = TinyDB(storage=MemoryStorage)

db.purge()

dataset = [
    {"stock": ["cookies", "apple"]},
    {"stock": ["brownies"]},
    {"stock": ["cake"]},
    {"stock": ["cookies", "milk"]},
    {"stock": ["cake"]},
    {"stock": ["pie", "cake"]},
    {"stock": ["apple"]},
]

table = db.table('yumyum')
table.insert_multiple(dataset)
# search
yum = Query()
docs = table.search(yum.stock.any(["cake", "apple"]))
# modify docs
for doc in docs:
    for i, stock in enumerate(doc["stock"]):
        if stock == "cake":
            doc["stock"][i] = "water"
        if stock == "apple":
            doc["stock"][i] = "honey"
# write back
table.write_back(docs)

Or ~replace~ write_back with new documents

db.purge()

table = db.table('yumyum')
table.insert_multiple(dataset)

docs = table.all()
# save doc_ids
doc_ids = [doc.doc_id for doc in docs]
# modify docs
for i, doc in enumerate(docs):
    if len(doc["stock"]) == 1:
        docs[i] = {"best": doc["stock"][0]}
    elif len(doc["stock"]) > 1:
        docs[i] = {"good": doc["stock"]}

table.write_back(docs, doc_ids)

Could be a convenient feature :)

I'm looking to add new functionality to update. I would like to add an increment function first of all. I thought of something like this:

db.update(inc('int'), cond, eids)

And then change the update's present functionality to set, so:

db.update(set({'char': 'a'}), cond, eids)

Alternatively an interface such as this may be preferable:

db.update(cond, eids).inc('int')` and `db.update(cond, eids).set({'char': 'a'})

This change would allow the addition of new functions such as max, min, etc going forward.

I'm looking for thoughts/opinions and how best to structure this?

Hello, I am looking for some thoughts on a Tinydb extension that i am mulling over how to implement. Considering there is an active discussion about ideas for a 4.0 #284 I thought this might be a good time to bring it up.

To start off with here is a quick description of what I am doing in my application that I am interested in integrating directly into a TinyDB extension. Currently I have a DB of a size that is on the upper end of what TinyDB is recommended for. In that DB i have a particular field that is used very often in DB "queries". The values of that field are not guaranteed to be unique, however the number of entries that share that same value will always be small relative to the number of entries in the DB. Additionally the values associated with the field are sort-able. To speed up my application i have been pulling the entries out of the DB and creating a SortedCollection sorted by that particular field and I maintain the state of that collection along with every database modification, kind of like smart-cache. Then when I get to a point where i would normally perform a query on that field (or a combination of that field and something else), i use the sorted collection for faster lookup.

What I am trying to figure out is what would be the best way to integrate this behavior into TinyDB, so that it can be done transparently with the query interface. To start i know i would need to create a TableClass to implement the management and usage of the sorted "Index(s)". There also must be some way of specifying which field(s) should be indexed, either statically at the creation of the database object or dynamically based on the query being used. I am considering whether or not it makes sense to subclass the Query class so that the index is only used for a specific type of query.

Finally I need a way to determine, within each query consuming call, whether the query can be performed on an index or must be done on the database directly. I am imagining that I would need to do something like decomposing the query "path" to determine if it is a direct query of an indexed field or if it is a top level AND of a query on that field and something else. I think those are the only cases that would work, a top level OR definitely wouldn't.

I am interested to here any thoughts anyone might have on the feasibility of this kind of extension, how i might go about implementing it, or anything i may have missed. Despite having gotten this far, and created a few simple customization to TinyDB, I am still very new to this library and python in general. Any assistance would be greatly appriciated.

Hello, considers https://github.com/msiemens/tinydb/issues/226.

I have added basic __repr__ implementation for mentioned classes, would like to know if you see it as sufficient and which else classes should be implemented within the same way.

You may try the reprs with:

from tinydb import database, queries

print(repr(database.TinyDB('test.json')))
print(repr(database.Table(database.JSONStorage('test.json'), 'some name')))

Fruit = queries.Query()
print(repr(Fruit.type == 'peach'))

Achieving results:

TinyDB(tables={'_default'}, tables_count=1, default_table_documents_count=0, all_tables_documents_count=['_default=0'])
Table(name='some name', total=1, storage=<tinydb.storages.JSONStorage object at 0x10212c2e8>)
QueryImpl('==', ('type',), 'peach')

I was writing a package to build a simple graph on top of TinyDB, and came into a problem similar to #97. After reading the TinyDB source code, it isn't really possible (or at least comes with a huge performance penalty).

At the moment, whilst you can replace the Table class used by the TinyDB class, you have no control over the Element or StorageProxy classes. Instances of StorageProxy are created by the TinyDB class, and passed to the Table class. The StorageProxy then creates Element instances, with no way of changing it.

If I were to extend Table and replace the _read method with my own in order to use my own objects, there would be quite a performance penalty. The reason being, JSONStorage by way of using the json package has created dict objects for every element, then StorageProxy has gone and replaced each of those dicts with an Element object, and then I would replace them again with my own. Additionally, whilst I could pass my object to the json module with the object_hook argument, this object would then be replaced by the StorageProxy anyway, defeating the purpose.

The way I see it, the only purpose of the Element class is to give the element the eid attribute. I think this can be acheived in 2 ways;

1. It would be possible to create a custom json decoder, based upon json/simplejson. I have looked over the source code for those packages, and it would be possible to maintain a 'breadcrumb' list as the elements were decoded, and thus the eid would be known as the element is created. In this instance, the element object to create could be defined by the object_hook argument. The problem with this approach is that you couldn't use a different json package, as they would not have the capability of determining the eid as they decoded. For this reason, I don't think this is a good option.
2. Make the eid a field on the element. Then, any json package will work, and the object_hook argument that a lot of json packages seem to implement would work too.

I think option 2 opens up a lot of possibilies. The 'default' Element used by TinyDB could just be the dict as per json, or a slightly extended version with __getattr__ that allows the eid to be returned as a property. An optional Element could allow all fields to be accessed as properties. And of course, people can pass in their own objects.

There's a lot more specific implementation details I could go into, but I'll let you respond to this first.

What are your thoughts on passing the element from the write context (e.g. Table's insert) to the storage instead of all the data. This would allow storages that don't have to write all of the data every time any one (or a few) elements are created/modified/deleted.

BTW, you really nailed the API in this project. Nice and simple.

Since JSON is the default serilization format for TinyDB there's the datetime problem:

>>> from tinydb import TinyDB
>>> from datetime import datetime
>>> db = TinyDB("db.json")
>>> db.insert({"date": datetime.now()})
...
TypeError: datetime.datetime(2015, 2, 21, 17, 24, 17, 828569) is not JSON serializable

Many other databases handle datetime conversion for the user, and I would very much like TinyDB to do the same (It's usually fixed by specifying a custom encoding and a corresponding decoder when reading from the database).

Do you think this is a good idea?

Hi Markus,

My use-case for TinyDB requires a timestamp to be added to each document upon insertion and when that document is updated. This is useful for understanding database activities, establishing statistics and trends etc.

I have implemented this feature, nonetheless I think this is a fairly simple addition and would like to discuss and see if other people are interested in this feature.

A user settable flag is defined in the Table class. The default value is false (i.e. the feature is not active) so as to not break compatibility.

On insertion, the field 'created' is added to the document dictionary and initialized with the current UTC time in string format. The field 'updated' is also added but is initialized as empty string ''.

On update of a document, the field 'updated' is set to the current UTC time in string format.

I don't really care what time zone/format is chosen as long as it is consistent, UTC seems like a good choice. This might be a point of issue for some people and maybe it will require some customization down the track, although I like the simplicity of having only one time format.

Let me know what you think.

Cheers,

Miles

Setting create_dirs=True in __init__ will create the necessary parent directories to ensure db file will be created. This will further lessen what the user has to worry about.

My colleague had this problem today, trying to install through pip. My package, that he's trying to install, and which depends on tinydb, has the setup.py requirement for tinydb>=2.2. Are we missing something silly here or is there a problem with the uploaded 2.3 version to pip? Thanks!

Searching for tinydb>=2.2
Reading https://pypi.python.org/simple/tinydb/
Best match: tinydb 2.3.0-b
Downloading https://pypi.python.org/packages/source/t/tinydb/tinydb-2.3.0-b.zip#md5=15f0720f29a7f673a4bec54cd5ec6350
Processing tinydb-2.3.0-b.zip
Writing /tmp/easy_install-udrz_6/tinydb-2.3.0/setup.cfg
Running tinydb-2.3.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-udrz_6/tinydb-2.3.0/egg-dist-tmp-HZLQA2
No eggs found in /tmp/easy_install-udrz_6/tinydb-2.3.0/egg-dist-tmp-HZLQA2 (setup script problem?)
error: Could not find required distribution tinydb>=2.2

Lately, I've been thinking about how to move TinyDB forward and what the next steps are. And the more I thought about it, the more I became convinced, that TinyDB needs a v4.0.0.

Motivation

Why would we want a TinyDB v4.0? Why introducing a backwards-incompatible releases? In my view, the reasons to publish a new major release is threefold:

• Remove deprecated functionality that's been waiting to be removed for more than two years now.
• Fix design issues that have been introduced by a lack of vision for extension mechanisms in TinyDB.
• Simplify the architecture in order to fix other issues that cannot be solved without breaking backwards compatibility.

To elaborate on these reasons:

Deprecations

TinyDB is 6 years old now. The first stable release (v1.0.0) was published in July 2013. A year later in September 2014 there has been a major release (v2.0.0) that changed the data format and improved the API. Again a year later in November 2015, the next major release (v3.0.0) cleaned up the query language architecture and started moving non-core functionality to external packages.

Version 3.0.0 is now almost 4 years old. In the meantime TinyDB continued to evolve, including shifting the terminology from elements to documents in v3.6.0 and the deprecation of ujson support. Both of these changes have been major cleanups, but there hasn't been a major release of TinyDB that would actually get rid of the deprecated features. This results in cluttered code which makes it harder to understand the TinyDB software design/architecture.

In addition, Python 2, which TinyDB supports, will reach its end of life at the end of 2019. As TinyDB has quite a few places where it has to do extra work to support both Python 2 and 3 from the same code base, dropping Python 2 support would simplify the code even further.

TinyDB v4.0.0 would simplify the source code by removing deprecated features and in turn make it easier to understand the source and to develop one's own extensions. In addition, only Python 3 would be supported.

Extension Mechanisms

Right from the start, there have been two ways to extend TinyDB: Custom Storages and Custom Middlewares. As the popularity and usage of TinyDB increased, so did requests to make it possible to extend other parts of TinyDB. Thus, Custom Table Classes and Custom Storage Proxy Classes have been added. In addition, mechanisms to modify the default table class name and parameters as well as the default storage class have been introduced. As as result there are no less than seven places where TinyDB's behaviour can be modified.

Except for the first two, all extension mechanisms have been introduced as a result of user requests. At the time of each request, it seemed to be the best option to follow the path of least resistance when adding a new extension mechanism, refraining from any soft of breaking changes. But looking back it is apparent, that there was no real concept of how extending TinyDB should work in general.

TinyDB v4.0.0 would remove all extension mechanisms except for Custom Storages. All other extension mechanisms would be replaced by a unified extension concept as detailed below.

Architecture & API

To be honest, I'm not particularly proud about TinyDB's internal software architecture. As TinyDB evolved gradually, often simplicity of the software architecture was neglected. Now we're in a state, where there's a lot of unneeded indirection. Data access uses up to 5 classes, two of which are some form of proxy class: TinyDB → Table → StorageProxy → DocumentProxy → Storage. This makes TinyDB's source code complicated and also impacts performance (see #250). Fixing these design issues requires rearchitecting TinyDB. But this in turn requires breaking backwards compatibility as some extension mechanisms rely on the old software architecture.

Additionally, there's been discussion about inconsistencies in TinyDB's API regarding purging data (see #103). Removing these inconsistencies would break backwards compatibility.

TinyDB v4.0.0. would simplify the internal software architecture and remove inconsistencies from its API, making it easier to understand how TinyDB works and thus making it easier to extend.

Proposals

Deprecations

For the reasons outlined above, I propose to

• Put TinyDB v3 into maintenance mode, implementing only bug fixes but not adding new features,
• Remove all deprecated features,
  • Remove Element/Element ID APIs (following #158),
  • Fix Two test failures when ujson installed (#262), and
• Drop Python 2 support

Extension Mechanisms

I propose to replace all existing extension mechanisms with Custom Storages and Inheritance. Custom Storages continue to be a useful extension mechanism that is difficult to replicate by other means. In addition to Custom Storages, the main way to extend TinyDB and to modify its behaviour would be inheritance – to create subclasses of TinyDB. A famous example of this approach is Flask:

The Flask class has many methods designed for subclassing. You can quickly add or customize behavior by subclassing Flask (see the linked method docs) and using that subclass wherever you instantiate an application class.

In addition, the Flask docs state:

As you grow your codebase, don’t just use Flask – understand it. Read the source. Flask’s code is written to be read; its documentation is published so you can use its internal APIs. Flask sticks to documented APIs in upstream libraries, and documents its internal utilities so that you can find the hook points needed for your project.

With the new extension approach, TinyDB would aim to follow the same path: Instead of adding new extension mechanisms endlessly, users would be encouraged to subclass TinyDB and other classes in order to modify, how TinyDB fetches the last ID, how it calculates the next ID, how the default table name is determined, and other behaviours.

Implementing this requires making useful internal TinyDB methods part of the public API and documenting them in a way that makes it easy to overload them with custom behaviour. The documentation should provide examples of what types of extensions are possible by subclassing. Also, the source code itself should have its code comments reworked to make it easy to understand how TinyDB works from the first reading of the source code (based on ideas like literate programming).

The main challenge of implementing this approach is to find the right balance of how much of TinyDB's internal methods should become part of the public API. Making too few methods part of the public API makes it difficult to modify all aspects of TinyDB's behaviour. But making too many methods part of the public API makes it difficult to continue to evolve TinyDB without breaking the existing API and existing extensions and in addition cluttering the public API too much.

My approach regarding which methods to include in the public API would be to be conservative and – at first – include too few methods rather than too many. The reason behind this is that it's possible to move more methods to the public API after the fact without breaking the existing API whereas the opposite would break existing usage.

Architecture & API Changes

I propose to simplify to rearchitect TinyDB to use the following classes:

• TinyDB class
  • Create and manage tables
  • Forward calls to the default table
• Table class
  • Receive a storage instance from the TinyDB class
  • Modify table data
  • Cache query results to avoid unneeded I/O
• Storage class
  • Read and write data to a storage
• Query class
  • Provide searching and filtering
• Document class
  • Provide a thin wrapper around stored data that remembers the document's ID

There may be additional internal classes (such as the QueryImp and LRUCache classes we currently have), the classes outlined above should do the lion share of the work for TinyDB. All in all, the new architecture should provide a clear separation of concerns and responsibilities. Simplifying the architecture in this way would allow to fix issue #250 (StorageProxy read performance is abysmal). Also, a simple architecture makes it easy to understand how TinyDB works which in turn would impact how easy it is to extend TinyDB using inheritance (see above). In other words: Having a simpler architecture should make it easier to extend TinyDB.

In addition to architecture changes, I propose to also simplify TinyDB's API. For one thing, we could fix issue #103 (Inconsistency with purge functions) by making function names consistent between the TinyDB and Table classes. For another thing, I would propose to remove the write_back method as it complicates the API, probably is rarely used and can be implemented by subclassing, if needed. Also, I would like to make process_elements part of the internal API again as it's a core method of how data is manipulated and probably should not be modified by subclassing.

Feedback Requested

If you have thoughts, questions, comments or ideas regarding a possible TinyDB v4.0.0, especially regarding the proposals outlined above, feel free to comment and discuss on this issue 🙂

Bumps certifi from 2022.9.24 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

As a follow-up to #329 , how can I query for multiple docs using their IDs? Like

doc_ids = [1, 2, 5]
db.search(doc_ids=doc_ids)

db.get(doc_id) exists but only for a single document. I could iterate over the list of doc IDs yet this might not be as efficient.

Revamp of #468. Fixes #467.

I was in a position to fix this one because during the python-atomicwrites package fiasco I read somewhere that the author basically said people should be using os.replace anyway. Other platform-dependent errors were fixed basically by closing file handles in the right order.

Hey there! I'm trying to use tinydb in a project using mypy (thanks for the plugin), pylint and flake8. While mypy works and understands the query syntax thanks to the provided plugin, pylint and flake8 will complain about the following:

database.search(query["test"] == True)
pylint: singleton-comparison error: should be "query["test"] is True"
flake8: E712 use "if query["test"] is True: or if query["test"]:"

Apparently using those syntax changes as advised by these linting frameworks results in an error, because then the query is not a Callable anymore but directly a boolean value.

The only thing I found to "work" is to silence both linters on every line that makes use of the query language. Please tell me I'm just stupid and there is another, straight forward way of doing this :pray:

"Fun"-fact: I tried to circumvent this by passing a lambda function as query which works in code and pleases pylint and flake8 but then mypy will start to break out in tears because there is an obvious type mismatch between some random Callable and a a QueryLike object.

Hi, I noticed that the JsonStorage uses the write method to rewrite the entire file. However if a crash occurs while the file is being written, the db could be damaged. Why don't write to a temporary file and rename? You can check what I mean in PR #468 :)