This isn't quite ready, I need to add test coverage for migrating old databases, esp. around the conditionals in __delitem__ and __contains__ by manually executing db.conn.execute() statements to add "old-style bytes". Let me know what y'all think anyway, @ziky90, @piskvorky

Details of the implementation can be found in the comments of methods decode_key, SqliteDict.__keysearch, and SqliteDict.__getitem__ about the v1.2- and v2.0+ differences.

I covered some of the more complex problems, such as, when a key is set in v1.2 using unicode, then later deleted also using unicode:

>>> db[u'unicode'] = 'stuff'    # v1.2
>>> db[u'unicode'] = 'rewritten' # v2.0

That the original utf-8 encoded b'unicode' gets deleted after the new pickled-unicode key u'unicode' is inserted. Some careful test coverage about transparently migrating old database keys into new ones is required. In this scenario, db.keys() would contain b'unicode' if it was written by v1.2, but after a rewrite with v2.0, then db.keys() would contain u'unicode', which might be tricky enough to warrant a mention in the README.

I'm afraid if I say too much more it will only confuse things :)

Development process was recorded:

• https://www.livecoding.tv/video/foss-gardening-sqlitedict-8/
• https://www.livecoding.tv/video/foss-gardening-sqlitedict-9/

Python 3.5.1 Windows 10 sqlitedict 1.4.0

t.py:

from sqlitedict import SqliteDict
db = SqliteDict('test.db', autocommit=True)
#db.close()
print('end')

    def __del__(self):
        # like close(), but assume globals are gone by now (do not log!)
        self.close(do_log=False)  # <- never return

i guess thread is daemon. when del silently already thread has stopped never process event in queue

Hi,

The stack = traceback.extract_stack()[:-1] instruction makes inserts too slow for my use case.

This PR adds a flag to disable this behavior if needed.

According to some basic testing, it speeds up inserts by ~20x:

Test code:

from sqlitedict import SqliteDict

def insert(outer_stack):
    d = SqliteDict(outer_stack=outer_stack)
    for i in range(10000):
        d["key_{}".format(i)] = "value_{}".format(i)
    d.close()

%%timeit
insert(outer_stack=True)

=> 2.68 s ± 42.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
insert(outer_stack=False)

=> 121 ms ± 3.04 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Took me some time to find the culprit, but after upgrading to sqlitedict 2.0.0 the writing is significantly slower.

I am writing with:

with SqliteDict("tmp.db") as tmp:
    tmp["tmp"] = 1

Is this expected due to some new functionality?

I ran into a problem in one of my programs where it was not committing some changes because my script exited shortly after the updates.

Here is a example script that reliably reproduces the effect (for me):

import json
import sqlitedict
import time

class Dict(sqlitedict.SqliteDict):
    def __init__(self, name):
        sqlitedict.SqliteDict.__init__(
            self,
            f'{name}.sqlite', autocommit=True,
            encode=json.dumps, decode=json.loads )

d = Dict('d')
now = int(time.time())

for i in range(100):
    d[i] = now

If I follow that up with a select, I only see 18 entries (sometimes I see zero entries):

$ sqlite3 d.sqlite 'SELECT * from unnamed'
0|1553346705
1|1553346705
2|1553346705
3|1553346705
4|1553346705
5|1553346705
6|1553346705
7|1553346705
8|1553346705
9|1553346705
10|1553346705
11|1553346705
12|1553346705
13|1553346705
14|1553346705
15|1553346705
16|1553346705
17|1553346705
18|1553346705

In the special case that I just open the database, read some item and close it again, the multi-threading support causes significant overhead. I would therefore suggest to make it optional.

How could this be achieved in a way that requires only small code changes?

    assert list() == d.setdefault('tkey', [])
    assert list(d) == ['tkey']
    assert len(d['tkey']) == 0
    d.setdefault('tkey', []).append(1)
    print list(d)
    print d['tkey']
    assert d['tkey'] == [1]

['tkey']
[]
Traceback (most recent call last):
  File "sqlitedict.py", line 368, in 
    assert d['tkey'] == [1]
AssertionError

setdefault doesn't set list object in storage. I had to implement it manually like:

if key in storage:
    oldVal = storage[key]
    oldVal.append(val)
    storage[key] = oldVal
else:
    storage[key] = val

Update the README.rst:

• step by step explanation on how to use the library (read/write/etc). "it's just dict with a commit" really doesn't cut it.
• add section on performance. this thing work effectively with large files, even larger than memory. it's AWESOME. that alone is a reason to use, much better than a json.dumps(allobjects) when starting up a project.
• highlight support for JSON. json is great, pickle is blanket banned in many places due to critical security vulnerability.

I wanted to say this library is AMAZING. I am gonna use that for so many things! And to think I almost skipped it because the intro was meh for a first use.

This change modifies how "autocommit" works. It forces each modification to block until the commit is complete. Without this, if the program exited after modifying the sqlitedict, the changes might not make it to disk. With this change, all changes are guaranteed to be on disk before the modification action returns.

Fixes #93 .

fails in python3.6 with

Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File ".../site-packages/sqlitedict.py", line 371, in run cursor.execute('PRAGMA synchronous=OFF') sqlite3.OperationalError: Safety level may not be changed inside a transaction

Closes Issue #24

This is a first pass implementation, requesting feedback.

Because of the asynchronous "fire and forget" nature of calling SqliteMultithread.execute(), there is no way for the calling thread to be made aware of an exception in the inner thread.

We handle such exception gracefully, logging to level ERROR the inner exception, the exception only, and the outer stacktrace (the true offending code from the caller's thread).

This might have a slight performance impact.

For the select* family of methods, the inner exception is raised in by calling thread before returning, but for "fire and forget" calls of the execute* family of methods, such as db['key'] = 'value', we may only throw the exception:

• on commit: somewhat mirroring asyncio's exceptions thrown when accessing the result of a Future.
• on close or __del__, (which now awaits confirmation before returning, allowing exceptions to be caught).
• at any subsequent statement after the time that the inner exception occurred.

commit() is made blocking by default (awaiting confirmation), except by implied commit when using by autocommit=True.

For the patient, the work was recorded:

• https://www.livecoding.tv/video/foss-gardening-sqlitedict-5/
• https://www.livecoding.tv/video/foss-gardening-sqlitedict-6/
• https://www.livecoding.tv/video/foss-gardening-sqlitedict-7/

Apologies for opening a issue for clearing some doubts.

It would be great if you enable discussion section for this repository.

Questions

1. Can SQLiteDict handle around 80 million key-value pair?
2. Once 80 million key-value pairs are stored in SQLiteDict is there any optimized way/ method to query every key-value pairs?

Thanks @vigneshshettyin

According to my measurements, the WAL journal mode has even better performance than batch update with autocommit=False. If you agree, it can be mentioned in the documentation...

If autocommit changed, it was not propagated to SqliteMultithread, since its autocommit value was set only in initialization phase. According to the documentation we can temporarily disable autocommit if we need performance, but the performance was still the same as with keeping autocommit enabled.

Test case 1

If autocommit is False, then the dict should not be saved if we forget to call commit. (This is the case if we initialize SqliteDict with autocommit=False)

my_dict = sqlitedict.SqliteDict(filename="autocommit-test.sqlite", autocommit=True)
my_dict.autocommit = False
my_dict["a"] = 12
del my_dict

my_dict = sqlitedict.SqliteDict(filename="autocommit-test.sqlite")
print("a" in my_dict) # Should be False, but it is True (commit was called)
my_dict.terminate()

Test case 2

Performance measusements with this file:

import sqlitedict
import time
import contextlib

@contextlib.contextmanager
def measure(name):
    start = time.time()
    yield
    end = time.time()
    print(f"Elapsed for {name}: {end-start:.1f} seconds")

def fill_values(in_dict):
    for index in range(1_000):
        in_dict[index] = index

with measure("autocommit=False -> True"):
    my_dict = sqlitedict.SqliteDict(filename="mytest.sqlite", autocommit=False)
    my_dict.autocommit = True
    fill_values(my_dict)
    my_dict.commit()
    my_dict.terminate()

with measure("autocommit=True"):
    my_dict = sqlitedict.SqliteDict(filename="mytest.sqlite", autocommit=True)
    fill_values(my_dict)
    my_dict.commit()
    my_dict.terminate()

with measure("autocommit=False"):
    my_dict = sqlitedict.SqliteDict(filename="mytest.sqlite", autocommit=False)
    fill_values(my_dict)
    my_dict.commit()
    my_dict.terminate()

with measure("autocommit=True -> False"):
    my_dict = sqlitedict.SqliteDict(filename="mytest.sqlite", autocommit=True)
    my_dict.autocommit = False
    fill_values(my_dict)
    my_dict.commit()
    my_dict.terminate()

Results:

# Original
❯ python .\sqlitedict-test.py
Elapsed for autocommit=False -> True: 1.9 seconds
Elapsed for autocommit=True: 2.1 seconds
Elapsed for autocommit=False: 0.1 seconds
Elapsed for autocommit=True -> False: 1.5 seconds # <------

# Fixed version
❯ python .\sqlitedict-test.py
Elapsed for autocommit=False -> True: 1.9 seconds
Elapsed for autocommit=True: 2.2 seconds
Elapsed for autocommit=False: 0.1 seconds
Elapsed for autocommit=True -> False: 0.2 seconds # <------

You can be inspired with this pull request and reject this one, or I may can change it to follow the contributing guides. For me using WAL journal mode with autocommit was a perfect solution, I just wanted to let you know that it did not work as I expected.

Currently, .itervalues() execute SQL SELECT value FROM "%s" ORDER BY rowid;

Feature request:

.itervalues()[1:5] execute SQL SELECT value FROM "%s" ORDER BY rowid LIMIT 4 OFFSET 1.

please check singlton https://github.com/pycontribs/tendo/issues/55

So sending a kill signal to process with sqlitedict will not close (not all the time) the db connection and the proccess can hang forever. Causing the "singlton" lock file to no be removed.

What can cause it ?