I had a strange issue on my site and i made some test and it seems that when adding an object of a model, invalidation is not done for queries like all() Of course the all query (or other) is invalidated when an object is updated.

Exemple:

from django.db import models
import caching.base
class MyTest(caching.base.CachingMixin, models.Model):
  vartest = models.IntegerFields()
  objects = caching.base.CachingManager()
  def __repr__:
    return '<MyTest(id=%d, var=%d)>' % (self.id, self.vartest)

Then in a python/django shell:

MyTest.objects.create(vartest=1)
=> <MyTest(id=1, var=1)>
MyTest.objects.all()
=> [<MyTest(id=1, var=1)>]
MyTest.objects.create(vartest=2)
=> <MyTest(id=2, var=2)>
MyTest.objects.all()
=> [<MyTest(id=1, var=1)>]

t=MyTest.objects.get(id=1)
t.vartest=3
t.save()
t
=> <MyTest(id=1, var=3)>
MyTest.objects.all()
=> [<MyTest(id=1, var=3)>, <MyTest(id=2, var=2)>]

I understand why this happens but not really how to correct it

PS : i use the latest version, but tried too without our recent "if not to_cache"

OSS project looking for a good home

I haven't worked in the Django world for a few years so I don't have any motivation to maintain this project. Most of the maintenance involves keeping up with the release cycle for Django.

If i update on object in a current locale, say "fr", all keys for locale "fr" for this object will be invalidated, but not if after that a user in an other locale "en" check the object (common case in multilangual site), so i think you souldn't use the locale, or invalidate for all locales (which is more work, so i thing no locale is the best solution)

FYI, i'll fork the project in order to remove the locale

I doesn't seem like there is a way to configure Django-Cache-Machine to use multiple memecache nodes. I'm trying to use it with Amazon's Elasticache with no success. Is this an issue or am I completely missing something (very likely...)?

Many thanks! Lyle

As we're using new_keys to build up flush_keys and search_keys we still might need the proper keys in obj_keys to avoid objects not being invalidated properly.

There are some tests missing currently unfortunately but even without the patch the test-suite is failing for me locally so it's quite hard to reason about any particular changes. This should be straight forward enough though.

Let me know what you think :)

Hi,

There are some opened issues talking about problems with invalidation. I've implemented a couple of tests which current django cache machine fails on it:

def test_update_after_put(self):
        """ Test invalidation after do a put and insert in a previous cached filter """
        user1 = User.objects.create(name='John')
        user1.save()
        user2 = User.objects.create(name='Matt')
        user2.save()

        users = User.objects.filter(name__icontains='Jo')
        first_count = users.count()
        user2.name = 'Joseph'
        user2.save()
        users = User.objects.filter(name__icontains='Jo')
        assert users.count() == (first_count + 1)

    def test_update_after_post(self):
        """ Test invalidation after do a post and get again a list of a model """
        user1 = User.objects.create(name='Tom')
        user1.save()
        users = User.objects.all()
        initial_count = users.count()

        user2 = User.objects.create(name='Jerry')
        user2.save()
        users = User.objects.all()
        assert users.count() == (initial_count+1)

The problem here was being that django_cache_machine caches a Queryset and it is flushed only if some object associated with this queryset is changed. But if previously to the timeout, you do a update or create operation this queryset is not invalidated and returns older results.

I've forked this repo and create a new flush key associated to the model. It saves QS flush keys associated with the model. In this case, if new objects are created or updated all the queryset associated to the Model will be flushed.

It's not implemented any logical for filters, so we can flush some queryset which not be modified.

You can found it here: https://github.com/asketsus/django-cache-machine It has passed Travis-CI (including new tests).

Also this issue is fixed: https://github.com/jbalogh/django-cache-machine/issues/87

Related issue: https://github.com/jbalogh/django-cache-machine/issues/75 Related issue: https://github.com/jbalogh/django-cache-machine/issues/62

I've not sent a pull request due to these changes modify logical of caches. If you want to create a branch for me perfect :)

This problem is present in both PyPI version and latest github version of django-cache-machine. It manifests itself like this: Traceback: File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/core/handlers/base.py" in get_response

1.                 response = wrapped_callback(request, _callback_args, *_callback_kwargs)
   

   File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/views/generic/base.py" in view

2.         return self.dispatch(request, _args, *_kwargs)
   

   File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/views/decorators/csrf.py" in wrapped_view

3.     return view_func(_args, *_kwargs)
   

   File "/data/cache/buildout/eggs/djangorestframework-2.3.6-py2.7.egg/rest_framework/views.py" in dispatch

4.         response = self.handle_exception(exc)
   

   File "/data/cache/buildout/eggs/djangorestframework-2.3.6-py2.7.egg/rest_framework/views.py" in dispatch

5.         response = handler(request, _args, *_kwargs)
   

   File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/utils/decorators.py" in _wrapper

6.         return bound_func(_args, *_kwargs)
   

   File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/utils/decorators.py" in _wrapped_view

7.                 response = view_func(request, _args, *_kwargs)
   

   File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/utils/decorators.py" in bound_func

8.             return func(self, _args2, *_kwargs2)
   

   File "/home/jmv/linguist/staging/src/lingq_api/views.py" in wrapper

9.     language = get_object_or_404(profile.languages(), code=language)
   

   File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/shortcuts/init.py" in get_object_or_404

10.     return queryset.get(_args, *_kwargs)
    

    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/db/models/query.py" in get

11.     num = len(clone)
    

    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/db/models/query.py" in len

12.     self._fetch_all()
    

    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/db/models/query.py" in _fetch_all

13.         self._result_cache = list(self.iterator())
    

    File "/home/jmv/linguist/staging/parts/django-cache-machine/caching/base.py" in iter

14.             self.cache_objects(to_cache)
    

    File "/home/jmv/linguist/staging/parts/django-cache-machine/caching/base.py" in cache_objects

15.     cache.add(query_key, objects, timeout=self.timeout)
    

    File "/data/cache/buildout/eggs/django_devserver-0.8.0-py2.7.egg/devserver/utils/stats.py" in wrapped

16.     return stats.run(func, key, logger, _args, *_kwargs)
    

    File "/data/cache/buildout/eggs/django_devserver-0.8.0-py2.7.egg/devserver/utils/stats.py" in run

17.     value = func(_args, *_kwargs)
    

    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/core/cache/backends/memcached.py" in add

18.     return self._cache.add(key, value, self._get_memcache_timeout(timeout))
    

    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/core/cache/backends/memcached.py" in _get_memcache_timeout

19.     elif int(timeout) == 0:
    

Exception Type: TypeError at /api/languages/es/progress/ Exception Value: int() argument must be a string or a number, not 'object'

It was very difficult to find the culprit, but finally I've found out that CachedQuerySet stores Django DEFAULT_TIMEOUT in ints instances, which is a marker object. Django caching code compares this value with DEFAULT_TIMEOUT and if they are not identical, assumes that timeout is an integer value. But in unpickled instances, timeout value is always some different object.

My fix is not the most elegant, but I've spent 5+ hours trying to debug this issue and it makes any development using d-c-m impossible (bug appears and disappears at random).

The test cases I added in PR #115 demonstrate this behaviour. Here's the output of the unit tests running Django 1.7, 1.8, and 1.9. Only the first test setup (locmem_settings) is shown for brevity.

$ pip install -q django==1.7.11
$ python run_tests.py
Running tests for: locmem_settings
nosetests --verbosity=1
Creating test database for alias 'default'...
Creating test database for alias 'master2'...
....................S.............................
----------------------------------------------------------------------
Ran 50 tests in 1.645s

OK (SKIP=1)
Destroying test database for alias 'default'...
Destroying test database for alias 'master2'...

$ pip install -q django==1.8.9
$ python run_tests.py
Running tests for: locmem_settings
nosetests --verbosity=1
Creating test database for alias 'default'...
Creating test database for alias 'master2'...
....................S.............................
----------------------------------------------------------------------
Ran 50 tests in 0.900s

OK (SKIP=1)
Destroying test database for alias 'default'...
Destroying test database for alias 'master2'...

$ pip install -q django==1.9.2
$ python run_tests.py
Running tests for: locmem_settings
nosetests --verbosity=1
Creating test database for alias 'default'...
Creating test database for alias 'master2'...
....................S.............EE..............
======================================================================
ERROR: test_no_cache_values (tests.test_cache.CachingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/repos/django-cache-machine/tests/test_cache.py", line 567, in test_no_cache_values
    result = list(Addon.objects.filter(val__gt=130).values('val', 'author1'))
  File "/home/tim/repos/django-cache-machine/ve2/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/tim/repos/django-cache-machine/ve2/local/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/tim/repos/django-cache-machine/caching/base.py", line 114, in __iter__
    obj.from_cache = False
AttributeError: 'dict' object has no attribute 'from_cache'
-------------------- >> begin captured logging << --------------------
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:7:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:f8e6ed0ba783d44d957651a900d19f71'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:8:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:a91db8631bbb1c6e774f50c9b1402a86'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.addon:6:default', u'o:testapp.user:7:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:f8e6ed0ba783d44d957651a900d19f71', ':flush:85031e20aaa9a988dd8025d209b90565'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:8:default', u'o:testapp.addon:7:default', u'o:testapp.user:7:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:58f67a49fbe9ba0710be353b033086aa', ':flush:f8e6ed0ba783d44d957651a900d19f71', ':flush:a91db8631bbb1c6e774f50c9b1402a86'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:8:default', u'o:testapp.addon:8:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:2e31847edde14c9ede08eb7b280d3a5b', ':flush:a91db8631bbb1c6e774f50c9b1402a86'])
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test_no_cache_values_list (tests.test_cache.CachingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/repos/django-cache-machine/tests/test_cache.py", line 580, in test_no_cache_values_list
    result = list(Addon.objects.filter(val__gt=130).values_list('val', 'author1'))
  File "/home/tim/repos/django-cache-machine/ve2/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/tim/repos/django-cache-machine/ve2/local/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/tim/repos/django-cache-machine/caching/base.py", line 114, in __iter__
    obj.from_cache = False
AttributeError: 'tuple' object has no attribute 'from_cache'
-------------------- >> begin captured logging << --------------------
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:9:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:d283a2e5956cc2bea6bf68bfa0bcb73b'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:10:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:818a832060799c3a553eacb0b87bdd0a'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:9:default', u'o:testapp.addon:9:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:5012189813f2036adefa54f9e636c9b9', ':flush:d283a2e5956cc2bea6bf68bfa0bcb73b'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:10:default', u'o:testapp.user:9:default', u'o:testapp.addon:10:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:3fd60669f1fc41bdbb2034882750b7a3', ':flush:818a832060799c3a553eacb0b87bdd0a', ':flush:d283a2e5956cc2bea6bf68bfa0bcb73b'])
caching.invalidation: DEBUG: in expand_flush_lists
caching.invalidation: DEBUG: deleting object keys: set([u'o:testapp.user:10:default', u'o:testapp.addon:11:default'])
caching.invalidation: DEBUG: clearing flush lists: set([':flush:818a832060799c3a553eacb0b87bdd0a', ':flush:2e12aac9cc4290d47a0e3eba675bab90'])
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 50 tests in 1.052s

FAILED (SKIP=1, errors=2)
Destroying test database for alias 'default'...
Destroying test database for alias 'master2'...

Upon an initial investigation, this appears to be caused by the fact that QuerySet no longer calls self._clone directly.

This pull request adds support for a CACHE_EMPTY_QUERYSETS setting, which can be used to revert the changes in 47b7801, if caching of empty querysets is really desired.

I have a very simple caching model which is inherits from CachingMixin. When I call save() in Django 1.7 (to add a new instance), the cache is not invalidated (the new object does not show up in subsequent queries.)

When I switch back to Django 1.6, the same code works perfectly. Pulling the CachingManager and using 1.7 also works.

I am on trunk.

Hi all,

I do have a very strange issue:

1. I Have added the CachingMixin to my model ==> everything works fine
2. I have added the objects = CachingManager() to my model ==> all models validat, development server starts at the command line as expected. But it will not serve any browser requests (just hanging). Strange is that I can open the shell and and retrieve data through the model.

Any help is highly appreciated. I am not very confident to move this to production (event if it looks like it is working), as long I have not understood this issue. I also have no clue how to further debug.

Regards, Juergen

File "/usr/local/lib/python3.7/site-packages/caching/invalidation.py", line 251, in get_redis_backend host, port = server.split(':') ValueError: too many values to unpack (expected 2)

With redis://:[email protected]:6379/0

Adding a new field to a cached model causes an error on deployment because our code expects the new field to exist (ProgrammingError: column example_abc does not exist). Of course, the new field does not exist in previously-cached objects.

Any recommendations on how to handle schema migrations with Cache Machine?

A few ideas come to mind:

1. Invalidate the entire cache when the schema changes. Not ideal because it's far too broad and can cause a run on the DB for high-traffic sites.
2. Track all the queries associated with a given table and invalidate only those associated with a model. Better than clearing the whole cache, but this comes with the overhead of maintaining the flush lists of queries for a table and can still cause a run on the DB.
3. Create middleware that invalidates individual cached queries/objects when a migration-related error occurs. E.g., on a ProgrammingError, delete any cached query and retry the query. This feels clunky because any definition of "migration-related error" will probably be imprecise.
4. Do some cache warming. After a schema migration, instead of invalidating the cache, pull from the DB to update each cached query, and deploy when that's done. This seems like the best option, though it adds complexity and time to deployment.

Thoughts?

I am seeing very slow (minutes per request) performance when lists are invalidated. I traced that to the recursive flush list expansion. In its present form it allows for a flush list to be hit multiple times. The fix is to simply check if a list has already been expanded, and if it is - skip it.

I have created PR #106 . Would someone please review and pull in the changes into the main repository.

Thanks!