Watched variable changed

[w0rse]

We're encountering the following problem with cacheops on various places in our site. It all started after we changed wsgi settings (see below) to fight memory leaks. In particular, we blame added multithreading.

Traceback (most recent call last):

  File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  ...

  File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 82, in __len__
    self._result_cache = list(self.iterator())

  File "/usr/lib/python2.6/site-packages/cacheops/query.py", line 327, in iterator
    self._cache_results(cache_key, results)

  File "/usr/lib/python2.6/site-packages/cacheops/query.py", line 231, in _cache_results
    cache_thing(self.model, cache_key, results, cond_dnf, timeout=self._cachetimeout)

  File "/usr/lib/python2.6/site-packages/cacheops/query.py", line 57, in cache_thing
    txn.execute()

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 1515, in execute
    return execute(conn, stack)

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 1465, in _execute_transaction
    raise WatchError("Watched variable changed.")

WatchError: Watched variable changed.

Here's out set up:

Apache/2.2.16
mod_wsgi/3.3
Python/2.6.7
cacheops/0.9.1

httpd.conf

WSGIDaemonProcess test processes=4 threads=15 maximum-requests=200 stack-size=1048576 display-name=%{GROUP}
WSGIProcessGroup test
WSGISocketPrefix /var/run/wsgi
WSGIRestrictEmbedded On

settings.py

CACHEOPS_REDIS = {
    'host': '127.0.0.1',
    'port': 6379,
    'db': 1,
    'socket_timeout': 3,
}

CACHEOPS = {
        '*.*': ('all', 60*15),
}

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': '127.0.0.1:6379',
        'OPTIONS': {
            'DB': 3,
            'PASSWORD': '',
            'PARSER_CLASS': DefaultParser
        },
    },
}

[Suor]

Looks like threading issue with redis client. There were problems there which were likely fixed.
Try upgrading your python redis client ('redis' package) to latest version. If it won't help post an issue there - https://github.com/andymccurdy/redis-py

[w0rse]

Upgrading redis client didn't solve the problem. Posted an issue for redis-py: redis/redis-py#224

[w0rse]

Please review what @andymccurdy replied here: redis/redis-py#224 (comment)

[Suor]

Looks like my code here https://github.com/Suor/django-cacheops/blob/master/cacheops/invalidation.py#L111 have grown old and not compatible with redis-py connection pooling, more about it here redis/redis-py#224

I'll look into ways to fix it. Or if someone else does this I will appreciate a patch )

[andymccurdy]

I suspect you could use the code I wrote in this ticket: redis/redis-py#197

Assuming you haven't changed the logic, it's probably a drop in replacement for what you're trying to do.

[Suor]

Following andymccurdy's advice and his example code mostly I compiled a patch into pooling_refactor branch.
Tell me if it works, please.

[w0rse]

We installed pooling_refactor branch and getting this kind of errors on every request:

...

  File "/usr/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)

  File "/usr/lib/python2.6/site-packages/django/db/models/base.py", line 570, in save_base
    created=(not record_exists), raw=raw, using=using)

  File "/usr/lib/python2.6/site-packages/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)

  File "/usr/lib/python2.6/site-packages/cacheops/query.py", line 409, in _post_save
    invalidate_obj(old)

  File "/usr/lib/python2.6/site-packages/cacheops/invalidation.py", line 152, in invalidate_obj
    invalidate_from_dict(obj.__class__, obj.__dict__)

  File "/usr/lib/python2.6/site-packages/cacheops/invalidation.py", line 145, in invalidate_from_dict
    redis_conn.transaction(_invalidate)

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 243, in transaction
    pipe.watch(*watches)

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 1538, in watch
    return self.execute_command('WATCH', *names)

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 1405, in execute_command
    return self.immediate_execute_command(*args, **kwargs)

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 1424, in immediate_execute_command
    return self.parse_response(conn, command_name, **options)

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 1491, in parse_response
    self, connection, command_name, **options)

  File "/usr/lib/python2.6/site-packages/redis/client.py", line 289, in parse_response
    response = connection.read_response()

  File "/usr/lib/python2.6/site-packages/redis/connection.py", line 268, in read_response
    raise response

ResponseError: wrong number of arguments for 'watch' command

[Suor]

My bad, fixed that in my redis-py fork (I made a pull request).
Should work with https://github.com/Suor/redis-py and latest version from pooling_refactor branch

[Suor]

My patch got merged into redis-py master, so pooling_refactor should now work with it too

[Suor]

Encountered this problem myself, in no threading environment.
Found additional temporary solution - downgrading redis-py to 2.2.4. Cacheops master works better with it.