Get list of Cache Keys in Django

Question:

I’m trying to understand how Django is setting keys for my views. I’m wondering if there’s a way to just get all the saved keys from Memcached. something like a cache.all() or something. I’ve been trying to find the key with cache.has_key('test') but still cant figure out how the view keys are being named.

UPDATE: The reason I need this is because I need to manually delete parts of the cache but dont know the key values Django is setting for my cache_view key

Asked By: Brenden

||

Answers:

There are some weird workarounds you can do to get all keys from the command line, but there is no way to do this with memcached inside of Django. See this thread.

Answered By: Jeremy Blanchard

You can use memcached_stats from: https://github.com/dlrust/python-memcached-stats. This package makes it possible to view the memcached keys from within the python environment.

Answered By: dino

As mentioned there is no way to get a list of all cache keys within django. If you’re using an external cache (e.g. memcached, or database caching) you can inspect the external cache directly.

But if you want to know how to convert a django key to the one used in the backend system, django’s make_key() function will do this.

https://docs.djangoproject.com/en/1.8/topics/cache/#cache-key-transformation

>>> from django.core.cache import caches
>>> caches['default'].make_key('test-key')
u':1:test-key'
Answered By: Don Sarratt

If this is not too out of date, I have had similar issue, due I have had to iterate over whole cache. I managed it, when I add something to my cache like in following pseudocode:

#create caches key list if not exists
if not my_cache.get("keys"):
    my_cache.set("keys", [])

#add to my cache
my_cache.set(key, value)

#add key to keys
if key not in my_cache.get("keys"):
    keys_list = my_cache.get("keys")
    keys_list.append(key)
    my_cache.set("keys", keys_list)
Answered By: Bobesh

For RedisCache you can get all available keys with.

from django.core.cache import cache

cache.keys('*')
Answered By: Eduard Mukans

For debugging, you can temporarily switch to LocMemCache instead of PyMemcacheCache:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'unique-snowflake',
    }
}

then see this question:

from django.core.cache.backends import locmem
print(locmem._caches)
Answered By: Boris Verkhovskiy

The Memcached documentation recommends that instead of listing all the cache keys, you run memcached in verbose mode and see everything that gets changed. You should start memcached like this

memcached -vv

and then it will print the keys as they get created/updated/deleted.

Answered By: Boris Verkhovskiy

this helps.

Ref:

https://lzone.de/blog/How-to%20Dump%20Keys%20from%20Memcache

https://github.com/dlrust/python-memcached-stats

import re, telnetlib, sys

key_regex = re.compile(r"ITEM (.*) [(.*); (.*)]")
slab_regex = re.compile(r'STAT items:(.*):number')

class MemcachedStats:

    def __init__(self, host='localhost', port='11211'):
        self._host = host
        self._port = port
        self._client = None

    @property
    def client(self):
        if self._client is None:
            self._client = telnetlib.Telnet(self._host, self._port)
        return self._client

    def command(self, cmd):
        ' Write a command to telnet and return the response '
        self.client.write("{}n".format(cmd).encode())
        res = self.client.read_until('END'.encode()).decode()
        return res

    def slab_ids(self):
        ' Return a list of slab ids in use '
        slab_ids =  slab_regex.findall(self.command('stats items'))
        slab_ids = list(set(slab_ids))
        return slab_ids

    def get_keys_on_slab(self, slab_id, limit=1000000):
        cmd = "stats cachedump {} {}".format(slab_id, limit)
        cmd_output = self.command(cmd)
        matches = key_regex.findall(cmd_output)
        keys = set()
        for match_line in matches:
            keys.add(match_line[0])
        return keys

    def get_all_keys(self):
        slab_ids = self.slab_ids()
        all_keys = set()
        for slab_id in slab_ids:
            all_keys.update(self.get_keys_on_slab(slab_id))
        return list(all_keys)


def main():
    m = MemcachedStats()
    print(m.get_all_keys())

if __name__ == '__main__':
    main()
Answered By: Asif Mohammed

In my setup with Django 3.2 there is a method to get a "raw" client for Redis which you can get the keys from.

from django.core.cache import cache

cache.get_client(1).keys()
Answered By: softarn

For Redis Backend

I’m going to add this answer because I landed on this SO question searching for exactly the same question but using a different cache backend. Also with REDIS in particular if you are using the same REDIS server for multiple applications you will want to scope your cache keys with the KEY_PREFIX option otherwise you could end up with cache keys from another application.

My answer is for if you have setup KEY_PREFIX in your settings.py and if you are using either redis_cache.RedisCache or django.core.cache.backends.redis.RedisCache

e.g.

 CACHES = {
    "default": {
        "BACKEND": "redis_cache.RedisCache",
        "LOCATION": f"redis://localhost:6379",
        "KEY_PREFIX": "my_prefix",
    },
 }

or

 CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": f"redis://localhost:6379",
        "KEY_PREFIX": "my_prefix",
    },
 }

redis_cache.RedisCache

from django.conf import settings
from django.core.cache import cache

cache_keys = cache.get_client(1).keys(
   f"*{settings.CACHES['default']['KEY_PREFIX']}*"
)

django.core.cache.backends.redis.RedisCache

Doing some tests shows that using Django’s built in RedisCache may already be scoped but in my case I’m doing it to be explicit. calling .keys("*") will also return keys that belong to celery tasks

from django.conf import settings
from django.core.cache import cache

cache_keys = cache._cache.get_client().keys(
    f"*{settings.CACHES['default']['KEY_PREFIX']}*"
)

Bonus: Deleting all app keys

If you want to clear the cache for your specific app instead of ALL the keys in REDIS you’ll want to using the prior technique and then call cache.delete_many(cache_keys) instead of cache.clear() as the Django Docs warns that using cache.clear() will remove ALL keys in your cache, not just the ones that are created by your app.

Answered By: JJK
Categories: questions Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.