Getting values with the right type in Redis
Question:
I’m using redis in my python application to store simple values like counters and time stamp lists, but trying to get a counter and comparing it with a number I came across a problem.
If I do:
import redis
...
myserver = redis.Redis("localhost")
myserver.set('counter', 5)
and then try to get that value like this:
if myserver.get('counter') < 10:
myserver.incr('counter')
then I get a type error in the if statement because I’m comparing ‘5’ < 10, which means I’m storing an integer value and getting a string one (which can be considered as a different value).
My question is: is this supposed to work like that? I mean its a very basic type, I understand if I have to parse objects but an int? Seems that I’m doing something wrong.
Is there any configuration I’m missing?
Is there any way to make redis return the right type and not always a string?
I say this because its the same for lists and datetimes or even floating point values.
Could this be a problem with the redis-py client I’m using and not redis itself?
Answers:
Technically speaking you need to take care of that on your own.
However, have a look at this link, especially at the part of their README that refers to parsers and response callbacks, maybe that’s something you can use. Question would be whether this is an overkill for you or not.
Looks like that is how redis stores its data:
redis 127.0.0.1:6379> set counter 5
OK
redis 127.0.0.1:6379> type counter
string
redis 127.0.0.1:6379> incr counter
(integer) 6
redis 127.0.0.1:6379> type counter
string
If you really want to, you could probably monkeypatch the redis-py client to infer data types.
As @favoretti said, response callbacks will do the trick. It’s not complicate at all, just one line and all will be taken care of.
In [2]: import redis
In [3]: r = redis.Redis()
In [10]: r.set_response_callback('HGET', float)
In [11]: r.hget('myhash', 'field0')
Out[11]: 4.6
for hmget, it returns a list of strings, not one single string, so you need to construct a little more comprehensive callback function:
In [12]: r.set_response_callback('HMGET', lambda l: [float(i) for i in l])
In [13]: r.hmget('myhash', 'field0')
Out[13]: [4.6]
same for hgetall.
Here is my test. Two redis connections: one returns int type, the other float
import redis
age_field = redis.Redis()
age_field.set_response_callback('HGET', int)
age_field.hget('foo', 'age')
# OUT: 40
a =age_field.hget('foo', 'age')
type(a)
# OUT: <type 'int'>
gpa_field = redis.Redis()
gpa_field.set_response_callback('HGET', float)
gpa_field.hget('foo', 'gpa')
# OUT: 2.5
b = gpa_field.hget('foo', 'gpa')
type(b)
# OUT: <type 'float'>
You can set decode_respone like True
redis.StrictRedis(host="localhost", port=6379, db=0, decode_responses=True)
Though leveraging set_response_callback is fine for simple data types, if you’re wondering about the fastest-and-easiest route to storing things like dictionaries, list, tuples – and preserving the python-native datatypes they may-or-may-not contain – I recommend using python’s built-in pickle library:
# Imports and simplified client setup
>>> import pickle
>>> import redis
>>> client = redis.Redis()
# Store a dictionary
>>> to_store = {'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
>>> client.set('TestKey', pickle.dumps(to_store))
True
# Retrieve the dictionary you just stored.
>>> retrieved = pickle.loads(client.get('TestKey'))
{'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
Here’s a simple client that will reduce the pickle boilerplate in the above example and provide you with a clean interface for storing and retrieving native python datatypes to and from Redis:
"""Redis cache."""
import pickle
import redis
redis_host = redis.Redis()
class PythonNativeRedisClient(object):
"""A simple redis client for storing and retrieving native python datatypes."""
def __init__(self, redis_host=redis_host):
"""Initialize client."""
self.client = redis_host
def set(self, key, value, **kwargs):
"""Store a value in Redis."""
return self.client.set(key, pickle.dumps(value), **kwargs)
def get(self, key):
"""Retrieve a value from Redis."""
val = self.client.get(key)
if val:
return pickle.loads(val)
return None
redis_client = PythonNativeRedisClient()
Usage:
>>> from some_module import redis_client
>>> to_store = {'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
>>> redis_client.set('TestKey', to_store)
True
>>> retrieve = redis_client.get('TestKey')
{'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
Depending on needs, we have 3 ways to store data into Redis
String
- All data are stored and read back as bytes.
- This is the native way method that used by Redis.
- We could use
redis-cli to inspect and manipulate the data
- However, data may not be stored efficiently. For example
12345 is stored as string using at least 5 bytes even it could be represented by a 2-byte unsigned short.
- You could limited to use various data structures offered by Redis.
- The data stored within Redis is portable across different versions of Python and programming languages.
Pickle
- All Python data are converted into binary data using
pickle library before stored into Redis.
- Data must be converted back to Python data types using
pickle library before could be used.
- It could model complex and dynamic data structures not offered by Redis.
- Once the data is converted back to Python data types, We could use powerful Python to manipulate the data.
- It uses more memory within Redis as
pickle includes meta data, references in its output.
- It is not possible to use
redis-cli to manipulate the data, such as INCR.
- The data stored within Redis is not portable across different versions of Python and programming languages.
Struct
- All Python data are converted into binary data using
struct library before stored into Redis.
- Data must be converted back to Python data types using
struct library before could be used.
- It could model complex and static data structures not offered by Redis.
- Once the data is converted back to Python data types, We could use powerful Python to manipulate the data.
- It uses the least memory out of these 3 methods
- It is not possible to use
redis-cli to manipulate the data, such as INCR.
- The data stored within Redis is portable across different versions of Python and programming languages.
from redis import Redis
import sys
import pickle
import struct
redis = Redis()
# Data
d = [1234, 3.1415, b'Good', True]
k = ['int', 'float', 'byte', 'bool']
## Using String
print(f'Stringn{"_"*50}')
# Need to convert True to string or int or float
d2 = [1234, 3.1415, b'Good', int(True)]
redis.hset('k0', mapping=dict(zip(k,d2)))
print(redis.hmget('k0', *k))
print()
## Using Pickle
print(f'Picklen{"_"*50}')
packed = pickle.dumps(d)
print(f'Pickled data occupies {sys.getsizeof(packed)} bytes')
print(packed)
redis.set('k1', packed)
print(f'{pickle.loads(redis.get("k1"))}')
print()
## Using Struct
print(f'Structn{"_"*50}')
s = struct.Struct('H f 10s ?')
print(f'The structure occupies {s.size} bytes.')
packed = s.pack(*d)
print(f'{packed}')
redis.set('k2', s.pack(*d))
print(f'{s.unpack(redis.get("k2"))}')
Expected output:
String
__________________________________________________
[b'1234', b'3.1415', b'Good', b'1']
Pickle
__________________________________________________
Pickled data occupies 69 bytes
b'x80x04x95x19x00x00x00x00x00x00x00]x94(Mxd2x04G@t!xcaxc0x83x12oCx04Goodx94x88e.'
[1234, 3.1415, b'Good', True]
Struct
__________________________________________________
The structure occupies 19 bytes.
b'xd2x04x00x00Vx0eI@Goodx00x00x00x00x00x00x01'
(1234, 3.1414999961853027, b'Goodx00x00x00x00x00x00', True)
one solution could be to use a json.loads() to return int or float as a correct type, instead of str.
import json
import redis
redis = redis.from_url("redis://************")
def get(key, decode=True):
"""set decode to False when value stored as a string"""
value = redis.get(key)
if not decode:
return value
if value is not None:
try:
return json.loads(value)
except json.decoder.JSONDecodeError:
# not containing a JSON document
return value
redis.set("foo", 2)
redis.set("bar", 2.75)
get("foo")
# 2
get("bar")
# 2.75
I’m using redis in my python application to store simple values like counters and time stamp lists, but trying to get a counter and comparing it with a number I came across a problem.
If I do:
import redis
...
myserver = redis.Redis("localhost")
myserver.set('counter', 5)
and then try to get that value like this:
if myserver.get('counter') < 10:
myserver.incr('counter')
then I get a type error in the if statement because I’m comparing ‘5’ < 10, which means I’m storing an integer value and getting a string one (which can be considered as a different value).
My question is: is this supposed to work like that? I mean its a very basic type, I understand if I have to parse objects but an int? Seems that I’m doing something wrong.
Is there any configuration I’m missing?
Is there any way to make redis return the right type and not always a string?
I say this because its the same for lists and datetimes or even floating point values.
Could this be a problem with the redis-py client I’m using and not redis itself?
Technically speaking you need to take care of that on your own.
However, have a look at this link, especially at the part of their README that refers to parsers and response callbacks, maybe that’s something you can use. Question would be whether this is an overkill for you or not.
Looks like that is how redis stores its data:
redis 127.0.0.1:6379> set counter 5
OK
redis 127.0.0.1:6379> type counter
string
redis 127.0.0.1:6379> incr counter
(integer) 6
redis 127.0.0.1:6379> type counter
string
If you really want to, you could probably monkeypatch the redis-py client to infer data types.
As @favoretti said, response callbacks will do the trick. It’s not complicate at all, just one line and all will be taken care of.
In [2]: import redis
In [3]: r = redis.Redis()
In [10]: r.set_response_callback('HGET', float)
In [11]: r.hget('myhash', 'field0')
Out[11]: 4.6
for hmget, it returns a list of strings, not one single string, so you need to construct a little more comprehensive callback function:
In [12]: r.set_response_callback('HMGET', lambda l: [float(i) for i in l])
In [13]: r.hmget('myhash', 'field0')
Out[13]: [4.6]
same for hgetall.
Here is my test. Two redis connections: one returns int type, the other float
import redis
age_field = redis.Redis()
age_field.set_response_callback('HGET', int)
age_field.hget('foo', 'age')
# OUT: 40
a =age_field.hget('foo', 'age')
type(a)
# OUT: <type 'int'>
gpa_field = redis.Redis()
gpa_field.set_response_callback('HGET', float)
gpa_field.hget('foo', 'gpa')
# OUT: 2.5
b = gpa_field.hget('foo', 'gpa')
type(b)
# OUT: <type 'float'>
You can set decode_respone like True
redis.StrictRedis(host="localhost", port=6379, db=0, decode_responses=True)
Though leveraging set_response_callback is fine for simple data types, if you’re wondering about the fastest-and-easiest route to storing things like dictionaries, list, tuples – and preserving the python-native datatypes they may-or-may-not contain – I recommend using python’s built-in pickle library:
# Imports and simplified client setup
>>> import pickle
>>> import redis
>>> client = redis.Redis()
# Store a dictionary
>>> to_store = {'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
>>> client.set('TestKey', pickle.dumps(to_store))
True
# Retrieve the dictionary you just stored.
>>> retrieved = pickle.loads(client.get('TestKey'))
{'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
Here’s a simple client that will reduce the pickle boilerplate in the above example and provide you with a clean interface for storing and retrieving native python datatypes to and from Redis:
"""Redis cache."""
import pickle
import redis
redis_host = redis.Redis()
class PythonNativeRedisClient(object):
"""A simple redis client for storing and retrieving native python datatypes."""
def __init__(self, redis_host=redis_host):
"""Initialize client."""
self.client = redis_host
def set(self, key, value, **kwargs):
"""Store a value in Redis."""
return self.client.set(key, pickle.dumps(value), **kwargs)
def get(self, key):
"""Retrieve a value from Redis."""
val = self.client.get(key)
if val:
return pickle.loads(val)
return None
redis_client = PythonNativeRedisClient()
Usage:
>>> from some_module import redis_client
>>> to_store = {'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
>>> redis_client.set('TestKey', to_store)
True
>>> retrieve = redis_client.get('TestKey')
{'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
Depending on needs, we have 3 ways to store data into Redis
String
- All data are stored and read back as bytes.
- This is the native way method that used by Redis.
- We could use
redis-clito inspect and manipulate the data - However, data may not be stored efficiently. For example
12345is stored as string using at least 5 bytes even it could be represented by a 2-byte unsigned short. - You could limited to use various data structures offered by Redis.
- The data stored within Redis is portable across different versions of Python and programming languages.
Pickle
- All Python data are converted into binary data using
picklelibrary before stored into Redis. - Data must be converted back to Python data types using
picklelibrary before could be used. - It could model complex and dynamic data structures not offered by Redis.
- Once the data is converted back to Python data types, We could use powerful Python to manipulate the data.
- It uses more memory within Redis as
pickleincludes meta data, references in its output. - It is not possible to use
redis-clito manipulate the data, such asINCR. - The data stored within Redis is not portable across different versions of Python and programming languages.
Struct
- All Python data are converted into binary data using
structlibrary before stored into Redis. - Data must be converted back to Python data types using
structlibrary before could be used. - It could model complex and static data structures not offered by Redis.
- Once the data is converted back to Python data types, We could use powerful Python to manipulate the data.
- It uses the least memory out of these 3 methods
- It is not possible to use
redis-clito manipulate the data, such asINCR. - The data stored within Redis is portable across different versions of Python and programming languages.
from redis import Redis
import sys
import pickle
import struct
redis = Redis()
# Data
d = [1234, 3.1415, b'Good', True]
k = ['int', 'float', 'byte', 'bool']
## Using String
print(f'Stringn{"_"*50}')
# Need to convert True to string or int or float
d2 = [1234, 3.1415, b'Good', int(True)]
redis.hset('k0', mapping=dict(zip(k,d2)))
print(redis.hmget('k0', *k))
print()
## Using Pickle
print(f'Picklen{"_"*50}')
packed = pickle.dumps(d)
print(f'Pickled data occupies {sys.getsizeof(packed)} bytes')
print(packed)
redis.set('k1', packed)
print(f'{pickle.loads(redis.get("k1"))}')
print()
## Using Struct
print(f'Structn{"_"*50}')
s = struct.Struct('H f 10s ?')
print(f'The structure occupies {s.size} bytes.')
packed = s.pack(*d)
print(f'{packed}')
redis.set('k2', s.pack(*d))
print(f'{s.unpack(redis.get("k2"))}')
Expected output:
String
__________________________________________________
[b'1234', b'3.1415', b'Good', b'1']
Pickle
__________________________________________________
Pickled data occupies 69 bytes
b'x80x04x95x19x00x00x00x00x00x00x00]x94(Mxd2x04G@t!xcaxc0x83x12oCx04Goodx94x88e.'
[1234, 3.1415, b'Good', True]
Struct
__________________________________________________
The structure occupies 19 bytes.
b'xd2x04x00x00Vx0eI@Goodx00x00x00x00x00x00x01'
(1234, 3.1414999961853027, b'Goodx00x00x00x00x00x00', True)
one solution could be to use a json.loads() to return int or float as a correct type, instead of str.
import json
import redis
redis = redis.from_url("redis://************")
def get(key, decode=True):
"""set decode to False when value stored as a string"""
value = redis.get(key)
if not decode:
return value
if value is not None:
try:
return json.loads(value)
except json.decoder.JSONDecodeError:
# not containing a JSON document
return value
redis.set("foo", 2)
redis.set("bar", 2.75)
get("foo")
# 2
get("bar")
# 2.75