Saving dictionary of numpy arrays
Question:
So I have a DB with a couple of years worth of site data. I am now attempting to use that data for analytics – plotting and sorting of advertising costs by keyword, etc.
One of the data grabs from the DB takes minutes to complete. While I could spend some time optimizing the SQL statements I use to get the data I’d prefer to simply leave that class and it’s SQL alone, grab the data, and save the results to a data file for faster retrieval later. Most of this DB data isn’t going to change so I could write a separate python script to update the file every 24 hours and then use that file for this long running task.
The data is being returned as a dictionary of numpy arrays. When I use numpy.save('data', data) the file is saved just fine. When I use data2 = numpy.load('data.npy') it loads the file without error. However, the output data2 doesn’t not equal the original data.
Specifically the line data == data2 returns false. Additionally, if I use the following:
for key, key_data in data.items():
print key
it works. But when I replace data.items() with data2.items() then I get an error:
AttributeError: 'numpy.ndarray' object has no attribute 'items'
Using type(data) I get dict. Using type(data2) I get numpy.ndarray.
So how do I fix this? I want the loaded data to equal the data I passed in for saving. Is there an argument to numpy.save to fix this or do I need some form of simple reformatting function to reformat the loaded data into the proper structure?
Attempts to get into the ndarray via for loops or indexing all lead to errors about indexing a 0-d array. Casting like this dict(data2) also fails for iterating over a 0-d array. However, Spyder shows value of the array and it includes the data I saved. I just can’t figure out how to get to it.
If I need to reformat the loaded data I’d appreciate some example code on how to do this.
Answers:
Let’s look at a small example:
In [819]: N
Out[819]:
array([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])
In [820]: data={'N':N}
In [821]: np.save('temp.npy',data)
In [822]: data2=np.load('temp.npy')
In [823]: data2
Out[823]:
array({'N': array([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])}, dtype=object)
np.save is designed to save numpy arrays. data is a dictionary. So it wrapped it in a object array, and used pickle to save that object. Your data2 probably has the same character.
You get at the array with:
In [826]: data2[()]['N']
Out[826]:
array([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])
I really liked the deepdish (it saves them in HDF5 format):
>>> import deepdish as dd
>>> d = {'foo': np.arange(10), 'bar': np.ones((5, 4, 3))}
>>> dd.io.save('test.h5', d)
$ ddls test.h5
/bar array (5, 4, 3) [float64]
/foo array (10,) [int64]
>>> d = dd.io.load('test.h5')
for my experience, it seems to be partially broken for large datasets, though 🙁
When saving a dictionary with numpy, the dictionary is encoded into an array.
To have what you need, you can do as in this example:
my_dict = {'a' : np.array(range(3)), 'b': np.array(range(4))}
np.save('my_dict.npy', my_dict)
my_dict_back = np.load('my_dict.npy')
print(my_dict_back.item().keys())
print(my_dict_back.item().get('a'))
So you are probably missing .item() for the reloaded dictionary.
Check this out:
for key, key_d in data2.item().items():
print key, key_d
The comparison my_dict == my_dict_back.item() works only for dictionaries that does not have lists or arrays in their values.
EDIT: for the item() issue mentioned above, I think it is a better option to save dictionaries with the library pickle rather than with numpy.
SECOND EDIT: if not happy with pickle, and all the types in the dictionary are compatible with , json is an option as well.
So I have a DB with a couple of years worth of site data. I am now attempting to use that data for analytics – plotting and sorting of advertising costs by keyword, etc.
One of the data grabs from the DB takes minutes to complete. While I could spend some time optimizing the SQL statements I use to get the data I’d prefer to simply leave that class and it’s SQL alone, grab the data, and save the results to a data file for faster retrieval later. Most of this DB data isn’t going to change so I could write a separate python script to update the file every 24 hours and then use that file for this long running task.
The data is being returned as a dictionary of numpy arrays. When I use numpy.save('data', data) the file is saved just fine. When I use data2 = numpy.load('data.npy') it loads the file without error. However, the output data2 doesn’t not equal the original data.
Specifically the line data == data2 returns false. Additionally, if I use the following:
for key, key_data in data.items():
print key
it works. But when I replace data.items() with data2.items() then I get an error:
AttributeError: 'numpy.ndarray' object has no attribute 'items'
Using type(data) I get dict. Using type(data2) I get numpy.ndarray.
So how do I fix this? I want the loaded data to equal the data I passed in for saving. Is there an argument to numpy.save to fix this or do I need some form of simple reformatting function to reformat the loaded data into the proper structure?
Attempts to get into the ndarray via for loops or indexing all lead to errors about indexing a 0-d array. Casting like this dict(data2) also fails for iterating over a 0-d array. However, Spyder shows value of the array and it includes the data I saved. I just can’t figure out how to get to it.
If I need to reformat the loaded data I’d appreciate some example code on how to do this.
Let’s look at a small example:
In [819]: N
Out[819]:
array([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])
In [820]: data={'N':N}
In [821]: np.save('temp.npy',data)
In [822]: data2=np.load('temp.npy')
In [823]: data2
Out[823]:
array({'N': array([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])}, dtype=object)
np.save is designed to save numpy arrays. data is a dictionary. So it wrapped it in a object array, and used pickle to save that object. Your data2 probably has the same character.
You get at the array with:
In [826]: data2[()]['N']
Out[826]:
array([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])
I really liked the deepdish (it saves them in HDF5 format):
>>> import deepdish as dd
>>> d = {'foo': np.arange(10), 'bar': np.ones((5, 4, 3))}
>>> dd.io.save('test.h5', d)
$ ddls test.h5
/bar array (5, 4, 3) [float64]
/foo array (10,) [int64]
>>> d = dd.io.load('test.h5')
for my experience, it seems to be partially broken for large datasets, though 🙁
When saving a dictionary with numpy, the dictionary is encoded into an array.
To have what you need, you can do as in this example:
my_dict = {'a' : np.array(range(3)), 'b': np.array(range(4))}
np.save('my_dict.npy', my_dict)
my_dict_back = np.load('my_dict.npy')
print(my_dict_back.item().keys())
print(my_dict_back.item().get('a'))
So you are probably missing .item() for the reloaded dictionary.
Check this out:
for key, key_d in data2.item().items():
print key, key_d
The comparison my_dict == my_dict_back.item() works only for dictionaries that does not have lists or arrays in their values.
EDIT: for the item() issue mentioned above, I think it is a better option to save dictionaries with the library pickle rather than with numpy.
SECOND EDIT: if not happy with pickle, and all the types in the dictionary are compatible with , json is an option as well.