Numpy Array to base64 and back to Numpy Array – Python

Question

I am now trying to figure out how I can recover a numpy array from base64 data. This question and answer suggest it is possible: Reading numpy arrays outside of Python but an example is not given.

Using the code below as an example, how can I get a Numpy array from the base64 data if I know the dtype and the shape of the array?

import base64
import numpy as np

t = np.arange(25, dtype=np.float64)
s = base64.b64encode(t)
r = base64.decodestring(s)
q = ?????

I want a python statement to set q as a numpy array of dtype float64 so the result is an array identical to t. This is what the arrays encoded and decoded look like:

>>> t = np.arange(25,dtype=np.float64)
>>> t
array([  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
    11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,
    22.,  23.,  24.])
>>> s=base64.b64encode(t)
>>> s
'AAAAAAAAAAAAAAAAAADwPwAAAAAAAABAAAAAAAAACEAAAAAAAAAQQAAAAAAAABRAAAAAAAAAGEAAAAAAAAAcQAAAAAAAACBAAAAAAAAAIkAAAAAAAAAkQAAAAAAAACZAAAAAAAAAKEAAAAAAAAAqQAAAAAAAACxAAAAAAAAALkAAAAAAAAAwQAAAAAAAADFAAAAAAAAAMkAAAAAAAAAzQAAAAAAAADRAAAAAAAAANUAAAAAAAAA2QAAAAAAAADdAAAAAAAAAOEA='
>>> r = base64.decodestring(s)
>>> r
'x00x00x00x00x00x00x00x00x00x00x00x00x00x00xf0?x00x00x00x00x00x00x00@x00x00x00x00x00x00x08@x00x00x00x00x00x00x10@x00x00x00x00x00x00x14@x00x00x00x00x00x00x18@x00x00x00x00x00x00x1c@x00x00x00x00x00x00 @x00x00x00x00x00x00"@x00x00x00x00x00x00$@x00x00x00x00x00x00&@x00x00x00x00x00x00(@x00x00x00x00x00x00*@x00x00x00x00x00x00,@x00x00x00x00x00x00.@x00x00x00x00x00x000@x00x00x00x00x00x001@x00x00x00x00x00x002@x00x00x00x00x00x003@x00x00x00x00x00x004@x00x00x00x00x00x005@x00x00x00x00x00x006@x00x00x00x00x00x007@x00x00x00x00x00x008@'
>>> q = np.array( ????

The reason I am asking is because I am working on a project where I would like to store a lot of Numpy arrays in a MySQL database in an app powered by django.

Using this django snippet I can store base64 data in a textfield: http://djangosnippets.org/snippets/1669/

I want to write the arrays to the database as base64 instead of converting the arrays to a string of unicode.

Asked By: sequoia

||

Source

Answer 1

import base64
import numpy as np

t = np.arange(25, dtype=np.float64)
s = base64.b64encode(t)
r = base64.decodebytes(s)
q = np.frombuffer(r, dtype=np.float64)

print(np.allclose(q, t))
# True

Answered By: unutbu

Answer 2

The code below will encode it as base64. It will handle numpy arrays of any type/size without needing to remember what it was. It will also handle other arbitrary objects that can be pickled.

import numpy as np
import pickle
import codecs

obj = np.random.normal(size=(10, 10))
obj_base64string = codecs.encode(pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL), "base64").decode('latin1')
obj_reconstituted = pickle.loads(codecs.decode(obj_base64string.encode('latin1'), "base64"))

You can remove .decode(‘latin1’) and .encode(‘latin1’) if you just want the raw bytes.

Answered By: VoteCoffee

Numpy Array to base64 and back to Numpy Array – Python

Question:

Answers: