Serializing a function's source code in Python

Question

I am trying to serialize a code and send it as a json…

def f(x): return x*x
def fi(x): return int(x[0])

code_string = marshal.dumps(fi.func_code)

jsn = {"code":code_string)
json.dumps(jsn) # doesnt work if code_string is from fi

So… the above code block works if my function is f(x)

But fails for fi(x)

Original exception was:

Traceback (most recent call last):
  File "/home/mohitdee/Documents/python_scala/rdd.py", line 41, in <module>
    send_data(json.dumps(jsn))
  File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x83 in position 32: invalid start byte
[48001 refs]

How do i resolve this in python

Asked By: frazman

||

Source

Answer 1

You can serialize all kinds of live objects, including functions, using the cloud library implementation of pickle.

import cloud, pickle

def serialize(func):
    return cloud.serialization.cloudpickle.dumps(func)

def deserialize(string):
    return pickle.loads(string)

Answered By: slezica

Answer 2

Try encoding it with base64 or some other algorithm of this sort.

Answered By: Filip Malczak

Answer 3

Use pickle (or cPickle):

The pickle module implements a fundamental, but powerful algorithm for
serializing and de-serializing a Python object structure.

>>> import cPickle
>>> import json
>>> def fi(x):
...     return int(x[0])
... 
>>> fi(['1'])
1
>>> code_string = cPickle.dumps(fi)
>>> jsn = {"code": code_string}
>>> serialized = json.dumps(jsn)

>>> deserialized = json.loads(serialized)
>>> f = cPickle.loads(str(deserialized['code']))
>>> print f(['1'])
1

Answered By: alecxe

Answer 4

Marshall is a binary protocol, i.e. a bunch of bytes with very custom interpretation. It’s not text, it doesn’t conform to in any particular text encoding. It is, for the most part, just a sequence of bits. If you absolutely need to embed those in a text protocol like JSON, you need to escape the bytes that don’t make valid characters in the relevant encoding (to be safe, assume a subset of ASCII). The canonical solution is base64:

import base64

code_string = marshal.dumps(fi.func_code)
code_base64 = base64.b64encode(code_string)

jsn = {"code": code_base64}

Answered By: user395760

Serializing a function's source code in Python

Question:

Answers: