What's the best way to send an object over a network in Python?
Question:
I need to send objects around a network. I’m going to be using Twisted, and I’ve just started looking around the documentation for it.
As far as I know, the only way python implements sockets is through text. So how would I send an object using strings? Pickle? Or is there something better?
Answers:
The most general serialization on offer between Python end-points is the pickle format (in Python 2.any, be sure to use the cPickle module, and the -1 aka pickle.HIGHEST_PROTOCOL protocol; if you need interoperability between Python 2.any and Python 3.any more care is needed). For especially simple objects, the marshal module can sometimes be faster and more compact. For interoperation with non-Python endpoints, json may be best (or you could use xml to define or adopt other existing serialization formats), but those will likely be bulkier and slower to format and parse.
As far as I know, the only way python
implements sockets is through text.
Nope, all strings of bytes are welcome!-) You may be confused by the fact that in Python 2 a “normal string” is actually a string of bytes (“text” would be the unicode type); Python 3 sets things right and uses Unicode for “normal strings” and a specific byte string type for strings of bytes.
Strings of bytes are the general way in which any language will perform any form of serialization and deserialization, according to some protocol or other — such byte streams or blobs can go into networks, databases, plain files, etc, etc, of course.
Twisted offers its own serialization format, as part twisted.spread — it’s mostly for use with Perspective Broker (PB) but you could repurpose it for your own purposes if you don’t want to use PB for some special reason. The docs for the serialization part, twisted.spread.jelly, are here, and the summarize well the format’s goals…:
S-expression-based persistence of
python objects.
It does something very much like
Pickle; however, pickle’s main goal
seems to be efficiency (both in space
and time); jelly’s main goals are
security, human readability, and
portability to other environments.
If you care more about security, readability, and portability, than speed and compactness, then jelly might indeed serve you well.
Check out tlspyo
It runs Twisted in an isolated process so that the user doesn’t need to deal with the annoying reactor, defaults to TLS encryption (for security, especially when you use pickle as your serializer), and makes transferring python objects between machines over network very easy.
I need to send objects around a network. I’m going to be using Twisted, and I’ve just started looking around the documentation for it.
As far as I know, the only way python implements sockets is through text. So how would I send an object using strings? Pickle? Or is there something better?
The most general serialization on offer between Python end-points is the pickle format (in Python 2.any, be sure to use the cPickle module, and the -1 aka pickle.HIGHEST_PROTOCOL protocol; if you need interoperability between Python 2.any and Python 3.any more care is needed). For especially simple objects, the marshal module can sometimes be faster and more compact. For interoperation with non-Python endpoints, json may be best (or you could use xml to define or adopt other existing serialization formats), but those will likely be bulkier and slower to format and parse.
As far as I know, the only way python
implements sockets is through text.
Nope, all strings of bytes are welcome!-) You may be confused by the fact that in Python 2 a “normal string” is actually a string of bytes (“text” would be the unicode type); Python 3 sets things right and uses Unicode for “normal strings” and a specific byte string type for strings of bytes.
Strings of bytes are the general way in which any language will perform any form of serialization and deserialization, according to some protocol or other — such byte streams or blobs can go into networks, databases, plain files, etc, etc, of course.
Twisted offers its own serialization format, as part twisted.spread — it’s mostly for use with Perspective Broker (PB) but you could repurpose it for your own purposes if you don’t want to use PB for some special reason. The docs for the serialization part, twisted.spread.jelly, are here, and the summarize well the format’s goals…:
S-expression-based persistence of
python objects.It does something very much like
Pickle; however, pickle’s main goal
seems to be efficiency (both in space
and time); jelly’s main goals are
security, human readability, and
portability to other environments.
If you care more about security, readability, and portability, than speed and compactness, then jelly might indeed serve you well.
Check out tlspyo
It runs Twisted in an isolated process so that the user doesn’t need to deal with the annoying reactor, defaults to TLS encryption (for security, especially when you use pickle as your serializer), and makes transferring python objects between machines over network very easy.