Pyzmq – send message to STREAM socket
Question:
I am trying to implement simple example of connection between two STREAM sockets in pyzmq.
sender.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.bind("tcp://*:5555")
socket.connect("tcp://localhost:5556")
socket.send("message")
receiver.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.bind("tcp://*:5556")
message = socket.recv()
print("Received -> [ %s ]" % (message))
Output
Received [ b'x00kx8bEg' ]
Received [ b'' ]
I would like to ask what is the correct way to send message between STREAM sockets.
Answers:
Your socket.recv()-ed data are exactly matching the ZeroMQ specification, though they need not have made you happy and you doubt why you did get this, instead of nicely delivered exact copies of messages sent.
So, be patient and keep reading.
ZeroMQ recently added STREAM socket-archetype is rather specific
Anyone having a few years experience with ZeroMQ signalling/messaging tools will tell you that the recently ( v4.x ) added STREAM archetype is not the best choice for a ZeroMQ-process to a ZeroMQ-process intercommunication needs.
Why? Allmost all the gems, that the ZeroMQ tools have are and must be shortcut in STREAM, so as to allow a ZeroMQ socket access-point become able to “speak” to an opposite socket endpoint process, which knows nothing about ZeroMQ smart-sockets higher-level protocols.
Native pattern
The native pattern is used for communicating with TCP peers and allows asynchronous requests and replies in either direction.
ZMQ_STREAM
A socket of type ZMQ_STREAM is used to send and receive TCP data from a non-ØMQ peer, when using the tcp:// transport. A ZMQ_STREAM socket can act as client and/or server, sending and/or receiving TCP data asynchronously.
When receiving TCP data, a ZMQ_STREAM socket shall prepend a message part containing the identity of the originating peer to the message before passing it to the application. Messages received are fair-queued from among all connected peers.
When sending TCP data, a ZMQ_STREAM socket shall remove the first part of the message and use it to determine the identity of the peer the message shall be routed to, and unroutable messages shall cause an EHOSTUNREACH or EAGAIN error.
To open a connection to a server, use the zmq_connect() call, and then fetch the socket identity using the ZMQ_IDENTITY zmq_getsockopt() call.
To close a specific connection, send the identity frame followed by a zero-length message (see EXAMPLE section).
When a connection is made, a zero-length message will be received by the application. Similarly, when the peer disconnects (or the connection is lost), a zero-length message will be received by the application.
You must send one identity frame followed by one data frame. The ZMQ_SNDMORE flag is required for identity frames but is ignored on data frames.
EXAMPLE
void *ctx = zmq_ctx_new ();
assert ( ctx );
/* Create ZMQ_STREAM socket */
void *socket = zmq_socket ( ctx, ZMQ_STREAM );
assert ( socket );
int rc = zmq_bind ( socket, "tcp://*:8080" );
assert ( rc == 0 );
/* Data structure to hold the ZMQ_STREAM ID */
uint8_t id [256];
size_t id_size = 256;
/* Data structure to hold the ZMQ_STREAM received data */
uint8_t raw [256];
size_t raw_size = 256;
while ( 1 ) {
/* Get HTTP request; ID frame and then request */
id_size = zmq_recv ( socket, id, 256, 0 );
assert ( id_size > 0 );
do {
raw_size = zmq_recv ( socket, raw, 256, 0 );
assert ( raw_size >= 0 );
} while ( raw_size == 256 );
/* Prepares the response */
char http_response [] =
"HTTP/1.0 200 OKrn"
"Content-Type: text/plainrn"
"rn"
"Hello, World!";
/* Sends the ID frame followed by the response */
zmq_send ( socket, id, id_size, ZMQ_SNDMORE );
zmq_send ( socket, http_response, strlen ( http_response ), 0 );
/* Closes the connection by sending the ID frame followed by a zero response */
zmq_send ( socket, id, id_size, ZMQ_SNDMORE );
zmq_send ( socket, 0, 0, 0 );
}
zmq_close ( socket );
zmq_ctx_destroy ( ctx );
If you follow the description of the STREAM behaviour on the multi-connected socket cases, the sender side will happen to receive a fair-queue round-robin reads on the socket instance, which is connected ( 1x via the .connect() + Nx via .bind(), N = < 0, +INF ) ) to multiple endpoints, so far with zero control of either the count or/and the nature of the communicating peers, yet having a fair-queued round-robin mechanism on the socket.recv()-s. Definitely not a safe design practice.
Summary of ZMQ_STREAM characteristics
Compatible peer sockets none
Direction Bidirectional
Send/receive pattern Unrestricted
Outgoing routing strategy See text ( above )
Incoming routing strategy Fair-queued
Action in mute state EAGAIN
Here is the simplified example using pyzmq with one way connection as in the question.
sender.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.connect('tcp://localhost:5555')
id_sock = socket.getsockopt(zmq.IDENTITY)
socket.send(id_sock, zmq.SNDMORE)
socket.send(b'message')
receiver.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.bind('tcp://*:5555')
id_sock = socket.recv()
assert not socket.recv() # empty data here
assert socket.recv() == id_sock
message = socket.recv()
print('received:', message)
I am trying to implement simple example of connection between two STREAM sockets in pyzmq.
sender.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.bind("tcp://*:5555")
socket.connect("tcp://localhost:5556")
socket.send("message")
receiver.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.bind("tcp://*:5556")
message = socket.recv()
print("Received -> [ %s ]" % (message))
Output
Received [ b'x00kx8bEg' ]
Received [ b'' ]
I would like to ask what is the correct way to send message between STREAM sockets.
Your socket.recv()-ed data are exactly matching the ZeroMQ specification, though they need not have made you happy and you doubt why you did get this, instead of nicely delivered exact copies of messages sent.
So, be patient and keep reading.
ZeroMQ recently added STREAM socket-archetype is rather specific
Anyone having a few years experience with ZeroMQ signalling/messaging tools will tell you that the recently ( v4.x ) added STREAM archetype is not the best choice for a ZeroMQ-process to a ZeroMQ-process intercommunication needs.
Why? Allmost all the gems, that the ZeroMQ tools have are and must be shortcut in STREAM, so as to allow a ZeroMQ socket access-point become able to “speak” to an opposite socket endpoint process, which knows nothing about ZeroMQ smart-sockets higher-level protocols.
Native pattern
The native pattern is used for communicating with TCP peers and allows asynchronous requests and replies in either direction.
ZMQ_STREAMA socket of type
ZMQ_STREAMis used to send and receive TCP data from a non-ØMQ peer, when using thetcp://transport. AZMQ_STREAMsocket can act as client and/or server, sending and/or receiving TCP data asynchronously.When receiving TCP data, a
ZMQ_STREAMsocket shall prepend a message part containing the identity of the originating peer to the message before passing it to the application. Messages received are fair-queued from among all connected peers.When sending TCP data, a
ZMQ_STREAMsocket shall remove the first part of the message and use it to determine the identity of the peer the message shall be routed to, and unroutable messages shall cause anEHOSTUNREACHorEAGAINerror.To open a connection to a server, use the
zmq_connect()call, and then fetch the socket identity using theZMQ_IDENTITYzmq_getsockopt()call.To close a specific connection, send the identity frame followed by a zero-length message (see EXAMPLE section).
When a connection is made, a zero-length message will be received by the application. Similarly, when the peer disconnects (or the connection is lost), a zero-length message will be received by the application.
You must send one identity frame followed by one data frame. The
ZMQ_SNDMOREflag is required for identity frames but is ignored on data frames.EXAMPLE
void *ctx = zmq_ctx_new ();
assert ( ctx );
/* Create ZMQ_STREAM socket */
void *socket = zmq_socket ( ctx, ZMQ_STREAM );
assert ( socket );
int rc = zmq_bind ( socket, "tcp://*:8080" );
assert ( rc == 0 );
/* Data structure to hold the ZMQ_STREAM ID */
uint8_t id [256];
size_t id_size = 256;
/* Data structure to hold the ZMQ_STREAM received data */
uint8_t raw [256];
size_t raw_size = 256;
while ( 1 ) {
/* Get HTTP request; ID frame and then request */
id_size = zmq_recv ( socket, id, 256, 0 );
assert ( id_size > 0 );
do {
raw_size = zmq_recv ( socket, raw, 256, 0 );
assert ( raw_size >= 0 );
} while ( raw_size == 256 );
/* Prepares the response */
char http_response [] =
"HTTP/1.0 200 OKrn"
"Content-Type: text/plainrn"
"rn"
"Hello, World!";
/* Sends the ID frame followed by the response */
zmq_send ( socket, id, id_size, ZMQ_SNDMORE );
zmq_send ( socket, http_response, strlen ( http_response ), 0 );
/* Closes the connection by sending the ID frame followed by a zero response */
zmq_send ( socket, id, id_size, ZMQ_SNDMORE );
zmq_send ( socket, 0, 0, 0 );
}
zmq_close ( socket );
zmq_ctx_destroy ( ctx );
If you follow the description of the STREAM behaviour on the multi-connected socket cases, the sender side will happen to receive a fair-queue round-robin reads on the socket instance, which is connected ( 1x via the .connect() + Nx via .bind(), N = < 0, +INF ) ) to multiple endpoints, so far with zero control of either the count or/and the nature of the communicating peers, yet having a fair-queued round-robin mechanism on the socket.recv()-s. Definitely not a safe design practice.
Summary of ZMQ_STREAM characteristics
Compatible peer sockets none
Direction Bidirectional
Send/receive pattern Unrestricted
Outgoing routing strategy See text ( above )
Incoming routing strategy Fair-queued
Action in mute state EAGAIN
Here is the simplified example using pyzmq with one way connection as in the question.
sender.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.connect('tcp://localhost:5555')
id_sock = socket.getsockopt(zmq.IDENTITY)
socket.send(id_sock, zmq.SNDMORE)
socket.send(b'message')
receiver.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.STREAM)
socket.bind('tcp://*:5555')
id_sock = socket.recv()
assert not socket.recv() # empty data here
assert socket.recv() == id_sock
message = socket.recv()
print('received:', message)