json.dumps() won't return… not even after minutes

Question

I have a LoRa gateway that runs Linux and it can handle Python apps. I’m using a Python file provided by the manufacturer. It mimics a simple node-RED app. Details can be found here.

Messages are received by the gateway in order, but the problem is that the json.dumps() method won’t want to return after the call. I figured this out with putting print() functions into the original code.

Here is the important part of the code. The point is that the onMessage() function gets called when there is an uplink from an endpoint device and the onMessage() calls the rbPayloadFormatters(). I have never seen the result of the print(self.packet) line. When I let the rbPayloadFormatters() function return with the newMsg dictionary, I was able to see its content, printed from the onMessage() function.

### formats the payload message from the endpoint
def rbPayloadFormatters(self, msg):
    msgObj = json.loads(msg)
    newMsg = {}
    msgHex = base64.b64decode(msgObj["data"])
    newMsg["payload"] = binascii.hexlify(msgHex)
    newMsg["time"] = msgObj["tmst"]
    newMsg["snr"] = msgObj["lsnr"]
    newMsg["station"] = msgObj["appeui"]
    newMsg["avgsnr"] = msgObj["lsnr"]
    newMsg["lat"] = 0
    newMsg["lng"] = 0
    newMsg["rssi"] = msgObj["rssi"]
    newMsg["seqnumber"] = msgObj["seqn"]
    newMsg["deveui"] = msgObj["deveui"]
    newMsg["authorisation"] = self.rbAuthorization
    return json.dumps(newMsg)

#callback function initiated by on_message
def onMessage(self, mqtt_client, userdata, msg):
    self.packet = self.rbPayloadFormatters(msg.payload)
    pkt = json.loads(self.packet)
    self.devEUI = pkt["deveui"]
    self.payloadData = pkt["payload"]
    
    print(self.packet)

I have read that the stdlib json.dumps() can be slow but after minutes of waiting I was not able to see any printed json object on the console.

If you have any idea what is wrong, please don’t hesitate to answer to this post. Thank you.

Asked By: u2dragon

||

Source

Answer 1

Supporting Michael Butscher and snakecharmerb‘s comments this does not seem to be a problem with the json library.

I did some small-scale testing and tried to dump a dictionary equaling about 300 megabytes of data, which took around 5.5 seconds on average.

Code:

import json
import time

start = time.time()
data = {i:0 for i in range(10_000_000)}

print("size of object (Bytes):", data.__sizeof__())

j = json.dumps(data)

print("time to dump: ", time.time() - start)

Out:

size of object (Bytes): 335544384
time to dump:  5.49969482421875

Note that this won’t be representative of more complex real-world data.

In my personal experience json.dumps has been slow at times, but not nearing minute-long wait times (at least for moderate amounts of data).

I would recommend having a look at Python profiling, which will provide a quick report of where and what your code is spending time doing.

Example from the docs:

import cProfile
import re
cProfile.run('re.compile("foo|bar")')

Out:

      214 function calls (207 primitive calls) in 0.002 seconds

Ordered by: cumulative time

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000    0.002    0.002 {built-in method builtins.exec}
     1    0.000    0.000    0.001    0.001 <string>:1(<module>)
     1    0.000    0.000    0.001    0.001 __init__.py:250(compile)
     1    0.000    0.000    0.001    0.001 __init__.py:289(_compile)
     1    0.000    0.000    0.000    0.000 _compiler.py:759(compile)
     1    0.000    0.000    0.000    0.000 _parser.py:937(parse)
     1    0.000    0.000    0.000    0.000 _compiler.py:598(_code)
     1    0.000    0.000    0.000    0.000 _parser.py:435(_parse_sub)

With the details you have provided it seems like loading the original message in rbPayloadFormatters with msgObj = json.loads(msg) doesn’t take a long time, as ‘json.loads’ is usually equally or more time intensive than json.dumps, which tells me that the issue lies elsewhere.

Hope this was of help, and good luck with further debugging!

Answered By: duggurd

Answer 2

Today I continued the debugging and first tried Michael Butscher’s advice and started to comment out lines which add keys to newMsg.

Turned out that the json.dumps() returns and the rbPayloadFormatters() returns as well if I comment out the next two lines.

msgHex = base64.b64decode(msgObj["data"])
newMsg["payload"] = binascii.hexlify(msgHex)

The explanation for the never returning json.dumps() is the data type of the msgHex and newMsg["payload"]. Both of them has type bytes and json doesn’t support this type. (I think, I should have received an error… but I didn’t.)

To solve my problem I simply converted the msgHex to string.

newMsg["payload"] = str(binascii.hexlify(msgHex))

Thank you everyone for the contribution.

Answered By: u2dragon

json.dumps() won't return… not even after minutes

Question:

Answers: