Appending JSON in function loop

Question

I’m retrieving data from sensors using MQTT and I am using this to be able to work at what times the sensors were set off. I am then trying to write this a JSON.
Currently I am having issues where everytime it append the JSON with the new data but it will also add the same title.

The code I am using is python and current script to write to the JSON in within a function which gets looped indefinitely.

    def on_message(client, userdata, msg): 
           if sensone > 0.275 and sensone < 0.315 and c == 0:
                print("Timer Started")
                t0 = time.time()
                dt0 = datetime.datetime.now()
                dt0 = dt0.strftime('%Y-%m-%d %H:%M:%S')
                c += 1   
           if senstwo > 0.275 and senstwo < 0.315 and c == 1:
                t1 = time.time()
                dt1 = datetime.datetime.now()
                dt1 = dt1.strftime('%Y-%m-%d %H:%M:%S')
                with open('SensorTimes.json', 'a') as outfile:
                    datalog = {}
                    datalog['SensorTimes'] = []
                    datalog['SensorTimes'].append({
                        "Start Time":dt0, 
                        "End Time":dt1, 
                        "Total Time":round(t1-t0, 2)
                    })
                    json.dump(datalog, outfile, indent=4)
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("172.18.65.123", 1883)

client.loop_forever()

I have removed parts of the script are not relevant, I’ve put it all within the if statement because if it is not between the limits I do not want the data recorded.

Current Output:

{
    "SensorTimes": [
        {
            "Start Time": "2019-03-28 09:03:10",
            "End Time": "2019-03-28 09:03:12",
            "Total Time": 2.22
        }
    ]
}{
    "SensorTimes": [
        {
            "Start Time": "2019-03-28 09:03:38",
            "End Time": "2019-03-28 09:03:41",
            "Total Time": 3.02
        }
    ]
}

Expected Output:

    {
        "SensorTimes": [
            {
                "Start Time": "2019-03-28 09:03:10",
                "End Time": "2019-03-28 09:03:12",
                "Total Time": 2.22
            },
            {
                "Start Time": "2019-03-28 09:03:38",
                "End Time": "2019-03-28 09:03:41",
                "Total Time": 3.02
            }
        ]
    }

Any help on this is appreciated.

Asked By: JackU

||

Source

Answer 1

You are actually opening your file with the append tag ("a") so it’s appending to your file the content you write into it when you use json.dump()

To prevent that, first read your file with a open('SensorTimes.json', 'r').read() to get its content. Then translate it into a json. Then append to this json your new content. And write it to your file using open('SensorTimes.json', 'w').write(newJson) or json.dump(newJson, open('SensorTimes.json', 'w'))

You can also open your file directly to read and write in it using the "r+" tag

Answered By: Nenri

Answer 2

After trying many different combinations of things I got it to work!

I put the JSON append into a separate function which gets called in the if statement and this return to the original function.

datalog = {}
datalog['SensorTimes'] = []
def json_update(dt1, t1, dt0, t0):
    with open(r'SensorTimes.json', 'w') as outfile:
        datalog['SensorTimes'].append({
                    "Start Time":dt0, 
                    "End Time":dt1, 
                    "Total Time":round(t1-t0, 2)
                })   
        json.dump(datalog, outfile, indent=4)
    return

This gives the expected output and will work all the time within the indefinite function. This adds the new dictionary to the JSON and does not overwrite the existing dictionary/content of the JSON.

Answered By: JackU

Answer 3

A general comment/suggestion for people trying to append data in json files while looping… do not try difficult maneuvering with the json package.
Just go for the easiest approach and work with pythonic tools such as lists!
create list, initiate loop, append/extend at each iteration and then, when the loop is over, use the classic json.dump(list, file) to save the data in the desired file.
Hope this comment helped

Answered By: Andrea Bragantini

Appending JSON in function loop

Question:

Answers: