Processing Data from an MQTT stream using Paho-MQTT

Question

I have a microcontroller that is streaming data to a MQTT broker and also a python script using the Paho-MQTT package to subscribe to the topic that my microcontroller is publishing to. In all of the examples that I see with Paho-MQTT the script is constantly looping the client as seen below:

def subscribe(client: mqtt_client):
    def on_message(client, userdata, msg):
        messageQueue = msg.payload.decode()
        print(messageQueue)

    client.subscribe(topic)
    client.on_message = on_message


def run():
    client = connect_mqtt()
    subscribe(client)
    client.loop_forever()

run()

The problem that I’m running into is that I need to actually process the data that is being sent and since loop_forever is a blocking process, I can’t ever break through to do anything with the data. I’ve looked into multi-threading since it seemed like that would allow me to run both the MQTT client and the processing simultaneously, but it seems like there is no way to transfer variables between threads. Is there something that I’m doing wrong, or how should I manage this?

Thanks in Advance!

I’ve looked into multi-threading since it seemed like that would allow me to run both the MQTT client and the processing simultaneously, but it seems like there is no way to transfer variables between threads. I’ve also tried stopping the loop to process and then restarting the loop but that also hasn’t worked in that it never gets to the parts of my code where I process the data, it just forever is listening.

Asked By: Xofox

||

Source

Answer 1

You may want to take a look at the "Network loop" section of the documentation, which shows you that calling client.loop_forever() isn’t your only option.

The problem that I’m running into is that I need to actually process the data that is being sent and since loop_forever is a blocking process, I can’t ever break through to do anything with the data.

You can process the data in your on_message function; that code gets called for every message received. Instead of simply calling print(messageQueue), you can perform whatever processing is necessary:

def subscribe(client: mqtt_client):
    def on_message(client, userdata, msg):
        messageQueue = msg.payload.decode()
        print("PROCESS THE MESSAGE HERE")

    client.subscribe(topic)
    client.on_message = on_message

but it seems like there is no way to transfer variables between threads

You don’t transfer variables between threads; you transfer data between threads. If you use the loop_start method to start the client loop in a background thread, you can use a Queue to pass messages from the client thread to your main thread for processing. That might look like:

from queue import Queue
import paho.mqtt.client as mqtt  # import the client1


def run_client(q):
    def on_message(client, userdata, msg):
        msg = msg.payload.decode()
        q.put(msg)

    client = mqtt.Client("example")  # create new instance
    client.connect("test.mosquitto.org")  # connect to broker
    client.subscribe("TELEMTRY")
    client.on_message = on_message
    client.loop_start()

def run():
    q = Queue()
    run_client(q)

    while True:
        msg = q.get()

        # Process message here
        print("processing:", msg)

if __name__ == "__main__":
    run()

Answered By: larsks

Processing Data from an MQTT stream using Paho-MQTT

Question:

Answers: