PySide6 – high datarate from DLL to QTableWidget

Question:

I have a python (3.11) project with PySide6. Application connects to DLL for CAN communication with other nodes. As part of configuration, approx 50-70 CAN messages are sent in short timeframe (but I added delays up to 50ms per message) and still see data loss.

A DLL callback function properly receives all packets from low-level interface driver (done in C) and if I try to directly call UI elements, I am receiving access denied issues (which is clear as UI seems to be in another thread).

Therefore I have a signal, that is emitted on every DLL message, and transfers data to UI.
If, in the UI signal emitted handler, I only print same packet with print() function, data is nicely displayed in correct order.

    # Signal in a class
    signal_dll_packet_received = Signal(object)

    # Main function called from DLL to process various events
    def dll_event_function(self, inpacket):
        packet = inpacket.contents
        print('DLL PACKET:', packet.data.can_msg)
        self.signal_dll_packet_received.emit(packet)

    def signal_dll_packet_received_emitted(self, packet):
        self.new_ui_window.process_packet(packet)

But since I have a QTableWidget, I have to perform table search for record and update the value for specific cell. When doing so, I am observing:

  • Some packets are lost in the UI – looks like because UI is too slow and cannot process DLL messages fast enough.
  • Some packets seem to be "overwritten", meaning that even if 2 different packets arrive through DLL, UI seems both as the same (looks like same pointer memory).

What is the Qt’s correct way to transfer "high throughput" data from DLL to the UI and not have data loss.

I have tried things like:

  • Make a deepcopy of the packet and continue with a copy
  • Test-wise remove tablemodel.beginResetModel() and endResetModel() functions, that may have an impact on performance.

Should I be implementing some kind of ring buffer or is there any recommended Qt’s way?

New UI window class method

    # method in window class for table manipulation
    def process_packet(self, packet):
        print('UI PACKET:', packet.data.can_msg)
        
        # DO WORK with data in QTableWidget
        # If this is ignored, then it works properly

Here you see that DLL packet log printed several messages with different data,
but UI packet (that went through emit + table modification), in a long chunk of data, only sees data format from last DLL packet -> feels like memory overwrite.

At the beginning, sequence is correct -> DLL first, UI follows, DLL again, UI follows. later it goes to several DLL first, then several UIs, but with wrong data..

DLL PACKET: sec=36913; used=996589; len =  8; data = 01 00 52 41 54 43 4B 4C
UI PACKET: sec=36913; used=996589; len =  8; data = 01 00 52 41 54 43 4B 4C
DLL PACKET: sec=36914; used=027252; len =  8; data = 01 00 45 30 30 31 00 00
UI PACKET: sec=36914; used=027252; len =  8; data = 01 00 45 30 30 31 00 00
DLL PACKET: sec=36914; used=057970; len =  4; data = 02 00 07 00
DLL PACKET: sec=36914; used=087962; len =  4; data = 02 00 02 00
DLL PACKET: sec=36914; used=118081; len =  6; data = 02 00 02 00 03 00
DLL PACKET: sec=36914; used=148968; len =  4; data = 02 00 00 00
DLL PACKET: sec=36914; used=179087; len =  6; data = 02 00 3C 00 14 00
DLL PACKET: sec=36914; used=210279; len =  8; data = 02 00 00 00 00 00 00 00
DLL PACKET: sec=36914; used=240934; len =  4; data = 02 00 00 00
DLL PACKET: sec=36914; used=271181; len =  8; data = 02 00 52 41 54 43 4B 4C
DLL PACKET: sec=36914; used=302253; len =  8; data = 02 00 45 30 30 32 00 00
DLL PACKET: sec=36914; used=332939; len =  4; data = 03 00 0B 00
DLL PACKET: sec=36914; used=362938; len =  4; data = 03 00 03 00
DLL PACKET: sec=36914; used=393082; len =  6; data = 03 00 02 00 03 00
DLL PACKET: sec=36914; used=423929; len =  4; data = 03 00 00 00
DLL PACKET: sec=36914; used=454072; len =  6; data = 03 00 3C 00 14 00
DLL PACKET: sec=36914; used=485224; len =  8; data = 03 00 00 00 00 00 00 00
DLL PACKET: sec=36914; used=516342; len =  4; data = 03 00 00 00
DLL PACKET: sec=36914; used=547190; len =  8; data = 03 00 52 41 54 43 4B 4C
DLL PACKET: sec=36914; used=578189; len =  8; data = 03 00 45 30 30 33 00 00
DLL PACKET: sec=36914; used=608964; len =  4; data = 04 00 17 00
DLL PACKET: sec=36914; used=638963; len =  4; data = 04 00 04 00
DLL PACKET: sec=36914; used=669074; len =  6; data = 04 00 02 00 03 00
DLL PACKET: sec=36914; used=699954; len =  4; data = 04 00 00 00
DLL PACKET: sec=36914; used=730065; len =  6; data = 04 00 14 00 00 00
DLL PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
!!! <- see here -> all UI packets have content from last DLL packet
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
UI PACKET: sec=36914; used=761640; len =  8; data = 04 00 00 00 00 00 00 00
DLL PACKET: sec=36914; used=791959; len =  4; data = 04 00 00 00
UI PACKET: sec=36914; used=791959; len =  4; data = 04 00 00 00
DLL PACKET: sec=36914; used=822167; len =  8; data = 04 00 52 41 54 43 4B 4C
UI PACKET: sec=36914; used=822167; len =  8; data = 04 00 52 41 54 43 4B 4C
DLL PACKET: sec=36914; used=853238; len =  8; data = 04 00 45 30 30 34 00 00
UI PACKET: sec=36914; used=853238; len =  8; data = 04 00 45 30 30 34 00 00

How to solve it?

Asked By: tilz0R

||
Source

Answers:

The correct way is to separate reading from the BUS from the GUI, a worker thread should be reading from the BUS and putting results in a queue.

The main thread will have a timed event (QTimer) to read the data in the queue every … 100 milliseconds and update the screen. (simply keep reading the queue until it’s empty),

you can emit an event every time a message arrives, but doing 70 updates to the GUI every second might be slow, so the timer will make the GUI more responsive.

This way you are certain no data is dropped as the worker will be woken by the OS every time a message arrives.

Running DLL functions drops the GIL, so the extra thread has minimal impact on your application.

Answered By: Ahmed AEK
Categories: questions Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.