Best practice for communication between different modules in python
Question:
I revisited the development of an application in python I wrote 5 years ago (It was my first interesting project/application in python 2.7, and as I revisit it now I am struggling to make heads or tails). The application had the following units/modules:
- got data from a camera (Data Aquisition module – DAQ)
- processed the data and determined whether motor should be moved (Control)
- Moved an actuator (Actuator)
- displayed the data and activated switches on several tk windows. (Display)
Back then (in python 2.7) I developed separate modules for each unit mentioned above. Different threads were spawned for each unit I used a Queue to pass the image data, and the control commands between the Control and the Actuator module.
As the options (and the buzzwords) for Interprocess Communication have multiplied, I was hoping to get an idea of which concepts should I look into. e.g. in 3.10 there is the asyncio with concepts like Coroutines and Tasks, Synchronization Primitives, Subprocesses, Queues.
My main goal is to be able to write separate units that run independently, so that (in theory) it is easier to debug the unit (or even write unit tests).
UPDATE:
As it was mentioned in the comment, what I am describing is communication between threads of the same process, therefore the IPC tag might not be appropriate. I will elaborate below why I chose the IPC tag.
Although I previously developed the software with a single process and communication between the different threads, I recall that this was not optimal because of GIL. GIL imposed sleeping on the Data Acquisition thread at random intervals which made the data collection random at high sampling rates (the application could work adequately because of the low requirements).
Ideally, I would like to investigate a scheme where I could have a separate process collecting data. To my understanding different processes run on different processors and therefore should not be affected by GIL.
Answers:
Interprocess Communication specifically refers to multiple separate processes communicating. It sounds like you have multiple modules communicating within a single process. (asyncio and threads are intra-process, they do not span processes.) You can use queues if you need for that.
If you need something to be a separate process (e.g. to sidestep the GIL), you can use the multiprocessing module and its objects and primitives for communication. You should howver be aware that there’s somewhat significant overhead in serialization there that you wouldn’t have in a single process.
I revisited the development of an application in python I wrote 5 years ago (It was my first interesting project/application in python 2.7, and as I revisit it now I am struggling to make heads or tails). The application had the following units/modules:
- got data from a camera (Data Aquisition module – DAQ)
- processed the data and determined whether motor should be moved (Control)
- Moved an actuator (Actuator)
- displayed the data and activated switches on several tk windows. (Display)
Back then (in python 2.7) I developed separate modules for each unit mentioned above. Different threads were spawned for each unit I used a Queue to pass the image data, and the control commands between the Control and the Actuator module.
As the options (and the buzzwords) for Interprocess Communication have multiplied, I was hoping to get an idea of which concepts should I look into. e.g. in 3.10 there is the asyncio with concepts like Coroutines and Tasks, Synchronization Primitives, Subprocesses, Queues.
My main goal is to be able to write separate units that run independently, so that (in theory) it is easier to debug the unit (or even write unit tests).
UPDATE:
As it was mentioned in the comment, what I am describing is communication between threads of the same process, therefore the IPC tag might not be appropriate. I will elaborate below why I chose the IPC tag.
Although I previously developed the software with a single process and communication between the different threads, I recall that this was not optimal because of GIL. GIL imposed sleeping on the Data Acquisition thread at random intervals which made the data collection random at high sampling rates (the application could work adequately because of the low requirements).
Ideally, I would like to investigate a scheme where I could have a separate process collecting data. To my understanding different processes run on different processors and therefore should not be affected by GIL.
Interprocess Communication specifically refers to multiple separate processes communicating. It sounds like you have multiple modules communicating within a single process. (asyncio and threads are intra-process, they do not span processes.) You can use queues if you need for that.
If you need something to be a separate process (e.g. to sidestep the GIL), you can use the multiprocessing module and its objects and primitives for communication. You should howver be aware that there’s somewhat significant overhead in serialization there that you wouldn’t have in a single process.