Python – Issue with creating a Thread subclass within a Process subclass – threading & multiprocessing

Question

I’m having trouble with trying to get a Thread (threading.Thread) subclass to work within a Process (multiprocessing.Process) subclass.

Here is the simplest working example to demonstrate the issue.
I make one "SubProcess" (instance of multiprocessing.Process), which will contain a child "WorkerThread" (instance of threading.Thread). Execution terminates at subProcess.start()

import multiprocessing
import threading

class SubProcess(multiprocessing.Process):
    def __init__(self):
        multiprocessing.Process.__init__(self, daemon=True)

        #Instantiate this one worker
        self.workerThread = WorkerThread()

    def run(self):
        #Start the worker
        self.workerThread.start()

class WorkerThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self, daemon=True)

    def run(self):
        #Real work code goes here
        pass


if __name__ == '__main__':
    #Program starts here
    
    #Instantiate the SubProcess class - works fine
    subProcess = SubProcess()

    #Start the subProcess - Execution stops here - see Traceback below
    subProcess.start()

    subProcess.join()

Here is the output traceback:

Traceback (most recent call last):
  File "[***]simplestExampleError.py", line 31, in <module>
    subProcess.start()
  File "C:Python39libmultiprocessingprocess.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:Python39libmultiprocessingcontext.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:Python39libmultiprocessingcontext.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:Python39libmultiprocessingpopen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:Python39libmultiprocessingreduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

[***]>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:Python39libmultiprocessingspawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:Python39libmultiprocessingreduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Access is denied

I have reviewed this similar question, but the key difference is that they are defining the "WorkerThread" to point to a function, with predefined arguments (not a subclass of ‘threading.Thread’).
Example: Within the subProcess, they define workerThread = Thread(target=print_all_the_things, args=("a", self.num))

I require the ability to change the WorkerThread’s class variables while it is running.
Example: In the subProcess, I would do something like self.workerThread.input = "INPUT" while both are running

Any help would be greatly appreciated!

Asked By: CookieCruncher

||

Source

Answer 1

I don’t see any reason with the code you posted (of course, you only posted very minimal code) why the instantiation of the worker thread needs to be done in the SubProcess.__init__ method. I would simply do the initialization of the workerThread attribute in the run method:

class SubProcess(multiprocessing.Process):
    def __init__(self):
        multiprocessing.Process.__init__(self, daemon=True)

    def run(self):
        #Instantiate this one worker
        self.workerThread = WorkerThread()
        #Start the worker
        self.workerThread.start()

I don’t think it is a fatal flaw to derive classes from Process and Thread classes but it is not very flexible. What if the code using your SubProcess class wanted to assign a name attribute to the process? In your current implementation it can’t without your redesigning the __init__ method. The following implementation is just cleaner and makes the classes reusable in non-multiprocessing, non-multithreading scenarios (for which I would possibly choose different method names other than run that better describe the processing done by the method):

import multiprocessing
import threading

class SubProcess:
    def __init__(self):
        # set whatever attributes are required (that can be pickled)
        ...

    def run(self): # or a more descriptive method name
        #Instantiate this one worker
        worker = Worker()
        self.workerThread = threading.Thread(target=worker.run, daemon=True)
        #Start the worker
        self.workerThread.start()

class Worker:
    def __init__(self):
        # set whatever attributes are required
        ...

    def run(self): # or a more descriptive method name
        #Real work code goes here
        pass


if __name__ == '__main__':
    subprocess = SubProcess()
    # In this context, the Process does not need to be a daemon process
    # since we are waiting for its complettion:
    p = multiprocessing.Process(target=subprocess.run, name='SubProcess')
    p.start()
    p.join()

Answered By: Booboo

Python – Issue with creating a Thread subclass within a Process subclass – threading & multiprocessing

Question:

Answers: