Python error [WinError 6] while using multiprocessing.Process with dill instead of standard Pickle

Question

I have this Process class which overrides standard multiprocessing.Process class in order to catch exceptions occurred in child process. Also I use dill and override multiprocessing standard Pickler because I need to.

import dill
import traceback
import multiprocessing

dill.Pickler.dumps, dill.Pickler.loads = dill.dumps, dill.loads
multiprocessing.reduction.ForkingPickler = dill.Pickler
multiprocessing.reduction.dump = dill.dump

class Process(multiprocessing.Process):

def __init__(self, *args, **kwargs):
    multiprocessing.Process.__init__(self, *args, **kwargs)
    self._parent_conn, self._child_conn = multiprocessing.Pipe()
    self._exception = None

def run(self):
    try:
        multiprocessing.Process.run(self)
        self._child_conn.send(None)
    except Exception as e:
        tb = traceback.format_exc()
        self._child_conn.send((e, tb))

@property
def exception(self):
    if self._parent_conn.poll():
        self._exception = self._parent_conn.recv()
    return self._exception

Sadly, when I try to run child process where I run test function, it runs well until ANY operation associated with Pipe. It can be closing, sending, anything it always crashes on:

TEST
Process Process-1:
Traceback (most recent call last):
  File "c:Users<user>DocumentsProjectssprintersourceprocess.py", line 20, in run
    self._child_conn.send(None)
  File "C:Program FilesWindowsAppsPythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0libmultiprocessingconnection.py", line 211, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:Program FilesWindowsAppsPythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0libmultiprocessingconnection.py", line 285, in _send_bytes
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
OSError: [WinError 6] The handle is invalid

What is important to notice is that test function runs well but sending the info to the Pipe causes problem. Here is also the code that I run process from:

from source.process import Process

if __name__ == "__main__":

module = importlib.import_module('tests.api.test_api')  # Finding and importing module
object = getattr(module, 'TestApi')  # Getting Class from module
test_api = object() # Initializing object

process = Process(
    target=getattr(object, 'test_api_one'),
    args=(test_api,))

process.start()
process.join()
if process.exception:
    error, traceback = process.exception
    print(traceback)

This is caused by dill.. because if I use standard Pickler – it goes well without error. But I need to use dill due to other operations. Is there any workaround for this kind of situation? Why Pipe is having problem while sending info?

Asked By: Alraku

||

Source

Answer 1

I’m the dill author. The error you are getting doesn’t seem to be coming from dill. To me it seems more likely that the replacement of the serialization objects in multiprocessing may not have been done as fully needed.

OSError: [WinError 6] The handle is invalid is not an error that dill should generate… so I’d be surprised if dill is the root of the issue. I’d suggest you trying to use multiprocess (I’m also the author) instead of multiprocessing. It’s a fork of multiprocessing where dill has replaced pickle. Then you don’t need to try to hack the ForkingPickler object, you just import the multiprocess module instead of multiprocessing.

If using multiprocess doesn’t solve your issue, then you may want to submit a ticket to the multiprocess GitHub issue tracker.

Answered By: Mike McKerns

Python error [WinError 6] while using multiprocessing.Process with dill instead of standard Pickle

Question:

Answers: