Python Multiprocessing custom manager with associated objects

Question

I’m trying to make a class object usable for multiple processes. Unfortunarely this seems to be more of an issue than I anticipated.

I have the following class object:

class BusObject:

    inputs: IOObject
    outputs: IOObject

    def __init__(self):
        self.inputs = IOObject()
        self.outputs = IOObject()

with the associated object IOObject

class IOObject:

    idx: List[int]              # signal index
    tag: List[str]              # signal tag

    def __init__(self):

        self.idx = []
        self.tag = []

This combination worked fine. But now I run into the requirement that I have to make the BusObject available to multiple processes.

Therefore I created a custom multiprocessing.manager

class CustomManager(BaseManager):
    pass

def main():

    manager = CustomManager()

    # Registration of custom classes to manager
    manager.register('BusObject', BusObject)
    # manager.register('IOObject', IOObject)

    manager.start()
    
    busObject = manager.BusObject()

Works – almost …

The problem is that the associated objects don’t seem to be registered as well.

I tried to register them, too, but even if I do I run into the error

AttributeError
'AutoProxy[BusObject]' object has no attribute 'inputs'

Any ideas?

Asked By: Viktor Katzy

||

Source

Answer 1

Since BusObject contains IOObject, try registering IOObject first, then BusObject.

Answered By: Roland Smith

Answer 2

Upon reading the docs here – https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.BaseManager.register – I ‘d say you won’t be able to access the .inputs and .outputs attributes directly – you can code your object like that, and the "owner" process of the real BusObject instance can do so – but in order to get instances for the IOObjects that are associated to a BUSObject (by the way, the term you are missing is this: associated objects, not "nested subclasses") – you have to retrieve then through methods that are registered in the method_to_typeid parameter of the call to .register().

So, you’d do

    ...
    def get_inputs(self):
        return self.inputs
    # same for outputs...

method (which otherwise looks silly in Python), and pass {'get_inputs': 'IOObject', 'get_outputs': 'IOObject'} as the method_to_typeid argument on the call to register.

That way, the other defaults of registering a class will create a proxy for BUSObject that will be able to return proxies to IOObject when you call these methods (but busobj_instance.inputs should remain opaque).

It is even possible this can play along with @property so you can register the property getter method as a proxy-returner and be able to use the names as attributes.

Of course, the IOOBject class must be properly registered as well.

Sorry for not providing a working example and testing out the possibilities – but that would take me at least a couple hours to get going.

Answered By: jsbueno

Python Multiprocessing custom manager with associated objects

Question:

Answers: