python multiprocessing : AttributeError: Can't pickle local object

Question

I have a method inside a class to return a func which parameters may change.

The Interface function accept two parameters, f and its args.I want to use mp.pool to accelerate it.However, it returns an error.

from multiprocessing import Pool
# from multiprocess import Pool
# from pathos.multiprocessing import ProcessingPool as Pool
import pickle
import dill


class Temp:
    def __init__(self, a):
        self.a = a

    def test(self):
        def test1(x):
            return self.a + x

        return test1


def InterfaceFunc(f, x):
    mypool = Pool(4)
    return list(mypool.map(f, x))


if __name__ == "__main__":
    t1 = Temp(1).test()
    x = [1, 2, 3, 1, 2]

    res1 = list(map(t1, x))
    print(res1)

    res2 = InterfaceFunc(t1, x)

it raise the same error:

AttributeError: Can't pickle local object 'Temp.test.<locals>.test1'

I have tried 3 method:

What can multiprocessing and dill do together?

Replace pickle in Python multiprocessing lib

Python Multiprocessing Pool Map: AttributeError: Can't pickle local object

Method 1, 2 :

 from multiprocess import Pool
 from pathos.multiprocessing import ProcessingPool as Pool

It raise error:

 File "E:UsersllAnaconda3libsite-packagesdill_dill.py", line 577, in _load_type
    return _reverse_typemap[name]
KeyError: 'ClassType'

Method3 needs to change the code , however I cant simply move the func out of the class because I need f to be a parameter for the Interface.

Do you have any suggestions? I’m an inexperienced newcomer.

Asked By: abcyunice

||

Source

Answer 1

Python can’t pickle the closure, but all you really need is something that you can call that retains state. The __call__ method makes a class instance callable, so use that

from multiprocessing import Pool

class TempTest1:

    def __init__(self, a):
        self.a = a

    def __call__(self, x):
        return self.a + x

class Temp:
    def __init__(self, a):
        self.a = a

    def test(self):
        return TempTest1(self.a)

def InterfaceFunc(f, x):
    mypool = Pool(4)
    return list(mypool.map(f, x))

if __name__ == "__main__":
    t1 = Temp(1).test()
    x = [1, 2, 3, 1, 2]

    res1 = list(map(t1, x))
    print(res1)

    res2 = InterfaceFunc(t1, x)
    print(res2)

Answered By: tdelaney

Answer 2

A bit late to the party, but passing the class function (the one you want) into a wrapper function did the trick for me:

from multiprocessing import Pool

def myWorkFunc(load):
    return load["fu"](load["i"])

class Test():
    def __init__(self):
        self.a = 1

    def add_one(self, x):
        print('adding', x)
        return x + 1

    def do_something(self):
        loads = [{"i": i, "fu": self.add_one} for i in range(4)]
        with Pool(4) as mypool:
            return mypool.map(func=myWorkFunc, iterable=loads)

if __name__ == '__main__':
    T = Test()
    print(T.do_something())

Answered By: Patty

Answer 3

I tested multiple ways to handle this error and I think the easiest one would be to get to the file that is giving you this error and substitute 1 with 2.

import pickle
import dill as pickle

dill library extends the features of the pickle and will handle this error for you.

Good Luck!

Answered By: arsalan Yaghoubi

python multiprocessing : AttributeError: Can't pickle local object

Question:

Answers: