Pickling boost python functions

Question

For the use with joblib.Parallel, I need to be able to pickle a boost::python function.
When I try to do so, I get a

TypeError: can't pickle builtin_function_or_method objects

As far as I understand, the function should be pickled by fully qualified name only.
I don’t see why this is not possible.
Any ideas?

Asked By: Andreas Mueller

||

Source

Answer 1

If you want to use your boost method in a joblib.Parallel object, maybe you could use a wrapper around your boost method :

from joblib import Parallel, delayed
from boost import boost_function

class Wrapper(object):
    def __init__(self, method_name, module_name):
        self.method_name = method_name
        self.module_name = module_name

    def __call__(self, *args, **kwargs):
        method = __import__(self.module_name, globals(), locals(), [self.method_name,])
        return method(*args, **kwargs)

Parallel(n_jobs=1)(delayed(Wrapper("boost_module_name_with_dots", "boost_method_name")(i) for i in range(10))

Answered By: Cédric Julien

Answer 2

The problem is that joblib is trying to transform into a pickle function that is fed into joblib.delayed. It cannot transform the Boost.Python.function object. To solve this problem, you can feed a callable that does not directly have a Boost.Python.function in it. To do this, you can use the approach suggested by https://stackoverflow.com/a/7089719/20222376 with the wrapper.

I ran into this problem when using rdkit. Here is an example of a solution with it that works in 2022.

import importlib

from joblib import Parallel, delayed


class Wrapper:
    def __init__(self, method_name, module_name):
        self.method_name = method_name
        self.module = importlib.import_module(module_name)

    @property
    def method(self):
        return getattr(self.module, self.method_name)

    def __call__(self, *args, **kwargs):
        return self.method(*args, **kwargs)


wrapped_mol_from_smiles = Wrapper("MolFromSmiles", "rdkit.Chem")
smiles = ["CCC", "CC"]
mols = Parallel(n_jobs=2, backend="loky")(
    delayed(wrapped_mol_from_smiles)(x) for x in smiles
)
print(mols)

Answered By: Kirill Milash

Pickling boost python functions

Question:

Answers: