yet another confusion with multiprocessing error, 'module' object has no attribute 'f'

Question

I know this has been answered before, but it seems that executing the script directly “python filename.py” does not work. I have Python 2.6.2 on SuSE Linux.

Code:

#!/usr/bin/python
# -*- coding: utf-8 -*-
from multiprocessing import Pool
p = Pool(1)
def f(x):
    return x*x
p.map(f, [1, 2, 3])

Command line:

> python example.py
Process PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 231, in _bootstrap
    self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
File "/usr/lib/python2.6/multiprocessing/queues.py", line 339, in get
    return recv()
AttributeError: 'module' object has no attribute 'f'

Asked By: gatoatigrado

||

Source

Answer 1

This one works:

#!/usr/bin/python
# -*- coding: utf-8 -*-
from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == "__main__":
    p = Pool(1)
    p.map(f, [1, 2, 3])

I’m not 100% sure why your code does not work, but I guess the reason is that child processes launched by the multiprocessing module try to import the main module (to have access to the methods you defined), and the if __name__ == "__main__" stanza is required not to execute the initialization code where you set up your pool.

Answered By: Tamás

Answer 2

Restructure your code so that the f() function is defined before you create instance of Pool. Otherwise the worker cannot see your function.

#!/usr/bin/python
# -*- coding: utf-8 -*-

from multiprocessing import Pool

def f(x):
    return x*x

p = Pool(1)
p.map(f, [1, 2, 3])

Answered By: Bartosz

Answer 3

One possibility is that your python file has the same name as a module:

test.py
test/
- __init__.py

in pickle.py, you have the error coming from:

    def find_class(self, module, name):
      # Subclasses may override this
      __import__(module)
      mod = sys.modules[module] # <- here mod will reference your test/__init__.py
      klass = getattr(mod, name)
      return klass

Answered By: user1143523

Answer 4

The problem I had was solved by using if __name__ == "__main__" as pointed out by Tamás; in Eclipse for Windows the examples do not work under the interpreter.
This is explained in
http://docs.python.org/2/library/multiprocessing

Answered By: Pluto

Answer 5

This comes from the fact that with p = Pool(1) the main process forks processes (threads vs processes) before it creates the function f. As stated in Bartosz answer the spawned processes do not have access to the new function.

def f1(x):
    ...

p = Pool(1) # p is spawned and is now an independent process, knows f1

def f(x): # p doesn't not share this object
    ...

Answered By: yosemite_k

yet another confusion with multiprocessing error, 'module' object has no attribute 'f'

Question:

Answers: