Embedded python: multiprocessing not working

Question

I’m using Python 3.1.4 that is embedded as a scripting environment in an application(x64).
So far I have encountered a lot of limitations with the embedded python. I don’t know if it is normal or if the programmers of the application have blocked some functionalities.

For example the following code isn’t working:

from multiprocessing import Process
def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

# --> error in forking.py: 'module' object has no attribute 'argv'
# print(sys.argv) gives the same error

sys.executable return the path to the application.

I’ve tried this as wel:

multiprocessing.forking.set_executable('C:Python31python.exe')
multiprocessing.set_executable('C:Python31python.exe')

Without success.

Is there a workaround possible ? It is very unlikely that I would have the leverage to make the developers of the application change something in their code.

Thanks

EDIT

I got it to work by adding the following:

sys.argv = ['c:/pathToScript/scipt.py']

I needed this line as well:

multiprocessing.set_executable('C:/Python31/python.exe')

Otherwise an other instance of the application open instead of running the code.

The only problem I have left is that I can’t use the methods that control the application itself (like: create_project(), add_report(),..). My primary goal was to be able to call multiple methods without the need to wait for the first one to finish completion. But I think this is just not possible.

Asked By: F. Justin

||

Source

Answer 1

By default, sys.argv is not available in embedded code:

Embedding Python

The basic initialization function is Py_Initialize(). This initializes
the table of loaded modules, and creates the fundamental modules
builtins, __main__, and sys. It also initializes the module search
path (sys.path).

Py_Initialize() does not set the “script argument list” (sys.argv). If
this variable is needed by Python code that will be executed later, it
must be set explicitly with a call to PySys_SetArgvEx(argc, argv,
updatepath) after the call to Py_Initialize()

On Windows, multiprocessing must spawn new processes from scratch. It uses a command line switch --multiprocessing-fork to distinguish child processes, and also transmits the original argv from parent to child.

Assigning sys.argv = ['c:/pathToScript/scipt.py'] before creating subprocesses, like you discovered,
would seem to be a good workaround.

A second relevant piece of documentation is that of multiprocessing.set_executable():

Sets the path of the Python
interpreter to use when starting a child process. (By default
sys.executable is used). Embedders will probably need to do some thing
like

set_executable(os.path.join(sys.exec_prefix, 'pythonw.exe'))
before
they can create child processes. (Windows only)

Answered By: Janne Karila

Answer 2

If it does not work, try to set sys._base_executable = os.path.join(sys.exec_prefix, ‘pythonw.exe’)

PyObject *msys = PyImport_ImportModule("sys");
PyObject *pyString = PyUnicode_FromWideChar(L"path/to/pythonw.exe", -1);
PyObject_SetAttrString(msys, "executable", pyString);
PyObject_SetAttrString(msys, "_base_executable", pyString);
Py_XDECREF(pyString);

Answered By: rodrigo antonio

Embedded python: multiprocessing not working

Question:

Answers:

Embedding Python