Why does PyImport_Import fail to load a module from the current directory?
Question:
I’m trying to run the embedding example and I can’t load a module from the current working directory unless I explicitly add it to sys.path then it works:
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(".")");
Shouldn’t Python look for modules in the current directory ?
Edit1: Tried just importing the module with:
Py_Initialize();
PyRun_SimpleString("import multiply");
And it still fails with the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named multiply
Edit2: From the sys.path docs:
If the script directory is not available (e.g. if the interpreter is
invoked interactively or if the script is read from standard input),
path[0] is the empty string, which directs Python to search modules
in the current directory first.
Not sure what it means by not available, but if I print sys.path[0] it’s not empty:
/usr/lib/pymodules/python2.7
Answers:
You need to call PySys_SetArgv(int argc, char **argv, int updatepath) for the relative imports to work. This will add the path of the script being executed to sys.path if updatepath is 0 (refer to the docs for more information).
The following should do the trick
#include <Python.h>
int
main(int argc, char *argv[])
{
Py_SetProgramName(argv[0]); /* optional but recommended */
Py_Initialize();
PySys_SetArgv(argc, argv); // must call this to get sys.argv and relative imports
PyRun_SimpleString("import os, sysn"
"print sys.argv, "\n".join(sys.path)n"
"print os.getcwd()n"
"import thingn" // import a relative module
"thing.printer()n");
Py_Finalize();
return 0;
}
I had exactly the same problem and I solved it just by adding the Py_Initialize(); and Py_Finalize();
Hope that can help you
What I had to do with python 3.5 is PySys_SetPath to be able to import from the cwd location:
QString qs = QDir::currentPath();
std::wstring ws = qs.toStdWString();
PySys_SetPath(ws.data());
The Qs in it is Qt.
To load a module from the current directory, there are many options.
- You need to configure Python (including the path configuration) through an initialization struct. See Python Initialization Configuration.
Py_Initialize():
//...
PyConfig config;
PyConfig_InitPythonConfig(&config);
config.module_search_paths_set = 1;
PyWideStringList_Append(&config.module_search_paths, L"."); // append local dir to path
Py_InitializeFromConfig(&config);
//...
pModule = PyImport_Import(pName); // import module
- A one-liner option is to run
PySys_SetArgv(0, NULL); also after Py_Initialize() to prepend current directory to Python’s Path. Although this option is deprecated since Python 3.11, it is simple and a valid option in many situations. This option takes advantage of the behavior of PySys_SetArgvEx(int argc, char **argv, int updatepath) when satisfying:
- non-zero
updatepath
argc = 0 or argv not pointing to an existing filename.
See docs. Valid from Python 2.7 onwards. For adding a specific directory, argv[0] should contain the path to a Python script in that directory (for example "NULL").PySys_SetArgv sets the updatepath value correctly for you. Although, as mentioned, this option is currently deprecated.
Example Project
Load a module defined in a Python script located in the current directory and call a function from that module, all specified with input arguments to a C application. Checks deleted for simplicity. This example is closely related to the example provided in the documentation page for embedding Python in a C/C++ application.
./hello.py
def sayHello1():
print("Hello from sayHello1!")
return 1
def sayHello2():
print("Hello from sayHello2!")
return 2
./main.c
(checks removed for simplicity)
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName = NULL;
PyObject *pModule = NULL;
PyObject *pFunc = NULL;
PyObject *pValue = NULL;
Py_Initialize();
// PySys_SetArgv(0, NULL); // Option 2 [deprecated]
// Option 1 - Python Initialization api
PyConfig config;
PyConfig_InitPythonConfig(&config);
config.module_search_paths_set = 1;
PyWideStringList_Append(&config.module_search_paths, L".");
Py_InitializeFromConfig(&config);
// ...
pName = PyUnicode_FromString(argv[1]);
pModule = PyImport_Import(pName);
Py_XDECREF(pName);
pFunc = PyObject_GetAttrString(pModule, argv[2]);
pValue = PyObject_CallNoArgs(pFunc); // actual call to the fcn
Py_XDECREF(pValue);
Py_XDECREF(pFunc);
Py_XDECREF(pModule);
Py_Finalize();
return 0;
}
Compilation (cmake):
cmake_minimum_required(VERSION 3.2)
project(python_c_api)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
find_package(Python COMPONENTS Interpreter Development)
cmake_path(GET Python_EXECUTABLE PARENT_PATH Python_BIN_DIR)
set(config_cmd_flags "--cflags")
execute_process(COMMAND ${Python_BIN_DIR}/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}-config ${config_cmd_flags}
OUTPUT_VARIABLE Python_FLAGS)
add_executable(python_extension main.c)
target_link_libraries(python_extension PUBLIC python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR})
target_link_directories(python_extension PUBLIC ${Python_LIBRARY_DIRS})
target_include_directories(python_extension PUBLIC ${Python_INCLUDE_DIRS})
set_target_properties(python_extension PROPERTIES
OUTPUT_NAME "call"
RUNTIME_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}
)
separate_arguments(Python_FLAGS_NORM UNIX_COMMAND "${Python_FLAGS}")
target_compile_options(python_extension PUBLIC ${Python_FLAGS_NORM})
cmake .
cmake --build .
Execution: (checks deleted for simplicity. Call the app as shown here).
./call hello sayHello1
Hello from sayHello1!
./call hello sayHello2
Hello from sayHello2!
Edit: This solution appends the current working directory to the search path, not the directory where the binary executable containing the embedded python is stored. Thus, if the application is called from a different directory, that directory will be appended to the search path.
Adding the path of the binary executable app (containing the python interpreter embedded) to the search path is not trivial and there doesn’t seem to exist an easy cross-platform solution. If this is required, I recommend wrap the app in another app or script running in the correct directory.
I’m trying to run the embedding example and I can’t load a module from the current working directory unless I explicitly add it to sys.path then it works:
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(".")");
Shouldn’t Python look for modules in the current directory ?
Edit1: Tried just importing the module with:
Py_Initialize();
PyRun_SimpleString("import multiply");
And it still fails with the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named multiply
Edit2: From the sys.path docs:
If the script directory is not available (e.g. if the interpreter is
invoked interactively or if the script is read from standard input),
path[0] is the empty string, which directs Python to search modules
in the current directory first.
Not sure what it means by not available, but if I print sys.path[0] it’s not empty:
/usr/lib/pymodules/python2.7
You need to call PySys_SetArgv(int argc, char **argv, int updatepath) for the relative imports to work. This will add the path of the script being executed to sys.path if updatepath is 0 (refer to the docs for more information).
The following should do the trick
#include <Python.h>
int
main(int argc, char *argv[])
{
Py_SetProgramName(argv[0]); /* optional but recommended */
Py_Initialize();
PySys_SetArgv(argc, argv); // must call this to get sys.argv and relative imports
PyRun_SimpleString("import os, sysn"
"print sys.argv, "\n".join(sys.path)n"
"print os.getcwd()n"
"import thingn" // import a relative module
"thing.printer()n");
Py_Finalize();
return 0;
}
I had exactly the same problem and I solved it just by adding the Py_Initialize(); and Py_Finalize();
Hope that can help you
What I had to do with python 3.5 is PySys_SetPath to be able to import from the cwd location:
QString qs = QDir::currentPath();
std::wstring ws = qs.toStdWString();
PySys_SetPath(ws.data());
The Qs in it is Qt.
To load a module from the current directory, there are many options.
- You need to configure Python (including the path configuration) through an initialization struct. See Python Initialization Configuration.
Py_Initialize():
//...
PyConfig config;
PyConfig_InitPythonConfig(&config);
config.module_search_paths_set = 1;
PyWideStringList_Append(&config.module_search_paths, L"."); // append local dir to path
Py_InitializeFromConfig(&config);
//...
pModule = PyImport_Import(pName); // import module
- A one-liner option is to run
PySys_SetArgv(0, NULL);also afterPy_Initialize()to prepend current directory to Python’s Path. Although this option is deprecated since Python 3.11, it is simple and a valid option in many situations. This option takes advantage of the behavior ofPySys_SetArgvEx(int argc, char **argv, int updatepath)when satisfying:
- non-zero
updatepath argc = 0orargvnot pointing to an existing filename.
See docs. Valid from Python 2.7 onwards. For adding a specific directory, argv[0] should contain the path to a Python script in that directory (for example "NULL").PySys_SetArgv sets the updatepath value correctly for you. Although, as mentioned, this option is currently deprecated.
Example Project
Load a module defined in a Python script located in the current directory and call a function from that module, all specified with input arguments to a C application. Checks deleted for simplicity. This example is closely related to the example provided in the documentation page for embedding Python in a C/C++ application.
./hello.py
def sayHello1():
print("Hello from sayHello1!")
return 1
def sayHello2():
print("Hello from sayHello2!")
return 2
./main.c
(checks removed for simplicity)
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName = NULL;
PyObject *pModule = NULL;
PyObject *pFunc = NULL;
PyObject *pValue = NULL;
Py_Initialize();
// PySys_SetArgv(0, NULL); // Option 2 [deprecated]
// Option 1 - Python Initialization api
PyConfig config;
PyConfig_InitPythonConfig(&config);
config.module_search_paths_set = 1;
PyWideStringList_Append(&config.module_search_paths, L".");
Py_InitializeFromConfig(&config);
// ...
pName = PyUnicode_FromString(argv[1]);
pModule = PyImport_Import(pName);
Py_XDECREF(pName);
pFunc = PyObject_GetAttrString(pModule, argv[2]);
pValue = PyObject_CallNoArgs(pFunc); // actual call to the fcn
Py_XDECREF(pValue);
Py_XDECREF(pFunc);
Py_XDECREF(pModule);
Py_Finalize();
return 0;
}
Compilation (cmake):
cmake_minimum_required(VERSION 3.2)
project(python_c_api)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
find_package(Python COMPONENTS Interpreter Development)
cmake_path(GET Python_EXECUTABLE PARENT_PATH Python_BIN_DIR)
set(config_cmd_flags "--cflags")
execute_process(COMMAND ${Python_BIN_DIR}/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}-config ${config_cmd_flags}
OUTPUT_VARIABLE Python_FLAGS)
add_executable(python_extension main.c)
target_link_libraries(python_extension PUBLIC python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR})
target_link_directories(python_extension PUBLIC ${Python_LIBRARY_DIRS})
target_include_directories(python_extension PUBLIC ${Python_INCLUDE_DIRS})
set_target_properties(python_extension PROPERTIES
OUTPUT_NAME "call"
RUNTIME_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}
)
separate_arguments(Python_FLAGS_NORM UNIX_COMMAND "${Python_FLAGS}")
target_compile_options(python_extension PUBLIC ${Python_FLAGS_NORM})
cmake .
cmake --build .
Execution: (checks deleted for simplicity. Call the app as shown here).
./call hello sayHello1
Hello from sayHello1!
./call hello sayHello2
Hello from sayHello2!
Edit: This solution appends the current working directory to the search path, not the directory where the binary executable containing the embedded python is stored. Thus, if the application is called from a different directory, that directory will be appended to the search path.
Adding the path of the binary executable app (containing the python interpreter embedded) to the search path is not trivial and there doesn’t seem to exist an easy cross-platform solution. If this is required, I recommend wrap the app in another app or script running in the correct directory.