Pybind11: "ImportError: DLL not found" when trying to import *.pyd in Python Interpreter
Question:
I built a .pyd in Visual Studio 2019 (Community) that provides a wrapper for some functionality that’s only present in the LibRaw. The solution compiles successfully without any warnings or errors. The project uses LibRaw, OpenCV and pybind11 as well as Python.h and the corresponding .lib-file.
When i try to import the .pyd inside the Python Interpreter i get:
C:UsersTim.Hiltsourcereposcr3_converterRelease>dir
Datenträger in Laufwerk C: ist Acer
Volumeseriennummer: EC36-E45E
Verzeichnis von C:UsersTim.Hiltsourcereposcr3_converterRelease
22.01.2020 11:28 <DIR> .
22.01.2020 11:28 <DIR> ..
22.01.2020 11:28 808 cr3_converter.exp
22.01.2020 11:28 3.068.361 cr3_converter.iobj
22.01.2020 11:28 785.552 cr3_converter.ipdb
22.01.2020 11:28 1.908 cr3_converter.lib
22.01.2020 11:28 4.190.208 cr3_converter.pdb
22.01.2020 11:28 953.856 cr3_converter.pyd
31.10.2019 16:22 26.408.085 IMG_0482_raw.CR3
7 Datei(en), 35.408.778 Bytes
2 Verzeichnis(se), 77.160.587.264 Bytes frei
C:UsersTim.Hiltsourcereposcr3_converterRelease>python
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 22:39:24) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cr3_converter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing cr3_converter: The specified module was not found.
>>> import cr3_converter.pyd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing cr3_converter: The specified module was not found.
>>>
The Paths to the needed .dlls (Python and OpenCV in this case; LibRaw was linked fully statically) are set in the system-path.
I ran Dependency-Walker, but could not find anything suspicious. Here is the corresponding Dependency Walker image. I also tried out another tool (Dependencies.exe, which is essentially a rewrite of Dependency Walker, but takes into account the API-MS-WIN-CORE-....dlls) and got an error, which looked like this:
When i hover over the missing .dll, i can see a api-ms-win... module could not be found on disk. I have searched and found the module and added its directory-path to the system-path. Now the module isn’t highlighted red anymore, but the C:WINDOWSSysWOW64WS2_32.dll (red highlight at top of screenshot) still shows to have missing imports. Could this be an issue?
How i’ve produced the .pyd:
- Created empty Visual Studio project (Win32; Python is also the 32-bit-installation)
- Changed project-settings to
.dll-configuration and .pyd-fileextension
- Added include-paths to the header files for OpenCV, LibRaw and Pybind11
- Added paths and input files for the linker for OpenCV, LibRaw and Python3.8
- Built the solution (No Errors, No Warnings)
- Tried to import the resulting
.pyd in the python-interpreter
I’ve seen this question, where the OP had the problem of different Python-.dlls being loaded by the library, but in my case the library referenced by Dependency Walker is the same as the one i have in my path-variable.
Answers:
I quietly assumed, that Windows searches for .dlls in the same directories as the ones listed in the systems (/users) PATH-variable.
However, that is not the case, as ProcMon revealed. For now, i copied the missing .dlls to the folder that contains the .pyd and everything works.
I would like to propose a more manageable way to solve the discussed issue.
The short answer is:
Specify all DLL’s directories by os.add_dll_directory(...) before you import your .pyd-module.
More details.
One may inspect DLLs loading issues by windbg (use ed ntdll!LdrpDebugFlags 1 command) or gflags (run gflags -i <your-app.exe> +sls). Here is example output:
65d8:3868 @ 1347896937 - LdrLoadDll - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpLoadDllInternal - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpResolveDllName - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpResolveDllName - RETURN: Status: 0x00000000
65d8:3868 @ 1347896937 - LdrpMinimalMapModule - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
ModLoad: 00007ffe`47360000 00007ffe`473c5000 D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpMinimalMapModule - RETURN: Status: 0x00000000
65d8:3868 @ 1347896953 - LdrpFindKnownDll - ENTER: DLL name: MyLib.dll
65d8:3868 @ 1347896953 - LdrpFindKnownDll - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpSearchPath - ENTER: DLL name: MyLib.dll
65d8:15e4 @ 1347896953 - LdrpComputeLazyDllPath - INFO: DLL search path computed: D:devMyLib;C:Python310;C:WindowsSYSTEM32
65d8:15e4 @ 1347896953 - LdrpResolveDllName - ENTER: DLL name: D:devMyLibMyLib.dll
65d8:6c30 @ 1347896953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpResolveDllName - ENTER: DLL name: C:Python310MyLib.dll
65d8:15e4 @ 1347896953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpResolveDllName - ENTER: DLL name: C:WindowsSYSTEM32MyLib.dll
65d8:15e4 @ 1347896953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpSearchPath - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpProcessWork - ERROR: Unable to load DLL: "MyLib.dll", Parent Module: "D:devMyLibMyLib.cp310-win_amd64.pyd", Status: 0xc0000135
It is clearly seen that DLLs are looked for only in D:devMyLib;C:Python310;C:WindowsSYSTEM32 directories. This means, that Python ignores %PATH% and do not use %PYTHONPATH. I dare to assume some relation with Modules/_ctypes/callproc.c — the LoadLibraryExW is called there with some specific flags.
This limitations can be overcome by the Windows API’s function AddDllDirectory. One may call it to extend directories list where libraries are looked for.
Fortunately Python provides os.add_dll_directory(...) method to do this. One should specify all DLL’s directories path by means of this method at first. Then .pyd-module will be loaded successfully.
I built a .pyd in Visual Studio 2019 (Community) that provides a wrapper for some functionality that’s only present in the LibRaw. The solution compiles successfully without any warnings or errors. The project uses LibRaw, OpenCV and pybind11 as well as Python.h and the corresponding .lib-file.
When i try to import the .pyd inside the Python Interpreter i get:
C:UsersTim.Hiltsourcereposcr3_converterRelease>dir
Datenträger in Laufwerk C: ist Acer
Volumeseriennummer: EC36-E45E
Verzeichnis von C:UsersTim.Hiltsourcereposcr3_converterRelease
22.01.2020 11:28 <DIR> .
22.01.2020 11:28 <DIR> ..
22.01.2020 11:28 808 cr3_converter.exp
22.01.2020 11:28 3.068.361 cr3_converter.iobj
22.01.2020 11:28 785.552 cr3_converter.ipdb
22.01.2020 11:28 1.908 cr3_converter.lib
22.01.2020 11:28 4.190.208 cr3_converter.pdb
22.01.2020 11:28 953.856 cr3_converter.pyd
31.10.2019 16:22 26.408.085 IMG_0482_raw.CR3
7 Datei(en), 35.408.778 Bytes
2 Verzeichnis(se), 77.160.587.264 Bytes frei
C:UsersTim.Hiltsourcereposcr3_converterRelease>python
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 22:39:24) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cr3_converter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing cr3_converter: The specified module was not found.
>>> import cr3_converter.pyd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing cr3_converter: The specified module was not found.
>>>
The Paths to the needed .dlls (Python and OpenCV in this case; LibRaw was linked fully statically) are set in the system-path.
I ran Dependency-Walker, but could not find anything suspicious. Here is the corresponding Dependency Walker image. I also tried out another tool (Dependencies.exe, which is essentially a rewrite of Dependency Walker, but takes into account the API-MS-WIN-CORE-....dlls) and got an error, which looked like this:
When i hover over the missing .dll, i can see a api-ms-win... module could not be found on disk. I have searched and found the module and added its directory-path to the system-path. Now the module isn’t highlighted red anymore, but the C:WINDOWSSysWOW64WS2_32.dll (red highlight at top of screenshot) still shows to have missing imports. Could this be an issue?
How i’ve produced the .pyd:
- Created empty Visual Studio project (Win32; Python is also the 32-bit-installation)
- Changed project-settings to
.dll-configuration and.pyd-fileextension - Added include-paths to the header files for OpenCV, LibRaw and Pybind11
- Added paths and input files for the linker for OpenCV, LibRaw and Python3.8
- Built the solution (No Errors, No Warnings)
- Tried to import the resulting
.pydin the python-interpreter
I’ve seen this question, where the OP had the problem of different Python-.dlls being loaded by the library, but in my case the library referenced by Dependency Walker is the same as the one i have in my path-variable.
I quietly assumed, that Windows searches for .dlls in the same directories as the ones listed in the systems (/users) PATH-variable.
However, that is not the case, as ProcMon revealed. For now, i copied the missing .dlls to the folder that contains the .pyd and everything works.
I would like to propose a more manageable way to solve the discussed issue.
The short answer is:
Specify all DLL’s directories by os.add_dll_directory(...) before you import your .pyd-module.
More details.
One may inspect DLLs loading issues by windbg (use ed ntdll!LdrpDebugFlags 1 command) or gflags (run gflags -i <your-app.exe> +sls). Here is example output:
65d8:3868 @ 1347896937 - LdrLoadDll - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpLoadDllInternal - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpResolveDllName - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpResolveDllName - RETURN: Status: 0x00000000
65d8:3868 @ 1347896937 - LdrpMinimalMapModule - ENTER: DLL name: D:devMyLibMyLib.cp310-win_amd64.pyd
ModLoad: 00007ffe`47360000 00007ffe`473c5000 D:devMyLibMyLib.cp310-win_amd64.pyd
65d8:3868 @ 1347896937 - LdrpMinimalMapModule - RETURN: Status: 0x00000000
65d8:3868 @ 1347896953 - LdrpFindKnownDll - ENTER: DLL name: MyLib.dll
65d8:3868 @ 1347896953 - LdrpFindKnownDll - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpSearchPath - ENTER: DLL name: MyLib.dll
65d8:15e4 @ 1347896953 - LdrpComputeLazyDllPath - INFO: DLL search path computed: D:devMyLib;C:Python310;C:WindowsSYSTEM32
65d8:15e4 @ 1347896953 - LdrpResolveDllName - ENTER: DLL name: D:devMyLibMyLib.dll
65d8:6c30 @ 1347896953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpResolveDllName - ENTER: DLL name: C:Python310MyLib.dll
65d8:15e4 @ 1347896953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpResolveDllName - ENTER: DLL name: C:WindowsSYSTEM32MyLib.dll
65d8:15e4 @ 1347896953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpSearchPath - RETURN: Status: 0xc0000135
65d8:15e4 @ 1347896953 - LdrpProcessWork - ERROR: Unable to load DLL: "MyLib.dll", Parent Module: "D:devMyLibMyLib.cp310-win_amd64.pyd", Status: 0xc0000135
It is clearly seen that DLLs are looked for only in D:devMyLib;C:Python310;C:WindowsSYSTEM32 directories. This means, that Python ignores %PATH% and do not use %PYTHONPATH. I dare to assume some relation with Modules/_ctypes/callproc.c — the LoadLibraryExW is called there with some specific flags.
This limitations can be overcome by the Windows API’s function AddDllDirectory. One may call it to extend directories list where libraries are looked for.
Fortunately Python provides os.add_dll_directory(...) method to do this. One should specify all DLL’s directories path by means of this method at first. Then .pyd-module will be loaded successfully.