python module dlls

Question:

Is there a way to make a python module load a dll in my application directory rather than the version that came with the python installation, without making changes to the python installation (which would then require I made an installer, and be careful I didn’t break other apps for people by overwrting python modules and changing dll versions globaly…)?

Specifically I would like python to use my version of the sqlite3.dll, rather than the version that came with python (which is older and doesn’t appear to have the fts3 module).

Asked By: Fire Lancer

||

Answers:

If your version of sqlite is in sys.path before the systems version it will use that. So you can either put it in the current directory or change the PYTHONPATH environment variable to do that.

Answered By: André

If you’re talking about Python module DLLs, then simply modifying sys.path should be fine. However, if you’re talking about DLLs linked against those DLLs; i.e. a libfoo.dll which a foo.pyd depends on, then you need to modify your PATH environment variable. I wrote about doing this for PyGTK a while ago, but in your case I think it should be as simple as:

import os
os.environ['PATH'] = 'my-app-dir' + os.pathsep + os.environ['PATH']

That will insert my-app-dir at the head of your Windows path, which I believe also controls the load-order for DLLs.

Keep in mind that you will need to do this before loading the DLL in question, i.e., before importing anything interesting.

sqlite3 may be a bit of a special case, though, since it is distributed with Python; it’s obviously kind of tricky to test this quickly, so I haven’t checked sqlite3.dll specifically.

Answered By: Glyph

Ok it turns out python always loads the dll in the same directory as the pyd file, regardless of what the python and os paths are set to.

So I needed to copy the _sqlite3.pyd from python/v2.5/DLLS to my apps directory where the new sqlite3.dll is, making it load my new dll, rather than the one that comes with python (since the pyd files seem to follow the PYTHONPATH, even though the actual dlls themselves don’t).

Answered By: Fire Lancer

The answer with modifying os.environ['PATH'] is right but it didn’t work for me because I use python 3.9.
Still I was getting an error:

ImportError: DLL load failed while importing module X: The specified
module could not be found.

Turned out since version python 3.8 they added a mechanism to do this more securely.
Read documentation on os.add_dll_directory https://docs.python.org/3/library/os.html#os.add_dll_directory

Specifically see python 3.8 what’s new:

DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library.

So now this is the new way to make it work in python 3.8 and later:

import os
os.add_dll_directory('my-app-dir')

Again, the old way is still correct and you will have to use it in python 3.7 and older:

import os
os.environ['PATH'] = 'my-app-dir' + os.pathsep + os.environ['PATH']

After that my module with a dll dependency has been successfully loaded.

Answered By: Andrei Smeltsov

I had the same issue as administrative rights to the default python library is blocked in a corporate environment and its extremely troublesome to perform installations.

What works for me:

  1. Duplicate the sqlite3 library in a new location
  2. Put in the latest sqlite3.dll (version you want from sqlite3 web) and the old _sqlite3.pyd into the new location or the new sqlite3 library. The old _sqlite3.pyd can be found in the default python library lib/DLLs folder.
  3. Go to the new sqlite3 library and amend the dbapi2.py as follows:
    Change "from _sqlite3 import *" to "from sqlite3._sqlite3 import *"
  4. Make sure python loads this new sqlite3 library first. Add the path to this library if you must.
Answered By: James L

I encountered the same problem for my .pyd that depends on cuda/cudnn/tensorrt.

I came with a little function I call before importing my module.

def add_cuda_to_path():
    if os.name != "nt":
        return
    path = os.getenv("PATH")
    if not path:
        return
    path_split = path.split(";")
    for folder in path_split:
        if "cuda" in folder.lower() or "tensorrt" in folder.lower():
            os.add_dll_directory(folder)

It is the easiest workaround I found, so I don’t need to hardcode any path.
I guess little improvement would to actually check for .dll file, but this snippet serves well my needs.

Then you can simply:

add_cuda_to_path()
import my_module_that_depends_on_cuda
Answered By: Olivier Le Doeuff
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.