Independent CDLL Library Instances with Ctypes

Question

I am trying to use ctypes and load the same compiled Fortran library twice, such that I have two independent instances of it, such that any module variables that the library contains are not stored in the same memory locations. The general solution described (for example, here: https://mail.python.org/pipermail/python-list/2010-May/575368.html) is to provide the full path to the library instead of just its name. However, I am not able to get this to work like this. Here is a minimal working example that demonstrates the problem:

test.f90:

module test
    use iso_c_binding, only: c_int
    implicit none
    integer :: n
contains
    integer(c_int) function get() bind(c, name='get')
        get = n
    end function get

    subroutine set(new_n) bind(c, name='set')
        integer(c_int), intent(in) :: new_n
        n = new_n
    end subroutine set
end module test

test.py:

import os
from ctypes import cdll, c_int, byref

if __name__ == '__main__':
    lib1 = cdll.LoadLibrary(os.path.abspath('test.so'))
    lib2 = cdll.LoadLibrary(os.path.abspath('test.so'))

    lib1.set(byref(c_int(0)))
    lib2.set(byref(c_int(1)))

    print(lib1.get())

The Fortran library is compiled using the command:

gfortran -shared -fPIC -o test.so test.f90

When I run python test.py I get 1 as output, while I want to get 0. Does anyone know how to make this work?

Asked By: D. de Vries

||

Source

Answer 1

[Python.Docs]: ctypes – A foreign function library for Python loads libraries (on Nix) using DlOpen. According to [Man7]: DLOPEN(3):

If the same shared object is loaded again with dlopen(), the same object handle is returned. The dynamic linker maintains reference counts for object handles, so a dynamically loaded shared object is not deallocated until dlclose() has been called on it as many times as dlopen() has succeeded on it.

I’ve prepared a small example.

Before going further, check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati’s answer) for details on a bug frequently encountered (also in the question) when working with CTypes.

dll00.c:

#if defined(_WIN32)
#  define DLL00_EXPORT_API __declspec(dllexport)
#else
#  define DLL00_EXPORT_API
#endif


static int val = -1;


DLL00_EXPORT_API int get()
{
    return val;
}


DLL00_EXPORT_API void set(int i)
{
    val = i;
}

code00.py:

#!/usr/bin/env python

import ctypes as cts
import os
import shutil
import sys


def get_dll_funcs(dll):
    get_func = dll.get
    get_func.argtypes = ()
    get_func.restype = cts.c_int
    set_func = dll.set
    set_func.argtypes = (cts.c_int,)
    set_func.restype = None
    return get_func, set_func


def main(*argv):
    dll00 = "./dll00.so"
    dll01 = "./dll01.so"
    dir00 = "dir00"
    os.makedirs(dir00, exist_ok=True)
    shutil.copy(dll00, dir00)
    shutil.copy(dll00, dll01)

    dll_names = [dll00, os.path.abspath(dll00), os.path.join(dir00, dll00), dll01]
    dlls = [cts.CDLL(item) for item in dll_names]

    for idx, dll in enumerate(dlls):
        print("Item {:d} ({:s}) was loaded at {:08X}".format(idx, dll_names[idx], dll._handle))
        set_func = get_dll_funcs(dll)[1]
        set_func(idx * 10)

    for idx, dll in enumerate(dlls):
        get_func = get_dll_funcs(dll)[0]
        print("Item {:d} get() returned {: d}".format(idx, get_func()))


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}n".format(" ".join(elem.strip() for elem in sys.version.split("n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("nDone.n")
    sys.exit(rc)

Output:

[cfati@cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q054243176]> ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[064bit prompt]>ls
code00.py  dll00.c
[064bit prompt]> gcc -fPIC -shared -o dll00.so dll00.c
[064bit prompt]> ls
code00.py  dll00.c  dll00.so
[064bit prompt]> python ./code.py
Python 3.5.2 (default, Nov 12 2018, 13:43:14) [GCC 5.4.0 20160609] 064bit on linux

Item 0 (./dll00.so) was loaded at 02437A80
Item 1 (/mnt/e/Work/Dev/StackOverflow/q054243176/dll00.so) was loaded at 02437A80
Item 2 (dir00/./dll00.so) was loaded at 02438690
Item 3 (./dll01.so) was loaded at 02438EF0
Item 0 get() returned  10
Item 1 get() returned  10
Item 2 get() returned  20
Item 3 get() returned  30

As seen from the output (also pay attention to the _handle attribute), trying to load the same .dll (via its path) more than once (same behavior on Win):

If located in the same path (even if it’s differently specified), doesn’t actually load it again, it just increases its RefCount
If either its name or location differs, it is loaded again

In short, to answer your question: simply copy it under a different name and load that.

Answered By: CristiFati

Independent CDLL Library Instances with Ctypes

Question:

Answers: