PyBind11 destructor not invoked?

As was mentioned in a comment, the proximate cause of this behavior is the Python garbage collector: When the reference counter for an object gets to zero, the garbage collector may destroy the object (and thereby invoke the c++ destructor) but it doesn’t have to do it at that moment.

This idea is elaborated more fully in the answer here:

https://stackoverflow.com/a/38238013/790979

As also mentioned in the above link, if you’ve got clean up to do at the end of an object’s lifetime in Python, a nice solution is context management, where you’d define __enter__ and __exit__ in the object’s wrapper (either in pybind11 or in Python itself), have __exit__ release the networking resources, and then, in the Python client code, something like:

with listener(print_string, 'tcp://127.0.0.1:9001', clientCertPath, serverCertPath, proxyCertPath, desiredSources, 'time_series_data', enableCurve, enableVerbose) as lstnr:
    # Run for a minute
    cnt = 0
    while cnt < 60:
        cnt += 1
        time.sleep(1)

GoFaster’s solution above is helpful and the correct approach but I just wanted to clarify and correct their assertion that

The Python context manager now calls the destructor on the C++ object thus smoothly releasing the networking resources

This is simply not true. The context manager only guarantees that __exit__ will be called, not that any destructor will be called. Let me demonstrate – here’s a managed resource implemented in C++:

class ManagedResource
{
public:
    ManagedResource(int i) : pi(std::make_unique<int>(i))
    {
        py::print("ManagedResource ctor");
    }

    ~ManagedResource()
    {
        py::print("ManagedResource dtor");
    }

    int get() const { return *pi; }

    py::object enter()
    {
        py::print("entered context manager");
        return py::cast(this);
    }

    void exit(py::handle type, py::handle value, py::handle traceback)
    {
        // release resources
        // pi.reset();
        py::print("exited context manager");
    }

private:
    std::unique_ptr<int> pi;
};

the python bindings:

    py::class_<ManagedResource>(m, "ManagedResource")
    .def(py::init<int>())
    .def("get", &ManagedResource::get)
    .def("__enter__", &ManagedResource::enter, R"""(
        Enter context manager.
    )""")
    .def("__exit__", &ManagedResource::exit, R"""(
        Leave context manager.
    )""");

and some python test code (note that the code above doesn’t (yet) release the resource in __exit__):

def f():
    with ManagedResource(42) as resource1:
        print(f"get = {resource1.get()}")
    print(f"hey look I'm still here {resource1.get()}") # not destroyed


if __name__ == "__main__":
    f()
    print("end")

which produces:

ManagedResource ctor
entered context manager
get = 42
exited context manager
hey look I'm still here 42
ManagedResource dtor
end

so the resource is constructed, acquiring memory, and accessed within the context manager. All good so far. However the memory is still accessible outside the context manager (and before the destuctor is called, which is decided by the python runtime and outside our control unless we force it with del, which completely defeats the point of the context manager.

But we didnt actually release the resource in __exit__. If you uncomment pi.reset() in that function, you’ll get this:

ManagedResource ctor
entered context manager
get = 42
exited context manager
Segmentation fault (core dumped)

this time, when you call get() outside the context manager, the ManagedResource object itself is still very much not destructed, but the resource inside it has been released,

And there’s even more danger: if you create a ManagedResource outside a with block, you’ll leak resource as __exit__ will never get called. To fix this, you’ll need to defer acquiring the resources from the constructor to the __enter__ method, as well as putting checks that the resource exists in get.

In short, the morals of this story are:

• you can’t rely on when/where python objects are destructed, even for context managers
• you can control acquisition and release of resources within a context manager
• resources should be acquired in the __enter__ method, not in the constructor
• resources should be released in the __exit__ method, not is the destructor
• you should put sufficient guards around access to the resources

A context-managed object is not an RAII resource itself, but a wrapper around a RAII resource.