'unlink()' does not work in Python's shared_memory on Windows

Question

I am using Python 3.8’s new shared_memory module and fail to free the shared memory without terminating the processes using it.

After creating and using a block shm of shared memory, I close it via shm.close() in all processes and finally free it via shm.unlink in the main process. However, the reseource monitor shows me that the memory is not freed up until the program is terminated. This is a serious problem for me, because my program needs to run for a long time. The problem can be reproduced on Windows/Python 3.8 with the following program:

from multiprocessing import shared_memory, Pool
from itertools import repeat
from time import sleep

def fun(dummy, name):
    
    # access shared memory
    shm = shared_memory.SharedMemory(name=name)
    
    # do work
    sleep(1)
    
    # release shared memory
    shm.close()
    
    return dummy

def meta_fun(pool):
    
    # create shared array
    arr = shared_memory.SharedMemory(create=True, size=500000000)
    
    # compute result
    result = sum(pool.starmap(fun, zip(range(10), repeat(arr.name))))
    
    # release and free memory
    arr.close()
    arr.unlink()
    
    return result

if __name__ == '__main__':
    
    # use one Pool for many method calls to save the time for repeatedly
    # creating processes
    with Pool() as pool:
        for i in range(100):
            print(meta_fun(pool))

Caution: when executing this script, you may quickly fill your entire memory! Watch the "virtual memory" panel in the resource monitor.

After doing some research, I found out that (1) the unlink() function does nothing on Windows:

def unlink(self):
    """Requests that the underlying shared memory block be destroyed.
    In order to ensure proper cleanup of resources, unlink should be
    called once (and only once) across all processes which have access
    to the shared memory block."""
    if _USE_POSIX and self._name:
        from .resource_tracker import unregister
        _posixshmem.shm_unlink(self._name)
        unregister(self._name, "shared_memory")

and (2) Windows seems to free up shared memory once the processeses that created/used it have stopped (see the comments here and here). This may be the cause for Python not handling this explicitly.

In response I have built an ugly workaround via saving and reusing the same shared memory block repeatedly without ever unlinking it. Obviously, this is not a satisfactory solution, especially if the sizes of the needed memory blocks change dynamically.

Is there a way I can manually free up the shared memory on Windows?

Asked By: Samufi

||

Source

Answer 1

This is a bug in the multiprocessing module, which was reported as Issue 40882. There is an open pull request that fixes it, PR 20684, though apparently it’s been slow to merge.

The bug is as follows: in SharedMemory.__init__, we have an invocation of the MapViewOfFile API without a corresponding UnmapViewOfFile, and the mmap object does not take ownership of it either (it maps the block again on its own).

In the meantime, you can monkey-patch the shared_memory module so that the missing UnmapViewOfFile call is added after mmap is constructed. You will probably have to rely on ctypes, as the _winapi module does not export UnmapViewOfFile, despite exporting MapViewOfFile (!). Something like this (not tested):

import ctypes, ctypes.wintypes
import multiprocessing, multiprocessing.shared_memory

UnmapViewOfFile = ctypes.windll.kernel32.UnmapViewOfFile
UnmapViewOfFile.argtypes = (ctypes.wintypes.LPCVOID,)
UnmapViewOfFile.restype = ctypes.wintypes.BOOL

def _SharedMemory_init(self, name=None, create=False, size=0):
    ... # copy from SharedMemory.__init__ in the original module
                try:
                    p_buf = _winapi.MapViewOfFile(
                        h_map,
                        _winapi.FILE_MAP_READ,
                        0,
                        0,
                        0
                    )
                finally:
                    _winapi.CloseHandle(h_map)
                try:
                    size = _winapi.VirtualQuerySize(p_buf)
                    self._mmap = mmap.mmap(-1, size, tagname=name)
                finally:
                    UnmapViewOfFile(p_buf)
    ... # copy from SharedMemory.__init__ in the original module

multiprocessing.shared_memory.SharedMemory.__init__ = _SharedMemory_init

Put the above code into a module and remember to load it before using anything from the multiprocessing module. Alternatively, you can directly edit the shared_memory.py file in the multiprocessing module’s directory to contain the required UnmapViewOfFile call. This is not the cleanest solution, but it’s meant to be temporary anyway (famous last words); the long-term solution is to have this fixed upstream (as is apparently underway).

Answered By: user3840170

Answer 2

Maybe not spot on but unlink still does not allow me to recreate the shm with the same name after the path user3840170 gave.

del worked for me (Py3.10)

Answered By: jwasys

'unlink()' does not work in Python's shared_memory on Windows

Question:

Answers: