Possible to share in-memory data between 2 separate processes?

mmap.mmap(0, 65536, 'GlobalSharedMemory')

I think the tag (“GlobalSharedMemory”) must be the same for all processes wishing to share the same memory.

http://docs.python.org/library/mmap.html

You can use the Python Multiprocessing module.

http://docs.python.org/library/multiprocessing.html#sharing-state-between-processes

Why not just use a database for the shared data? You have a multitude of lightweight options where you don’t need to worry about the concurrency issues: sqlite, any of the nosql/key-value breed of databases, etc.

Why not stick the shared data into memcache server? then both servers can access it quite easily.

You could write a C library to create and manipulate shared-memory arrays for your specific purpose, and then use ctypes to access them from Python.

Or, put them on the filesystem in /dev/shm (which is tmpfs). You’d save a lot of development effort for very little performance overhead: reads/writes from a tmpfs filesystem are little more than a memcpy.

There are a couple¹ of third party libraries available for low-level shared memory manipulations in Python:

• sysv_ipc
  • > For posix non-compliant systems
• posix_ipc
  • > Works in Windows with cygwin

Both of which are available via pip

[1] Another package, shm, is available but deprecated. See this page for a comparison of the libraries.

Example Code for C to Python Communication c/o Martin O’Hanlon:

shmwriter.c

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>

int main(int argc, const char **argv)
{
   int shmid;
   // give your shared memory an id, anything will do
   key_t key = 123456;
   char *shared_memory;

   // Setup shared memory, 11 is the size
   if ((shmid = shmget(key, 11, IPC_CREAT | 0666)) < 0)
   {
      printf("Error getting shared memory id");
      exit(1);
   }
   // Attached shared memory
   if ((shared_memory = shmat(shmid, NULL, 0)) == (char *) -1)
   {
      printf("Error attaching shared memory id");
      exit(1);
   }
   // copy "hello world" to shared memory
   memcpy(shared_memory, "Hello World", sizeof("Hello World"));
   // sleep so there is enough time to run the reader!
   sleep(10);
   // Detach and remove shared memory
   shmdt(shmid);
   shmctl(shmid, IPC_RMID, NULL);
}

shmreader.py

import sysv_ipc

# Create shared memory object
memory = sysv_ipc.SharedMemory(123456)

# Read value from shared memory
memory_value = memory.read()

# Find the 'end' of the string and strip
i = memory_value.find('')
if i != -1:
    memory_value = memory_value[:i]

print memory_value

Simple actually. You can just use shared memory. This example creates a list of tuples (python) in C++ and shares it with a python process that can then use the list of tuples. To use between two Python processes, just make your access as ACCESS_WRITE on the sender process and call the write method.

C++ (sender process):

#include <windows.h>
#include <stdio.h>
#include <conio.h>
#include <tchar.h>
#include <iostream>
#include <string>

#define BUF_SIZE 256
TCHAR szName[]=TEXT("Global\MyFileMappingObject");
TCHAR szMsg[]=TEXT("[(1, 2, 3), ('a', 'b', 'c', 'd', 'e'), (True, False), 'qwerty']");

int _tmain(int argc, _TCHAR* argv[])
{
     HANDLE hMapFile;
   LPCTSTR pBuf;

   hMapFile = CreateFileMapping(
                 INVALID_HANDLE_VALUE,    // use paging file
                 NULL,                    // default security
                 PAGE_READWRITE,          // read/write access
                 0,                       // maximum object size (high-order DWORD)
                 BUF_SIZE,                // maximum object size (low-order DWORD)
                 szName);                 // name of mapping object

   if (hMapFile == NULL)
   {
      _tprintf(TEXT("Could not create file mapping object (%d).n"),
             GetLastError());
      return 1;
   }
   pBuf = (LPTSTR) MapViewOfFile(hMapFile,   // handle to map object
                        FILE_MAP_ALL_ACCESS, // read/write permission
                        0,
                        0,
                        BUF_SIZE);

   if (pBuf == NULL)
   {
      _tprintf(TEXT("Could not map view of file (%d).n"),
             GetLastError());

       CloseHandle(hMapFile);
       return 1;
   }

   CopyMemory((PVOID)pBuf, szMsg, (_tcslen(szMsg) * sizeof(TCHAR)));
    _getch();

   UnmapViewOfFile(pBuf);

   CloseHandle(hMapFile);
    return 0;
}

Python (receiver process):

import mmap
shmem = mmap.mmap(0,256,"Global\MyFileMappingObject",mmap.ACCESS_READ)
msg_bytes = shmem.read()
msg_utf16 = msg_bytes.decode("utf-16")
code = msg_utf16.rstrip('')
yourTuple = eval(code)

If you data is simply tuples, and you’re willing to access these as either an

• (nrows x tuplewidth) np.ndarrays, or
• n 1d np.ndarrays

then I highly recommend using numpy’s wrapper for memmap.

My understanding is:

• you save your numpy arrays as a flat memmap file that holds the raw array contents
• each process points an ndarray to the memmap file as its backing data. The documentation link shows how.

This works great for read-only data.
If you want read-write, you’ll need to uses the multiprocess locks to protect access.

Because memmap uses paging to load the data its a blazingly fast way to access large datasets from disk. In fact, I don’t think modern OSs have any faster to loaded data from disk into memory than this — no serialization is involved.

You could use shared_memory in 3.8.

https://docs.python.org/3.8/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory