Skip to content

Table of content

Tips and Tutorials \ PYTHON - Share memory between objects with shared_memory

PYTHON - Share memory between objects with shared_memory

This tip shows how to share and manage memory between processes

Tools:
Difficulty:
Type: Tip
Categories: Code Optimization and Distribution

Overview

The shared_memory module provides a class, SharedMemory, for the allocation and management of shared memory between different processes on a multi-core machine or symmetric multiprocessor (SMP). This kind of shared memory allows different distinct processes to read and write in a common (or shared) region of volatile memory. Processes are commonly limited to access only their own memory space, but shared memory lets data be shared between processes, avoiding sending messages with those data to each other. Sharing data through memory can offer significative advantages in terms of performance compared to sharing data using a disk or other communication means implying serialization/deserialization and data copy.

Usage example

A complete documentation of multiprocessing module available features can be found here.

import numpy as np
from multiprocessing import shared_memory

#In the first Python interactive shell
a= np.zeros((5,5,3))
shm = shared_memory.SharedMemory(create=True, size=a.nbytes, name='shared')

# Now create a NumPy array backed by shared memory
b = np.ndarray(a.shape, dtype=input_array.dtype, buffer=shm.buf)
b[:] = a[:]  # Copy the original data into shared memory


# To get the shared array in an other python shell :    
existing_shm = shared_memory.SharedMemory(name='shared')
c = np.ndarray((5,5,3), dtype=np.float64, buffer=existing_shm.buf )


## Clean up from within the second Python shell
existing_shm.close()

# Clean up from within the first Python shell
shm.close()
shm.unlink()  # Free and release the shared memory block at the very end

Contacts

    Questions ? Get help on the Forum