Plotting with multiprocessing

Question:

Is it possible to plot in parallel processes? If not, can you please explain why this is the case?

In the script below I’ve provided an example of plotting in a loop, a ThreadPool and a multiprocessing Pool. In the loop and ThreadPool things work as expected. In the multiprocessing Pool we arrive with a blank plot.

import matplotlib.pyplot as plt
import numpy as np
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
from functools import partial


def plot(ax, x, y):
    ax.plot(x, y)


if __name__ == "__main__":
    x = np.linspace(0, 2 * np.pi, 100)
    y_arr = [np.sin(x), np.cos(x)]
    fig, ax = plt.subplots()
    # works as expected
    for yi in y_arr:
        plot(ax, x, yi)
    plt.show()
    # works as expected
    fig, ax = plt.subplots()
    with ThreadPool(5) as p:
        p.map(partial(plot, ax, x), y_arr)
    plt.show()
    # doesn't work
    fig, ax = plt.subplots()
    with Pool(5) as p:
        p.map(partial(plot, ax, x), y_arr)
    plt.show()

As a side note, running

with Pool(5) as p:
    p.map(partial(ax.plot, x), y_arr)

causes a hard crash multiprocessing/resou rce_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' prompting a crash report from my os.

Asked By: KCQs

||
Source

Answers:

The issue is with the multiprocessing pool. You have to plot in parallel since each pool is creating a new process that isn’t aware of the other plots in the backend. That’s why you are getting a blank plot. Use matplotlib.use('Agg').

import matplotlib
matplotlib.use('Agg')

import matplotlib.pyplot as plt
import numpy as np
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
from functools import partial


def plot(ax, x, y):
    ax.plot(x, y)


if __name__ == "__main__":
    x = np.linspace(0, 2 * np.pi, 100)
    y_arr = [np.sin(x), np.cos(x)]
    fig, ax = plt.subplots()
    for yi in y_arr:
        plot(ax, x, yi)
    plt.show()
    fig, ax = plt.subplots()
    with ThreadPool(5) as p:
        p.map(partial(plot, ax, x), y_arr)
    plt.show()
    fig, ax = plt.subplots()
    with Pool(5) as p:
        p.map(partial(plot, ax, x), y_arr)
    plt.show()
Answered By: tamarajqawasmeh

Is it possible to plot in parallel processes? If not, can you please explain why this is the case?

No, you cannot trivially plot to the same window from multiple processes, as each process can only draw to the windows it own, multiprocessing spawns multiple processes, and while each one can create its own window to draw to, you can’t have them all drawing to the same window, this is a limitation from the operating system.

If you are just saving the figures to disk you can use a non-gui backend using matplotlib.use('Agg'), and each process can paint its figure in memory and save it to disk using plt.savefig

the plot appears blank because each process has its own memory, and ax.plot is only modifying the copy of ax that exist in the child process memory, which the child process can show if you modify plot as follows.

import matplotlib.pyplot as plt
import numpy as np
from multiprocessing import Pool
from functools import partial

def plot(ax, x, y):
    ax.plot(x, y)
    plt.show()


if __name__ == "__main__":
    x = np.linspace(0, 2 * np.pi, 100)
    y_arr = [np.sin(x), np.cos(x)]
    fig, ax = plt.subplots()
    with Pool(5) as p:
        p.map(partial(plot, ax, x), y_arr)
    plt.show()

your get 3 windows, the first 2 are showing each child’s copy of ax and the third window is blank as it is showing the parent’s copy of ax which is unmodified.

Answered By: Ahmed AEK