Matplotlib runs out of memory when plotting in a loop
Question:
I have a fairly simple plotting routine that looks like this:
from __future__ import division
import datetime
import matplotlib
matplotlib.use('Agg')
from matplotlib.pyplot import figure, plot, show, legend, close, savefig, rcParams
import numpy
from globalconstants import *
def plotColumns(columnNumbers, t, out, showFig=False, filenamePrefix=None, saveFig=True, saveThumb=True):
lineProps = ['b', 'r', 'g', 'c', 'm', 'y', 'k', 'b--', 'r--', 'g--', 'c--', 'm--', 'y--', 'k--', 'g--', 'b.-', 'r.-', 'g.-', 'c.-', 'm.-', 'y.-', 'k.-']
rcParams['figure.figsize'] = (13,11)
for i in columnNumbers:
plot(t, out[:,i], lineProps[i])
legendStrings = list(numpy.zeros(NUMCOMPONENTS))
legendStrings[GLUCOSE] = 'GLUCOSE'
legendStrings[CELLULOSE] = 'CELLULOSE'
legendStrings[STARCH] = 'STARCH'
legendStrings[ACETATE] = 'ACETATE'
legendStrings[BUTYRATE] = 'BUTYRATE'
legendStrings[SUCCINATE] = 'SUCCINATE'
legendStrings[HYDROGEN] = 'HYDROGEN'
legendStrings[PROPIONATE] = 'PROPIONATE'
legendStrings[METHANE] = "METHANE"
legendStrings[RUMINOCOCCUS] = 'RUMINOCOCCUS'
legendStrings[METHANOBACTERIUM] = "METHANOBACTERIUM"
legendStrings[BACTEROIDES] = 'BACTEROIDES'
legendStrings[SELENOMONAS] = 'SELENOMONAS'
legendStrings[CLOSTRIDIUM] = 'CLOSTRIDIUM'
legendStrings = [legendStrings[i] for i in columnNumbers]
legend(legendStrings, loc='best')
dt = datetime.datetime.now()
dtAsString = dt.strftime('%d-%m-%Y_%H-%M-%S')
if filenamePrefix is None:
filenamePrefix = ''
if filenamePrefix != '' and filenamePrefix[-1] != '_':
filenamePrefix += '_'
if saveFig:
savefig(filenamePrefix+dtAsString+'.eps')
if saveThumb:
savefig(filenamePrefix+dtAsString+'.png', dpi=300)
if showFig: f.show()
close('all')
When I plot this in single iterations, it works fine. However, the moment I put it in a loop, matplotlib throws a hissy fit…
Traceback (most recent call last):
File "c4hm_param_variation_h2_conc.py", line 148, in <module>
plotColumns(columnNumbers, timeVector, out, showFig=False, filenamePrefix='c
4hm_param_variation_h2_conc_'+str(hydrogen_conc), saveFig=False, saveThumb=True)
File "D:phdprojectalexander paperpythonv3plotcolumns.py", line 48, in plo
tColumns
savefig(filenamePrefix+dtAsString+'.png', dpi=300)
File "C:Python25libsite-packagesmatplotlibpyplot.py", line 356, in savefi
g
return fig.savefig(*args, **kwargs)
File "C:Python25libsite-packagesmatplotlibfigure.py", line 1032, in savef
ig
self.canvas.print_figure(*args, **kwargs)
File "C:Python25libsite-packagesmatplotlibbackend_bases.py", line 1476, i
n print_figure
**kwargs)
File "C:Python25libsite-packagesmatplotlibbackendsbackend_agg.py", line
358, in print_png
FigureCanvasAgg.draw(self)
File "C:Python25libsite-packagesmatplotlibbackendsbackend_agg.py", line
314, in draw
self.figure.draw(self.renderer)
File "C:Python25libsite-packagesmatplotlibartist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:Python25libsite-packagesmatplotlibfigure.py", line 773, in draw
for a in self.axes: a.draw(renderer)
File "C:Python25libsite-packagesmatplotlibartist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:Python25libsite-packagesmatplotlibaxes.py", line 1735, in draw
a.draw(renderer)
File "C:Python25libsite-packagesmatplotlibartist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:Python25libsite-packagesmatplotliblegend.py", line 374, in draw
bbox = self._legend_box.get_window_extent(renderer)
File "C:Python25libsite-packagesmatplotliboffsetbox.py", line 209, in get
_window_extent
px, py = self.get_offset(w, h, xd, yd)
File "C:Python25libsite-packagesmatplotliboffsetbox.py", line 162, in get
_offset
return self._offset(width, height, xdescent, ydescent)
File "C:Python25libsite-packagesmatplotliblegend.py", line 360, in findof
fset
return _findoffset(width, height, xdescent, ydescent, renderer)
File "C:Python25libsite-packagesmatplotliblegend.py", line 325, in _findo
ffset_best
ox, oy = self._find_best_position(width, height, renderer)
File "C:Python25libsite-packagesmatplotliblegend.py", line 817, in _find_
best_position
verts, bboxes, lines = self._auto_legend_data()
File "C:Python25libsite-packagesmatplotliblegend.py", line 669, in _auto_
legend_data
tpath = trans.transform_path(path)
File "C:Python25libsite-packagesmatplotlibtransforms.py", line 1911, in t
ransform_path
self._a.transform_path(path))
File "C:Python25libsite-packagesmatplotlibtransforms.py", line 1122, in t
ransform_path
return Path(self.transform(path.vertices), path.codes,
File "C:Python25libsite-packagesmatplotlibtransforms.py", line 1402, in t
ransform
return affine_transform(points, mtx)
MemoryError: Could not allocate memory for path
This happens on iteration 2 (counting from 1), if that makes a difference. The code is running on Windows XP 32-bit with python 2.5 and matplotlib 0.99.1, numpy 1.3.0 and scipy 0.7.1.
EDIT: The code has now been updated to reflect the fact that the crash actually occurs at the call to legend(). Commenting that call out solves the problem, though obviously, I would still like to be able to put a legend on my graphs…
Answers:
Is each loop supposed to generate a new figure? I don’t see you closing it or creating a new figure instance from loop to loop.
This call will clear the current figure after you save it at the end of the loop:
pyplot.clf()
I’d refactor, though, and make your code more OO and create a new figure instance on each loop:
from matplotlib import pyplot
while True:
fig = pyplot.figure()
ax = fig.add_subplot(111)
ax.plot(x,y)
ax.legend(legendStrings, loc = 'best')
fig.savefig('himom.png')
# etc....
I’ve also run into this error. what seems to have fixed it is
while True:
fig = pyplot.figure()
ax = fig.add_subplot(111)
ax.plot(x,y)
ax.legend(legendStrings, loc = 'best')
fig.savefig('himom.png')
#new bit here
pylab.close(fig) #where f is the figure
running my loop stably now with fluctuating memory but no consistant increase
Answer from ninjasmith worked for me too – pyplot.close() enabled my loops to work.
From the pyplot tutorial, Working with multiple figures and axes:
You can clear the current figure with clf() and the current
axes with cla(). If you find this statefulness, annoying, don’t
despair, this is just a thin stateful wrapper around an object
oriented API, which you can use instead (see Artist tutorial)
If you are making a long sequence of figures, you need to be aware of
one more thing: the memory required for a figure is not completely
released until the figure is explicitly closed with close(). Deleting
all references to the figure, and/or using the window manager to kill
the window in which the figure appears on the screen, is not enough,
because pyplot maintains internal references until close() is called.
In my case, matplotlib version 3.5.0, As Hui Liu san says,
Following method can keep memory usage low
import matplotlib
print(matplotlib.__version__) #'3.5.0'
import matplotlib.pyplot as plt
plt.savefig('your.png')
# Add both in this order for keeping memory usage low
plt.clf()
plt.close()
— Added —-
With matplotlib version ‘3.7.2’,
Memory continued to increase even with the above method.
By adding the following code as described on this site(http://datasideoflife.com/?p=1443) and by Innuendo,
the memory increase has been eliminated.
import matplotlib
matplotlib.use('Agg') # for avoiding memory leak
I had a similar issue when I was using it from jupyter, putting plt.clf() and plt.close() in the loop did not work.
But this helped:
import matplotlib
matplotlib.use('Agg')
This disables interactive backend for matplotlib.
I have a fairly simple plotting routine that looks like this:
from __future__ import division
import datetime
import matplotlib
matplotlib.use('Agg')
from matplotlib.pyplot import figure, plot, show, legend, close, savefig, rcParams
import numpy
from globalconstants import *
def plotColumns(columnNumbers, t, out, showFig=False, filenamePrefix=None, saveFig=True, saveThumb=True):
lineProps = ['b', 'r', 'g', 'c', 'm', 'y', 'k', 'b--', 'r--', 'g--', 'c--', 'm--', 'y--', 'k--', 'g--', 'b.-', 'r.-', 'g.-', 'c.-', 'm.-', 'y.-', 'k.-']
rcParams['figure.figsize'] = (13,11)
for i in columnNumbers:
plot(t, out[:,i], lineProps[i])
legendStrings = list(numpy.zeros(NUMCOMPONENTS))
legendStrings[GLUCOSE] = 'GLUCOSE'
legendStrings[CELLULOSE] = 'CELLULOSE'
legendStrings[STARCH] = 'STARCH'
legendStrings[ACETATE] = 'ACETATE'
legendStrings[BUTYRATE] = 'BUTYRATE'
legendStrings[SUCCINATE] = 'SUCCINATE'
legendStrings[HYDROGEN] = 'HYDROGEN'
legendStrings[PROPIONATE] = 'PROPIONATE'
legendStrings[METHANE] = "METHANE"
legendStrings[RUMINOCOCCUS] = 'RUMINOCOCCUS'
legendStrings[METHANOBACTERIUM] = "METHANOBACTERIUM"
legendStrings[BACTEROIDES] = 'BACTEROIDES'
legendStrings[SELENOMONAS] = 'SELENOMONAS'
legendStrings[CLOSTRIDIUM] = 'CLOSTRIDIUM'
legendStrings = [legendStrings[i] for i in columnNumbers]
legend(legendStrings, loc='best')
dt = datetime.datetime.now()
dtAsString = dt.strftime('%d-%m-%Y_%H-%M-%S')
if filenamePrefix is None:
filenamePrefix = ''
if filenamePrefix != '' and filenamePrefix[-1] != '_':
filenamePrefix += '_'
if saveFig:
savefig(filenamePrefix+dtAsString+'.eps')
if saveThumb:
savefig(filenamePrefix+dtAsString+'.png', dpi=300)
if showFig: f.show()
close('all')
When I plot this in single iterations, it works fine. However, the moment I put it in a loop, matplotlib throws a hissy fit…
Traceback (most recent call last):
File "c4hm_param_variation_h2_conc.py", line 148, in <module>
plotColumns(columnNumbers, timeVector, out, showFig=False, filenamePrefix='c
4hm_param_variation_h2_conc_'+str(hydrogen_conc), saveFig=False, saveThumb=True)
File "D:phdprojectalexander paperpythonv3plotcolumns.py", line 48, in plo
tColumns
savefig(filenamePrefix+dtAsString+'.png', dpi=300)
File "C:Python25libsite-packagesmatplotlibpyplot.py", line 356, in savefi
g
return fig.savefig(*args, **kwargs)
File "C:Python25libsite-packagesmatplotlibfigure.py", line 1032, in savef
ig
self.canvas.print_figure(*args, **kwargs)
File "C:Python25libsite-packagesmatplotlibbackend_bases.py", line 1476, i
n print_figure
**kwargs)
File "C:Python25libsite-packagesmatplotlibbackendsbackend_agg.py", line
358, in print_png
FigureCanvasAgg.draw(self)
File "C:Python25libsite-packagesmatplotlibbackendsbackend_agg.py", line
314, in draw
self.figure.draw(self.renderer)
File "C:Python25libsite-packagesmatplotlibartist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:Python25libsite-packagesmatplotlibfigure.py", line 773, in draw
for a in self.axes: a.draw(renderer)
File "C:Python25libsite-packagesmatplotlibartist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:Python25libsite-packagesmatplotlibaxes.py", line 1735, in draw
a.draw(renderer)
File "C:Python25libsite-packagesmatplotlibartist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:Python25libsite-packagesmatplotliblegend.py", line 374, in draw
bbox = self._legend_box.get_window_extent(renderer)
File "C:Python25libsite-packagesmatplotliboffsetbox.py", line 209, in get
_window_extent
px, py = self.get_offset(w, h, xd, yd)
File "C:Python25libsite-packagesmatplotliboffsetbox.py", line 162, in get
_offset
return self._offset(width, height, xdescent, ydescent)
File "C:Python25libsite-packagesmatplotliblegend.py", line 360, in findof
fset
return _findoffset(width, height, xdescent, ydescent, renderer)
File "C:Python25libsite-packagesmatplotliblegend.py", line 325, in _findo
ffset_best
ox, oy = self._find_best_position(width, height, renderer)
File "C:Python25libsite-packagesmatplotliblegend.py", line 817, in _find_
best_position
verts, bboxes, lines = self._auto_legend_data()
File "C:Python25libsite-packagesmatplotliblegend.py", line 669, in _auto_
legend_data
tpath = trans.transform_path(path)
File "C:Python25libsite-packagesmatplotlibtransforms.py", line 1911, in t
ransform_path
self._a.transform_path(path))
File "C:Python25libsite-packagesmatplotlibtransforms.py", line 1122, in t
ransform_path
return Path(self.transform(path.vertices), path.codes,
File "C:Python25libsite-packagesmatplotlibtransforms.py", line 1402, in t
ransform
return affine_transform(points, mtx)
MemoryError: Could not allocate memory for path
This happens on iteration 2 (counting from 1), if that makes a difference. The code is running on Windows XP 32-bit with python 2.5 and matplotlib 0.99.1, numpy 1.3.0 and scipy 0.7.1.
EDIT: The code has now been updated to reflect the fact that the crash actually occurs at the call to legend(). Commenting that call out solves the problem, though obviously, I would still like to be able to put a legend on my graphs…
Is each loop supposed to generate a new figure? I don’t see you closing it or creating a new figure instance from loop to loop.
This call will clear the current figure after you save it at the end of the loop:
pyplot.clf()
I’d refactor, though, and make your code more OO and create a new figure instance on each loop:
from matplotlib import pyplot
while True:
fig = pyplot.figure()
ax = fig.add_subplot(111)
ax.plot(x,y)
ax.legend(legendStrings, loc = 'best')
fig.savefig('himom.png')
# etc....
I’ve also run into this error. what seems to have fixed it is
while True:
fig = pyplot.figure()
ax = fig.add_subplot(111)
ax.plot(x,y)
ax.legend(legendStrings, loc = 'best')
fig.savefig('himom.png')
#new bit here
pylab.close(fig) #where f is the figure
running my loop stably now with fluctuating memory but no consistant increase
Answer from ninjasmith worked for me too – pyplot.close() enabled my loops to work.
From the pyplot tutorial, Working with multiple figures and axes:
You can clear the current figure with
clf()and the current
axes withcla(). If you find this statefulness, annoying, don’t
despair, this is just a thin stateful wrapper around an object
oriented API, which you can use instead (see Artist tutorial)If you are making a long sequence of figures, you need to be aware of
one more thing: the memory required for a figure is not completely
released until the figure is explicitly closed withclose(). Deleting
all references to the figure, and/or using the window manager to kill
the window in which the figure appears on the screen, is not enough,
because pyplot maintains internal references untilclose()is called.
In my case, matplotlib version 3.5.0, As Hui Liu san says,
Following method can keep memory usage low
import matplotlib
print(matplotlib.__version__) #'3.5.0'
import matplotlib.pyplot as plt
plt.savefig('your.png')
# Add both in this order for keeping memory usage low
plt.clf()
plt.close()
— Added —-
With matplotlib version ‘3.7.2’,
Memory continued to increase even with the above method.
By adding the following code as described on this site(http://datasideoflife.com/?p=1443) and by Innuendo,
the memory increase has been eliminated.
import matplotlib
matplotlib.use('Agg') # for avoiding memory leak
I had a similar issue when I was using it from jupyter, putting plt.clf() and plt.close() in the loop did not work.
But this helped:
import matplotlib
matplotlib.use('Agg')
This disables interactive backend for matplotlib.