Right way to clean up a temporary folder in Python class
Question:
I am creating a class in which I want to generate a temporary workspace of folders that will persist for the life of the object and then be removed. I am using tempfile.mkdtemp() in the def __init__ to create the space, but I have read that I can’t rely on __del__ being called.
I am wanting something like this:
class MyClass:
def __init__(self):
self.tempfolder = tempfile.mkdtemp()
def ... #other stuff
def __del__(self):
if os.path.exists(self.tempfolder): shutil.rmtree(self.tempfolder)
Is there another/better way to handle this clean up? I was reading about with, but it appears to only be helpful within a function.
Answers:
Caveat: you can never guarantee that the temp folder will be deleted, because the user could always hard kill your process and then it can’t run anything else.
That said, do
temp_dir = tempfile.mkdtemp()
try:
<some code>
finally:
shutil.rmtree(temp_dir)
Since this is a very common operation, Python has a special way to encapsulate “do something, execute code, clean up”: a context manager. You can write your own as follows:
@contextlib.contextmanager
def make_temp_directory():
temp_dir = tempfile.mkdtemp()
try:
yield temp_dir
finally:
shutil.rmtree(temp_dir)
and use it as
with make_temp_directory() as temp_dir:
<some code>
(Note that this uses the @contextlib.contextmanager shortcut to make a context manager. If you want to implement one the original way, you need to make a custom class with __enter__ and __exit__ methods; the __enter__ would create and return the temp directory and the __exit__ delete it.
As stated by Bluewind you have to make sure to wrap the yield portion of the context manager inside of a try: finally statement otherwise any exceptions will not really be handled correctly inside of the context manager.
From Python 2.7 docs
At the point where the generator yields, the block nested in the with statement is executed. The generator is then resumed after the block is exited. If an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred. Thus, you can use a try…except…finally statement to trap the error (if any), or ensure that some cleanup takes place. If an exception is trapped merely in order to log it or to perform some action (rather than to suppress it entirely), the generator must reraise that exception. Otherwise the generator context manager will indicate to the with statement that the exception has been handled, and execution will resume with the statement immediately following the with statement.
Also if you are using Python 3.2+ you should check out this little gem which has all of the above wrapped up nicely for you
tempfile.TemporaryDirectory(suffix=”, prefix=’tmp’, dir=None)
This function creates a temporary directory using mkdtemp() (the supplied arguments are passed directly to the underlying function). The resulting object can be used as a context manager (see With Statement Context Managers). On completion of the context (or destruction of the temporary directory object), the newly created temporary directory and all its contents are removed from the filesystem.
The directory name can be retrieved from the name attribute of the returned object.
The directory can be explicitly cleaned up by calling the cleanup() method.
New in version 3.2.
A nice way to deal with temporary files and directories is via a context manager. This is how you can use tempfile.TemporaryFile or tempfile.NamedTemporaryFile — once you’ve exited the with statement (via normal exit, return, exception, or anything else) the file/directory and it’s contents will be removed from the filesystem.
For Python 3.2+, this is built in as tempfile.TemporaryDirectory:
import tempfile
with tempfile.TemporaryDirectory() as temp_dir:
... do stuff ...
For earlier Python versions you can easily create your own context manager to do exactly the same thing. The differences here from @katrielalex answer are the passing of args to mkdtemp() and the try/finally block to make sure the directory gets cleaned up if an exception is raised.
import contextlib
import shutil
@contextlib.contextmanager
def temporary_directory(*args, **kwargs):
d = tempfile.mkdtemp(*args, **kwargs)
try:
yield d
finally:
shutil.rmtree(d)
# use it
with temporary_directory() as temp_dir:
... do stuff ...
Note that if your process is hard-killed (eg. kill -9) then the directories won’t get cleaned up.
Another alternative using contextlib is to make your object closable, and use the closing context manager.
class MyClass:
def __init__(self):
self.tempfolder = tempfile.mkdtemp()
def do_stuff():
pass
def close(self):
if os.path.exists(self.tempfolder):
shutil.rmtree(self.tempfolder)
Then with the context manager:
from contextlib import closing
with closing(MyClass()) as my_object:
my_object.do_stuff()
Other answers have noted that you can use a contextmanager or require your users to explicitly call some type of clean up function. These are great to do if you can. However, sometimes there’s no where to hook up this cleanup because you are inside a large application and you are nested multiple layers down, and no one above you has cleanup methods or context managers.
In that case, you can use atexit: https://docs.python.org/2/library/atexit.html
import atexit
class MyClass:
def __init__(self):
self.tempfolder = tempfile.mkdtemp()
atexit.register(shutil.rmtree, self.tempfolder)
def ... #other stuff
I somewhat experimented with this and I am quite confident that, if you cannot use a context manager, the best solution as of this posting is:
class MyClass(object):
def __init__(self):
self.tempfolder = tempfile.TemporaryDirectory()
…
def __del__(self):
self.tempfolder.cleanup()
(Some conditional in __del__ may be reasonable if you cannot ensure __init__ to be called.)
Now, except from using the newer TemporaryDirectory instead of mkdtemp, this is not much different from what you were doing. Why do I still think this is the best you can do? Well, I tested several scenarios of program exit and similar (all on Linux) and:
-
I could not find a scenario where the temporary folder was not deleted even though I would expect that Python could decide that the respective instance of MyClass was not needed anymore. Automatic deletion happens as early as Python’s garbage collecting heuristics allow.
-
You can “help” the garbage collector with del myinstance and gc.collect(). Mind that del only decreases the reference count, so this does not ensure that garbage collection can happen and __del__ is called.
-
If you really want to ensure deletion (of the temporary directory), you can explicitly call myinstance.__del__(). If you can do this, you can probably also make MyClass itself a context manager.
-
The only case where the temporary folder persisted was when I hard-killed Python from the operating system – in which case I do not see how any solution within Python would work.
-
atexit (as suggested, e.g., by this answer) does not improve the situation: Either deletion happens without atexit anyway, or it does not happen even with atexit.
I am creating a class in which I want to generate a temporary workspace of folders that will persist for the life of the object and then be removed. I am using tempfile.mkdtemp() in the def __init__ to create the space, but I have read that I can’t rely on __del__ being called.
I am wanting something like this:
class MyClass:
def __init__(self):
self.tempfolder = tempfile.mkdtemp()
def ... #other stuff
def __del__(self):
if os.path.exists(self.tempfolder): shutil.rmtree(self.tempfolder)
Is there another/better way to handle this clean up? I was reading about with, but it appears to only be helpful within a function.
Caveat: you can never guarantee that the temp folder will be deleted, because the user could always hard kill your process and then it can’t run anything else.
That said, do
temp_dir = tempfile.mkdtemp()
try:
<some code>
finally:
shutil.rmtree(temp_dir)
Since this is a very common operation, Python has a special way to encapsulate “do something, execute code, clean up”: a context manager. You can write your own as follows:
@contextlib.contextmanager
def make_temp_directory():
temp_dir = tempfile.mkdtemp()
try:
yield temp_dir
finally:
shutil.rmtree(temp_dir)
and use it as
with make_temp_directory() as temp_dir:
<some code>
(Note that this uses the @contextlib.contextmanager shortcut to make a context manager. If you want to implement one the original way, you need to make a custom class with __enter__ and __exit__ methods; the __enter__ would create and return the temp directory and the __exit__ delete it.
As stated by Bluewind you have to make sure to wrap the yield portion of the context manager inside of a try: finally statement otherwise any exceptions will not really be handled correctly inside of the context manager.
From Python 2.7 docs
At the point where the generator yields, the block nested in the with statement is executed. The generator is then resumed after the block is exited. If an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred. Thus, you can use a try…except…finally statement to trap the error (if any), or ensure that some cleanup takes place. If an exception is trapped merely in order to log it or to perform some action (rather than to suppress it entirely), the generator must reraise that exception. Otherwise the generator context manager will indicate to the with statement that the exception has been handled, and execution will resume with the statement immediately following the with statement.
Also if you are using Python 3.2+ you should check out this little gem which has all of the above wrapped up nicely for you
tempfile.TemporaryDirectory(suffix=”, prefix=’tmp’, dir=None)
This function creates a temporary directory using mkdtemp() (the supplied arguments are passed directly to the underlying function). The resulting object can be used as a context manager (see With Statement Context Managers). On completion of the context (or destruction of the temporary directory object), the newly created temporary directory and all its contents are removed from the filesystem.
The directory name can be retrieved from the name attribute of the returned object.
The directory can be explicitly cleaned up by calling the cleanup() method.
New in version 3.2.
A nice way to deal with temporary files and directories is via a context manager. This is how you can use tempfile.TemporaryFile or tempfile.NamedTemporaryFile — once you’ve exited the with statement (via normal exit, return, exception, or anything else) the file/directory and it’s contents will be removed from the filesystem.
For Python 3.2+, this is built in as tempfile.TemporaryDirectory:
import tempfile
with tempfile.TemporaryDirectory() as temp_dir:
... do stuff ...
For earlier Python versions you can easily create your own context manager to do exactly the same thing. The differences here from @katrielalex answer are the passing of args to mkdtemp() and the try/finally block to make sure the directory gets cleaned up if an exception is raised.
import contextlib
import shutil
@contextlib.contextmanager
def temporary_directory(*args, **kwargs):
d = tempfile.mkdtemp(*args, **kwargs)
try:
yield d
finally:
shutil.rmtree(d)
# use it
with temporary_directory() as temp_dir:
... do stuff ...
Note that if your process is hard-killed (eg. kill -9) then the directories won’t get cleaned up.
Another alternative using contextlib is to make your object closable, and use the closing context manager.
class MyClass:
def __init__(self):
self.tempfolder = tempfile.mkdtemp()
def do_stuff():
pass
def close(self):
if os.path.exists(self.tempfolder):
shutil.rmtree(self.tempfolder)
Then with the context manager:
from contextlib import closing
with closing(MyClass()) as my_object:
my_object.do_stuff()
Other answers have noted that you can use a contextmanager or require your users to explicitly call some type of clean up function. These are great to do if you can. However, sometimes there’s no where to hook up this cleanup because you are inside a large application and you are nested multiple layers down, and no one above you has cleanup methods or context managers.
In that case, you can use atexit: https://docs.python.org/2/library/atexit.html
import atexit
class MyClass:
def __init__(self):
self.tempfolder = tempfile.mkdtemp()
atexit.register(shutil.rmtree, self.tempfolder)
def ... #other stuff
I somewhat experimented with this and I am quite confident that, if you cannot use a context manager, the best solution as of this posting is:
class MyClass(object):
def __init__(self):
self.tempfolder = tempfile.TemporaryDirectory()
…
def __del__(self):
self.tempfolder.cleanup()
(Some conditional in __del__ may be reasonable if you cannot ensure __init__ to be called.)
Now, except from using the newer TemporaryDirectory instead of mkdtemp, this is not much different from what you were doing. Why do I still think this is the best you can do? Well, I tested several scenarios of program exit and similar (all on Linux) and:
-
I could not find a scenario where the temporary folder was not deleted even though I would expect that Python could decide that the respective instance of
MyClasswas not needed anymore. Automatic deletion happens as early as Python’s garbage collecting heuristics allow. -
You can “help” the garbage collector with
del myinstanceandgc.collect(). Mind thatdelonly decreases the reference count, so this does not ensure that garbage collection can happen and__del__is called. -
If you really want to ensure deletion (of the temporary directory), you can explicitly call
myinstance.__del__(). If you can do this, you can probably also makeMyClassitself a context manager. -
The only case where the temporary folder persisted was when I hard-killed Python from the operating system – in which case I do not see how any solution within Python would work.
-
atexit(as suggested, e.g., by this answer) does not improve the situation: Either deletion happens withoutatexitanyway, or it does not happen even withatexit.