Python Subprocess: Too Many Open Files
Question:
I am using subprocess to call another program and save its return values to a variable. This process is repeated in a loop, and after a few thousands times the program crashed with the following error:
Traceback (most recent call last):
File "./extract_pcgls.py", line 96, in <module>
SelfE.append( CalSelfEnergy(i) )
File "./extract_pcgls.py", line 59, in CalSelfEnergy
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1166, in _execute_child
errpipe_read, errpipe_write = _create_pipe()
OSError: [Errno 24] Too many open files
Any idea how to solve this issue is much appreciated!
Code supplied from comments:
cmd = "enerCHARMM.pl -parram=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
Answers:
You can try raising the open file limit of the OS:
ulimit -n 2048
I guess the problem was due to the fact that I was processing an open file with subprocess:
cmd = "enerCHARMM.pl -par param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
Here the cmd variable contain the name of a file that has just been created but not closed. Then the subprocess.Popen calls a system command on that file. After doing this for many times, the program crashed with that error message.
So the message I learned from this is
Close the file you have created, then process it
opens file in subprocess. It is blocking call.
ss=subprocess.Popen(tempFileName,shell=True)
ss.communicate()
In Mac OSX (El Capitan) See current configuration:
#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
Set open files value to 10K :
#ulimit -Sn 10000
Verify results:
#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 10000
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
A child process created by Popen() may inherit open file descriptors (a finite resource) from the parent. Use close_fds=True on POSIX (default since Python 3.2), to avoid it. Also, “PEP 0446 — Make newly created file descriptors non-inheritable” deals with some remaining issues (since Python 3.4).
As others have noted, raise the limit in /etc/security/limits.conf and also file descriptors was an issue for me personally, so I did
sudo sysctl -w fs.file-max=100000
And added to /etc/sysctl.conf:
fs.file-max = 100000
Reload with:
sudo sysctl -p
Also if you want to make sure that your process is not affected by anything else (which mine was), use
cat /proc/{process id}/limits
to find out what the actual limits of your process are, as for me the software running the python scripts also had its limits applied which have overridden the system wide settings.
Posting this answer here after resolving my particular issue with this error and hopefully it helps someone.
Maybe you are invoking the command multiple times. If so, each time you’re doing stdout=subprocess.PIPE. Between each call try doing p.stdout.close().
If you are working on Linux you can easily debug this problem
1 – Start the command that eventually will fail due Too Many Open Files in a terminal.
python -m module.script
2 – Let it run for a while (so it can start opening the actual files) and whenever you believe it has done so just press CTRL+Z so the process will be suspended. You will have an output with the process id.
^Z
[2] + 35245 suspended python -m module.script
35245 is your PID.
3 – Now you can check what files are actually opened and not closed.
ls -alht /proc/35245/fd/
In my case I was doing something very similar to the original post, but I was creating a temporarily file with tempfile.mkstemp() before adding some data and actually running the subprocess.Popen.
In this case you need to close the file twice, once for adding the information and the second one due mkstemp
fd, path = tempfile.mkstemp()
with open(path, "wb") as f:
f.write(bytes('my data', encoding='utf8'))
f.close() # this is one time
process = subprocess.Popen("my command that requires the previous file" ,[...])
os.close(fd) # this is second time and the one I missed
Use context managers instead:
cmd = "enerCHARMM.pl -param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
with subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True) as p:
out, err = p.communicate()
This will close p.stdout and p.stderr after the last line.
Related codes in Python: https://github.com/python/cpython/blob/208a7e957b812ad3b3733791845447677a704f3e/Lib/subprocess.py#L1031-L1038
Related document: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
I am using subprocess to call another program and save its return values to a variable. This process is repeated in a loop, and after a few thousands times the program crashed with the following error:
Traceback (most recent call last):
File "./extract_pcgls.py", line 96, in <module>
SelfE.append( CalSelfEnergy(i) )
File "./extract_pcgls.py", line 59, in CalSelfEnergy
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1166, in _execute_child
errpipe_read, errpipe_write = _create_pipe()
OSError: [Errno 24] Too many open files
Any idea how to solve this issue is much appreciated!
Code supplied from comments:
cmd = "enerCHARMM.pl -parram=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
You can try raising the open file limit of the OS:
ulimit -n 2048
I guess the problem was due to the fact that I was processing an open file with subprocess:
cmd = "enerCHARMM.pl -par param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
Here the cmd variable contain the name of a file that has just been created but not closed. Then the subprocess.Popen calls a system command on that file. After doing this for many times, the program crashed with that error message.
So the message I learned from this is
Close the file you have created, then process it
opens file in subprocess. It is blocking call.
ss=subprocess.Popen(tempFileName,shell=True)
ss.communicate()
In Mac OSX (El Capitan) See current configuration:
#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
Set open files value to 10K :
#ulimit -Sn 10000
Verify results:
#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 10000
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
A child process created by Popen() may inherit open file descriptors (a finite resource) from the parent. Use close_fds=True on POSIX (default since Python 3.2), to avoid it. Also, “PEP 0446 — Make newly created file descriptors non-inheritable” deals with some remaining issues (since Python 3.4).
As others have noted, raise the limit in /etc/security/limits.conf and also file descriptors was an issue for me personally, so I did
sudo sysctl -w fs.file-max=100000
And added to /etc/sysctl.conf:
fs.file-max = 100000
Reload with:
sudo sysctl -p
Also if you want to make sure that your process is not affected by anything else (which mine was), use
cat /proc/{process id}/limits
to find out what the actual limits of your process are, as for me the software running the python scripts also had its limits applied which have overridden the system wide settings.
Posting this answer here after resolving my particular issue with this error and hopefully it helps someone.
Maybe you are invoking the command multiple times. If so, each time you’re doing stdout=subprocess.PIPE. Between each call try doing p.stdout.close().
If you are working on Linux you can easily debug this problem
1 – Start the command that eventually will fail due Too Many Open Files in a terminal.
python -m module.script
2 – Let it run for a while (so it can start opening the actual files) and whenever you believe it has done so just press CTRL+Z so the process will be suspended. You will have an output with the process id.
^Z
[2] + 35245 suspended python -m module.script
35245 is your PID.
3 – Now you can check what files are actually opened and not closed.
ls -alht /proc/35245/fd/
In my case I was doing something very similar to the original post, but I was creating a temporarily file with tempfile.mkstemp() before adding some data and actually running the subprocess.Popen.
In this case you need to close the file twice, once for adding the information and the second one due mkstemp
fd, path = tempfile.mkstemp()
with open(path, "wb") as f:
f.write(bytes('my data', encoding='utf8'))
f.close() # this is one time
process = subprocess.Popen("my command that requires the previous file" ,[...])
os.close(fd) # this is second time and the one I missed
Use context managers instead:
cmd = "enerCHARMM.pl -param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
with subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True) as p:
out, err = p.communicate()
This will close p.stdout and p.stderr after the last line.
Related codes in Python: https://github.com/python/cpython/blob/208a7e957b812ad3b3733791845447677a704f3e/Lib/subprocess.py#L1031-L1038
Related document: https://docs.python.org/3/library/subprocess.html#subprocess.Popen