Setting stacksize in a python script
Question:
I am converting a csh script to a python script. The script calls a memory-intensive executable which requires a very large stack, so the csh script sets the stacksize to unlimited:
limit stacksize unlimited
When I try to reproduce this script in python, I execute them in a very naive manner, using os.system, e.g.:
os.system('some_executable')
But I do not know how to tell the OS to run these executables with unlimited stacksize. Is there a way to specify stacksize for calls within a python script? Is there some low-level system call that I should be using? And is there a module (similar to shutil) which controls this?
Answers:
You’re looking for the Python setrlimit interface, resource.RLIMIT_STACK.
Note that standard users cannot raise their hard limits, only root (well, a process with the CAP_SYS_RESOURCE capability (see capabilities(7)) processes can raise their limits; so you may need to use the PAM pam_limits(8) limits.conf(5) file to raise the hard limits for the users in question.
You can just use the (u)limit command of your shell, if you want:
os.system('ulimit -s unlimited; some_executable')
Or (probably better) use resource.setrlimit:
resource.setrlimit(resource.RLIMIT_STACK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))
You can alter the stack size of the current process via threading.stack_size(), but I don’t know if that will be correctly inherited by subprocesses. That interface also requires a specific stack size – “unlimited” isn’t an option.
I have good experience with the following code. It doesn’t require any special user permissions:
import resource, sys
resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))
sys.setrecursionlimit(10**6)
It does however not seem to work with pypy.
If resource.setrlimit doesn’t work, you can also try using threads:
sys.setrecursionlimit(10**6)
import threading
threading.stack_size(2**26)
threading.Thread(target=main).start()
I am converting a csh script to a python script. The script calls a memory-intensive executable which requires a very large stack, so the csh script sets the stacksize to unlimited:
limit stacksize unlimited
When I try to reproduce this script in python, I execute them in a very naive manner, using os.system, e.g.:
os.system('some_executable')
But I do not know how to tell the OS to run these executables with unlimited stacksize. Is there a way to specify stacksize for calls within a python script? Is there some low-level system call that I should be using? And is there a module (similar to shutil) which controls this?
You’re looking for the Python setrlimit interface, resource.RLIMIT_STACK.
Note that standard users cannot raise their hard limits, only root (well, a process with the CAP_SYS_RESOURCE capability (see capabilities(7)) processes can raise their limits; so you may need to use the PAM pam_limits(8) limits.conf(5) file to raise the hard limits for the users in question.
You can just use the (u)limit command of your shell, if you want:
os.system('ulimit -s unlimited; some_executable')
Or (probably better) use resource.setrlimit:
resource.setrlimit(resource.RLIMIT_STACK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))
You can alter the stack size of the current process via threading.stack_size(), but I don’t know if that will be correctly inherited by subprocesses. That interface also requires a specific stack size – “unlimited” isn’t an option.
I have good experience with the following code. It doesn’t require any special user permissions:
import resource, sys
resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))
sys.setrecursionlimit(10**6)
It does however not seem to work with pypy.
If resource.setrlimit doesn’t work, you can also try using threads:
sys.setrecursionlimit(10**6)
import threading
threading.stack_size(2**26)
threading.Thread(target=main).start()