How do I save print statements when running a program in SLURM?
Question:
I am running a Python code that contains print statements via SLURM. Normally when I run the Python code directly via “python program.py” the print statements appear in the terminal. When I run my program via SLURM, as expected the print statements do not appear in the terminal. How can I save the print statements to a file so I can check them as the program is running? Below is my submission script that I submit via “sbatch submit.sh”. Notice that I’ve already tried two methods to write the output either to test1.out or test2.out. Please let me know where I’m going wrong!
#!/bin/bash
#SBATCH -J mysubmission
#SBATCH -p New
#SBATCH -n 1
#SBATCH -t 23:59:00
#SBATCH -o test1.out
module load gnu python
python program.py > test2.out
Answers:
By default, print
in Python is buffered, meaning that it does not write to files or stdout immediately, and needs to be ‘flushed’ to force the writing to stdout immediately.
See this question for available options.
The simplest option is to start the Python interpreter with the -u
option.
From the python
man page:
-u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
Note that there is internal buffering in xreadlines(), readlines() and
file-object iterators (“for line in sys.stdin”) which is not
influenced by this option. To work around this, you will want to use
“sys.stdin.readline()” inside a “while 1:” loop.
You can use:
python -u program.py > test2.out
And all your output will be saved to test2.out file.
You can put
sys.stdout.flush()
after the print statements of interest. Or since python 3.3
print("hey", flush=True)
is an option. Using unbuffered IO (with python -u
) could reduce performance if you print frequently, especially on a shared filesystem.
Above solutions did not yield live output for me. (You need to flush print and sync the file change.) Here is a code that you can simply prepend to your python script.
#!/usr/bin/env python3.7
import sys
import builtins
import os
sys.stdout = open("stdout.txt", "w", buffering=1)
def print(text):
builtins.print(text)
os.fsync(sys.stdout)
print("This is immediately written to stdout.txt")
That’s it.
Explanation
Redirect stdout and write the file buffer after every newline.
sys.stdout = open("stdout.txt", "w", buffering=1)
Overwrite the print function and sync the flushed file also on the operating system side.
def print(text):
builtins.print(text)
os.fsync(sys.stdout)
I am running a Python code that contains print statements via SLURM. Normally when I run the Python code directly via “python program.py” the print statements appear in the terminal. When I run my program via SLURM, as expected the print statements do not appear in the terminal. How can I save the print statements to a file so I can check them as the program is running? Below is my submission script that I submit via “sbatch submit.sh”. Notice that I’ve already tried two methods to write the output either to test1.out or test2.out. Please let me know where I’m going wrong!
#!/bin/bash
#SBATCH -J mysubmission
#SBATCH -p New
#SBATCH -n 1
#SBATCH -t 23:59:00
#SBATCH -o test1.out
module load gnu python
python program.py > test2.out
By default, print
in Python is buffered, meaning that it does not write to files or stdout immediately, and needs to be ‘flushed’ to force the writing to stdout immediately.
See this question for available options.
The simplest option is to start the Python interpreter with the -u
option.
From the python
man page:
-u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
Note that there is internal buffering in xreadlines(), readlines() and
file-object iterators (“for line in sys.stdin”) which is not
influenced by this option. To work around this, you will want to use
“sys.stdin.readline()” inside a “while 1:” loop.
You can use:
python -u program.py > test2.out
And all your output will be saved to test2.out file.
You can put
sys.stdout.flush()
after the print statements of interest. Or since python 3.3
print("hey", flush=True)
is an option. Using unbuffered IO (with python -u
) could reduce performance if you print frequently, especially on a shared filesystem.
Above solutions did not yield live output for me. (You need to flush print and sync the file change.) Here is a code that you can simply prepend to your python script.
#!/usr/bin/env python3.7
import sys
import builtins
import os
sys.stdout = open("stdout.txt", "w", buffering=1)
def print(text):
builtins.print(text)
os.fsync(sys.stdout)
print("This is immediately written to stdout.txt")
That’s it.
Explanation
Redirect stdout and write the file buffer after every newline.
sys.stdout = open("stdout.txt", "w", buffering=1)
Overwrite the print function and sync the flushed file also on the operating system side.
def print(text):
builtins.print(text)
os.fsync(sys.stdout)