Syntax for submitting a qsub job without an actual job file?
Question:
I would like to submit qsub jobs on the fly without creating discrete job files. So, let’s say I have a python script called “get_time.py” that simply reports the time. Instead of making a submission script like this:
cat>job.sub<<eof
#PBS -l walltime=1:00:00
cd $PBS_O_WORKDIR
get_time.py
eof
…and then submitting the job: qsub job.sub
I would like to be able to bypass the file creation step, and I’d image the construct would be something like this:
qsub -d . -e get_time.py
where -e is my imaginary parameter that tells qsub that the following is code to be sent to the scheduler, instead of using a discrete submission file.
Am I crazy? It seems like there should already be a simple solution for this, but I couldn’t find it anywhere. Thanks for any suggestions!
Answers:
You can pass the name of any executable script/file to qsub after you’ve provided all your options (-d ., etc.). Any arguments that come after the script name are treated as arguments for that script. For example, if I had the script test.py:
#!/usr/bin/python
from sys import argv
script, param = argv
print param
Then I can run
qsub test.py 2
and the output to my log will be 2.
Two important things to note:
- Your Python script must be executable (e.g.
chmod +x test.py).
- Your Python script must have a shebang (e.g.
#!/usr/bin/python).
Provided get_time.py is executable, and the shebang is correct, you should be able to simply run
qsub -d . get_time.py
If that fails for either reason, you can use the following construct
qsub -d . <<< "python get_time.py"
I don’t understand the question but…
echo 'date' | qsub
is not working well for you?
If you are ok with writing your job in a HERE document, as in your example, you can pass that HERE doc directly to qsub without a temporary file:
qsub <<eof
#PBS -l walltime=1:00:00
cd $PBS_O_WORKDIR
get_time.py
eof
Also note that you need to escape the dollar sign in $PBS_O_WORKDIR or else it will be interpolated before submitting the job and end up as an empty string.
In grid engine, -b y specifies that the thing to be executed is a binary, not a script:
qsub -b y get_time.py
The easiest for me is to add the command(s) I want to run in an echo-statement and pipe it to qsub. For example, in this case:
echo "get_time.py" | qsub -d . -l walltime=1:00:00
I would like to submit qsub jobs on the fly without creating discrete job files. So, let’s say I have a python script called “get_time.py” that simply reports the time. Instead of making a submission script like this:
cat>job.sub<<eof
#PBS -l walltime=1:00:00
cd $PBS_O_WORKDIR
get_time.py
eof
…and then submitting the job: qsub job.sub
I would like to be able to bypass the file creation step, and I’d image the construct would be something like this:
qsub -d . -e get_time.py
where -e is my imaginary parameter that tells qsub that the following is code to be sent to the scheduler, instead of using a discrete submission file.
Am I crazy? It seems like there should already be a simple solution for this, but I couldn’t find it anywhere. Thanks for any suggestions!
You can pass the name of any executable script/file to qsub after you’ve provided all your options (-d ., etc.). Any arguments that come after the script name are treated as arguments for that script. For example, if I had the script test.py:
#!/usr/bin/python
from sys import argv
script, param = argv
print param
Then I can run
qsub test.py 2
and the output to my log will be 2.
Two important things to note:
- Your Python script must be executable (e.g.
chmod +x test.py). - Your Python script must have a shebang (e.g.
#!/usr/bin/python).
Provided get_time.py is executable, and the shebang is correct, you should be able to simply run
qsub -d . get_time.py
If that fails for either reason, you can use the following construct
qsub -d . <<< "python get_time.py"
I don’t understand the question but…
echo 'date' | qsub
is not working well for you?
If you are ok with writing your job in a HERE document, as in your example, you can pass that HERE doc directly to qsub without a temporary file:
qsub <<eof
#PBS -l walltime=1:00:00
cd $PBS_O_WORKDIR
get_time.py
eof
Also note that you need to escape the dollar sign in $PBS_O_WORKDIR or else it will be interpolated before submitting the job and end up as an empty string.
In grid engine, -b y specifies that the thing to be executed is a binary, not a script:
qsub -b y get_time.py
The easiest for me is to add the command(s) I want to run in an echo-statement and pipe it to qsub. For example, in this case:
echo "get_time.py" | qsub -d . -l walltime=1:00:00