subprocess.call() arguments ignored when using shell=True w/ list
Question:
I am trying to get python’s subprocess.call method to accept some args commands through a list (consisting of a sequence of strings) as advised in the python documentation. To explore this behavior before putting it into my actual script, I opened up IPython, ran some commands involving different combinations of shell settings and args commands and got the following behavior:
In [3]: subprocess.call(['ls', '-%sl' %'a'])
total 320
drwxr-xr-x 20 Kohaugustine staff 680 Oct 15 16:55 .
drwxr-xr-x 5 Kohaugustine staff 170 Sep 12 17:16 ..
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 a.out
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 3 10:28 ex1-6
-rw-r--r--@ 1 Kohaugustine staff 204 Oct 3 10:28 ex1-6.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Oct 3 10:15 ex1-7
-rw-r--r--@ 1 Kohaugustine staff 71 Oct 3 10:15 ex1-7.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:22 hello
-rw-r--r--@ 1 Kohaugustine staff 58 Sep 12 16:27 hello.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@ 1 Kohaugustine staff 185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 21 10:06 temperature.o
-rw-r--r--@ 1 Kohaugustine staff 406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@ 1 Kohaugustine staff 582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@ 1 Kohaugustine staff 178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 23 17:21 temperature_ex1-5.o
Out[3]: 0
In [4]: subprocess.call(['ls', '-%sl' %'a'], shell=True)
a.out ex1-7 hello.c hello_2.o lesson_1-5.c temperature_ex1-4.c
ex1-6 ex1-7.c hello.o hello_3.o temperature.o temperature_ex1-5.c
ex1-6.c hello hello_1.o lesson_1-5 temperature_ex1-3.c temperature_ex1-5.o
Out[4]: 0
In [6]: subprocess.call(['ls', '-al'])
total 320
drwxr-xr-x 20 Kohaugustine staff 680 Oct 15 16:55 .
drwxr-xr-x 5 Kohaugustine staff 170 Sep 12 17:16 ..
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 a.out
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 3 10:28 ex1-6
-rw-r--r--@ 1 Kohaugustine staff 204 Oct 3 10:28 ex1-6.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Oct 3 10:15 ex1-7
-rw-r--r--@ 1 Kohaugustine staff 71 Oct 3 10:15 ex1-7.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:22 hello
-rw-r--r--@ 1 Kohaugustine staff 58 Sep 12 16:27 hello.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@ 1 Kohaugustine staff 185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 21 10:06 temperature.o
-rw-r--r--@ 1 Kohaugustine staff 406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@ 1 Kohaugustine staff 582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@ 1 Kohaugustine staff 178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 23 17:21 temperature_ex1-5.o
Out[6]: 0
In [7]: subprocess.call(['ls', '-al'], shell = True)
a.out ex1-7 hello.c hello_2.o lesson_1-5.c temperature_ex1-4.c
ex1-6 ex1-7.c hello.o hello_3.o temperature.o temperature_ex1-5.c
ex1-6.c hello hello_1.o lesson_1-5 temperature_ex1-3.c temperature_ex1-5.o
Out[7]: 0
It seems like whenever shell=True, the output seems to be the same as:
In [9]: subprocess.call(['ls'])
a.out ex1-7 hello.c hello_2.o lesson_1-5.c temperature_ex1-4.c
ex1-6 ex1-7.c hello.o hello_3.o temperature.o temperature_ex1-5.c
ex1-6.c hello hello_1.o lesson_1-5 temperature_ex1-3.c temperature_ex1-5.o
Out[9]: 0
I’m puzzled; what happened to the ‘-a’ option when I set shell=True? Didn’t the shell read it? I’ve read the Docs and that it says that when shell=True, it means that my specified command will be executed through the shell, so it should mean that ls -a was fed to the shell and acted upon by the shell. Then why the behavior in [4] and [7] ?
Also the pydocs doesn’t explain it directly (although it does say what subpprocess will NOT DO when we set shell=False); what does it mean when we let shell=False? Is a new process spawned in the OS without having the shell actually control it?
Also, in case it might seem really awkward that I’m using a format string in [3] and [4], its because in my actual script where I’ll be using subprocess.call, I will have to rely on these format strings to substitute in the appropriate command options. I cannot hardcode some of the command line options. Using a pure string for args is out of the question too because in my script there will be a method that has to do list operations on the commands. I don’t know if there might be a better way to go about this though, so if will really help if anyone can suggest something different.
Thank you very much!
Answers:
When you use shell=True with a list, extra arguments are passed to the shell itself, not to the command running in the shell. They can then be referred to from within the shell script (passed as argv[0]) as $0, $1, etc.
The easiest answer is "don’t do that": If you want to pass a list, don’t use shell=True; if you want to pass a string, always use shell=True.
That said, it is possible to form your command in such a way as to read those arguments. The below is an example that violates my above rule — a command you couldn’t implement[*1] without shell=True (and executable='/bin/bash' to avoid having a dependency on your operating system using bash for /bin/sh), because it relies on bash’s built-in version of printf (which supports %q as an extension):
subprocess.call([
"printf '%q\n' "$0" "$@"",
'these strings arern',
'"shell escaped" in the output from this command',
"so that the output can *safely* be run through eval",
"observe that no /tmp/owned file is created",
"including when the output of this script is run by bash as code:"
"$(touch /tmp/owned) "$(touch /tmp/owned)"",
'$(touch /tmp/owned) '$(touch /tmp/owned)'',
], shell=True, executable='/bin/bash')
[*1] – If one ignores that one could use /bin/bash as argv[0] with shell=False.
When shell is True, the first argument is appended to ["/bin/sh", "-c"]. If that argument is a list, the resulting list is
["/bin/sh", "-c", "ls", "-al"]
That is, only ls, not ls -al is used as the argument to the -c option. -al is used as the first argument the shell itself, not ls.
When using shell=True, you generally just want to pass a single string and let the shell split it according the shell’s normal word-splitting rules.
# Produces ["/bin/sh", "-c", "ls -al"]
subprocess.call("ls -al", shell=True)
In your case, it doesn’t see like you need to use shell=True at all.
I am trying to get python’s subprocess.call method to accept some args commands through a list (consisting of a sequence of strings) as advised in the python documentation. To explore this behavior before putting it into my actual script, I opened up IPython, ran some commands involving different combinations of shell settings and args commands and got the following behavior:
In [3]: subprocess.call(['ls', '-%sl' %'a'])
total 320
drwxr-xr-x 20 Kohaugustine staff 680 Oct 15 16:55 .
drwxr-xr-x 5 Kohaugustine staff 170 Sep 12 17:16 ..
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 a.out
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 3 10:28 ex1-6
-rw-r--r--@ 1 Kohaugustine staff 204 Oct 3 10:28 ex1-6.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Oct 3 10:15 ex1-7
-rw-r--r--@ 1 Kohaugustine staff 71 Oct 3 10:15 ex1-7.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:22 hello
-rw-r--r--@ 1 Kohaugustine staff 58 Sep 12 16:27 hello.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@ 1 Kohaugustine staff 185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 21 10:06 temperature.o
-rw-r--r--@ 1 Kohaugustine staff 406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@ 1 Kohaugustine staff 582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@ 1 Kohaugustine staff 178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 23 17:21 temperature_ex1-5.o
Out[3]: 0
In [4]: subprocess.call(['ls', '-%sl' %'a'], shell=True)
a.out ex1-7 hello.c hello_2.o lesson_1-5.c temperature_ex1-4.c
ex1-6 ex1-7.c hello.o hello_3.o temperature.o temperature_ex1-5.c
ex1-6.c hello hello_1.o lesson_1-5 temperature_ex1-3.c temperature_ex1-5.o
Out[4]: 0
In [6]: subprocess.call(['ls', '-al'])
total 320
drwxr-xr-x 20 Kohaugustine staff 680 Oct 15 16:55 .
drwxr-xr-x 5 Kohaugustine staff 170 Sep 12 17:16 ..
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 a.out
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 3 10:28 ex1-6
-rw-r--r--@ 1 Kohaugustine staff 204 Oct 3 10:28 ex1-6.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Oct 3 10:15 ex1-7
-rw-r--r--@ 1 Kohaugustine staff 71 Oct 3 10:15 ex1-7.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:22 hello
-rw-r--r--@ 1 Kohaugustine staff 58 Sep 12 16:27 hello.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x 1 Kohaugustine staff 8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@ 1 Kohaugustine staff 185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 21 10:06 temperature.o
-rw-r--r--@ 1 Kohaugustine staff 406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@ 1 Kohaugustine staff 582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@ 1 Kohaugustine staff 178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x 1 Kohaugustine staff 8496 Sep 23 17:21 temperature_ex1-5.o
Out[6]: 0
In [7]: subprocess.call(['ls', '-al'], shell = True)
a.out ex1-7 hello.c hello_2.o lesson_1-5.c temperature_ex1-4.c
ex1-6 ex1-7.c hello.o hello_3.o temperature.o temperature_ex1-5.c
ex1-6.c hello hello_1.o lesson_1-5 temperature_ex1-3.c temperature_ex1-5.o
Out[7]: 0
It seems like whenever shell=True, the output seems to be the same as:
In [9]: subprocess.call(['ls'])
a.out ex1-7 hello.c hello_2.o lesson_1-5.c temperature_ex1-4.c
ex1-6 ex1-7.c hello.o hello_3.o temperature.o temperature_ex1-5.c
ex1-6.c hello hello_1.o lesson_1-5 temperature_ex1-3.c temperature_ex1-5.o
Out[9]: 0
I’m puzzled; what happened to the ‘-a’ option when I set shell=True? Didn’t the shell read it? I’ve read the Docs and that it says that when shell=True, it means that my specified command will be executed through the shell, so it should mean that ls -a was fed to the shell and acted upon by the shell. Then why the behavior in [4] and [7] ?
Also the pydocs doesn’t explain it directly (although it does say what subpprocess will NOT DO when we set shell=False); what does it mean when we let shell=False? Is a new process spawned in the OS without having the shell actually control it?
Also, in case it might seem really awkward that I’m using a format string in [3] and [4], its because in my actual script where I’ll be using subprocess.call, I will have to rely on these format strings to substitute in the appropriate command options. I cannot hardcode some of the command line options. Using a pure string for args is out of the question too because in my script there will be a method that has to do list operations on the commands. I don’t know if there might be a better way to go about this though, so if will really help if anyone can suggest something different.
Thank you very much!
When you use shell=True with a list, extra arguments are passed to the shell itself, not to the command running in the shell. They can then be referred to from within the shell script (passed as argv[0]) as $0, $1, etc.
The easiest answer is "don’t do that": If you want to pass a list, don’t use shell=True; if you want to pass a string, always use shell=True.
That said, it is possible to form your command in such a way as to read those arguments. The below is an example that violates my above rule — a command you couldn’t implement[*1] without shell=True (and executable='/bin/bash' to avoid having a dependency on your operating system using bash for /bin/sh), because it relies on bash’s built-in version of printf (which supports %q as an extension):
subprocess.call([
"printf '%q\n' "$0" "$@"",
'these strings arern',
'"shell escaped" in the output from this command',
"so that the output can *safely* be run through eval",
"observe that no /tmp/owned file is created",
"including when the output of this script is run by bash as code:"
"$(touch /tmp/owned) "$(touch /tmp/owned)"",
'$(touch /tmp/owned) '$(touch /tmp/owned)'',
], shell=True, executable='/bin/bash')
[*1] – If one ignores that one could use /bin/bash as argv[0] with shell=False.
When shell is True, the first argument is appended to ["/bin/sh", "-c"]. If that argument is a list, the resulting list is
["/bin/sh", "-c", "ls", "-al"]
That is, only ls, not ls -al is used as the argument to the -c option. -al is used as the first argument the shell itself, not ls.
When using shell=True, you generally just want to pass a single string and let the shell split it according the shell’s normal word-splitting rules.
# Produces ["/bin/sh", "-c", "ls -al"]
subprocess.call("ls -al", shell=True)
In your case, it doesn’t see like you need to use shell=True at all.