Invoking C compiler using Python subprocess command
Question:
I am trying to compile a C program using Python and want to give input using “<” operator but it’s not working as expected.
If I compile the C program and run it by giving input though a file it works; for example
./a.out <inp.txt works
But similarly if I try to do this using a Python script, it did not quite work out as expected.
For example:
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x"])
and
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x","<inp.txt"])
Both script ask for input though terminal. But I think in the second script it should read from file. why both the programs are working the same?
Answers:
The shell does I/O redirection for a process. Based on what you’re saying, the subprocess module does not do I/O redirection like that. To demonstrate, run:
subprocess.call(["sh","-c", "./x <inp.txt"])
That runs the shell and should redirect the I/O. With your code, your program ./x is being given an argument <inp.txt which it is ignoring.
NB: the alternative call to subprocess.call is purely for diagnostic purposes, not a recommended solution. The recommended solution involves reading the (Python 2) subprocess module documentation (or the Python 3 documentation for it) to find out how to do the redirection using the module.
import subprocess
i_file = open("inp.txt")
subprocess.call("./x", stdin=i_file)
i_file.close()
If your script is about to exit so you don’t have to worry about wasted file descriptors, you can compress that to:
import subprocess
subprocess.call("./x", stdin=open("inp.txt"))
By default, the subprocess module does not pass the arguments to the shell. Why? Because running commands via the shell is dangerous; unless they’re correctly quoted and escaped (which is complicated), it is often possible to convince programs that do this kind of thing to run unwanted and unexpected shell commands.
Using the shell for this would be wrong anyway. If you want to take input from a particular file, you can use subprocess.Popen, setting the stdin argument to a file descriptor for the file inp.txt (you can get the file descriptor by calling fileno() a Python file object).
To complement @Jonathan Leffler’s and @alastair’s helpful answers:
Assuming you control the string you’re passing to the shell for execution, I see nothing wrong with using the shell for convenience. [1]
subprocess.call() has an optional Boolean shell parameter, which causes the command to be passed to the shell, enabling I/O redirection, referencing environment variables, …:
subprocess.call("./x <inp.txt", shell = True)
Note how the entire command line is passed as a single string rather than an array of arguments.
[1]
Avoid use of the shell in the following cases:
- If your Python code must run on platforms other than Unix-like ones, such as Windows.
- If performance is paramount.
- If you find yourself "outsourcing" tasks better handled on the Python side.
If you’re concerned about lack of predictability of the shell environment (as @alastair is):
-
subprocess.call with shell = True always creates non-interactive non-login instances of /bin/sh – note that it is NOT the user’s default shell that is used.
-
sh does NOT read initialization files for non-interactive non-login shells (neither system-wide nor user-specific ones).
- Note that even on platforms where
sh is bash in disguise, bash will act this way when invoked as sh.
-
Every shell instance created with subprocess.call with shell = True is its own world, and its environment is neither influenced by previous shell instances nor does it influence later ones.
-
However, the shell instances created do inherit the environment of the python process itself:
-
If you started your Python program from an interactive shell, then that shell’s environment is inherited. Note that this only pertains to the current working directory and environment variables, and NOT to aliases, shell functions, and shell variables.
-
Generally, that’s a feature, given that Python (CPython) itself is designed to be controllable via environment variables (for 2.x, see https://docs.python.org/2/using/cmdline.html#environment-variables; for 3.x, see https://docs.python.org/3/using/cmdline.html#environment-variables).
-
If needed, you can supply your own environment to the shell via the env parameter; note, however, that you’ll have to supply the entire environment in that event, potentially including variables such as USER and HOME, if needed; simple example, defining $PATH explicitly:
subprocess.call('echo $PATH', shell = True,
env = { 'PATH': '/sbin:/bin:/usr/bin' })
I am trying to compile a C program using Python and want to give input using “<” operator but it’s not working as expected.
If I compile the C program and run it by giving input though a file it works; for example
./a.out <inp.txt works
But similarly if I try to do this using a Python script, it did not quite work out as expected.
For example:
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x"])
and
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x","<inp.txt"])
Both script ask for input though terminal. But I think in the second script it should read from file. why both the programs are working the same?
The shell does I/O redirection for a process. Based on what you’re saying, the subprocess module does not do I/O redirection like that. To demonstrate, run:
subprocess.call(["sh","-c", "./x <inp.txt"])
That runs the shell and should redirect the I/O. With your code, your program ./x is being given an argument <inp.txt which it is ignoring.
NB: the alternative call to subprocess.call is purely for diagnostic purposes, not a recommended solution. The recommended solution involves reading the (Python 2) subprocess module documentation (or the Python 3 documentation for it) to find out how to do the redirection using the module.
import subprocess
i_file = open("inp.txt")
subprocess.call("./x", stdin=i_file)
i_file.close()
If your script is about to exit so you don’t have to worry about wasted file descriptors, you can compress that to:
import subprocess
subprocess.call("./x", stdin=open("inp.txt"))
By default, the subprocess module does not pass the arguments to the shell. Why? Because running commands via the shell is dangerous; unless they’re correctly quoted and escaped (which is complicated), it is often possible to convince programs that do this kind of thing to run unwanted and unexpected shell commands.
Using the shell for this would be wrong anyway. If you want to take input from a particular file, you can use subprocess.Popen, setting the stdin argument to a file descriptor for the file inp.txt (you can get the file descriptor by calling fileno() a Python file object).
To complement @Jonathan Leffler’s and @alastair’s helpful answers:
Assuming you control the string you’re passing to the shell for execution, I see nothing wrong with using the shell for convenience. [1]
subprocess.call() has an optional Boolean shell parameter, which causes the command to be passed to the shell, enabling I/O redirection, referencing environment variables, …:
subprocess.call("./x <inp.txt", shell = True)
Note how the entire command line is passed as a single string rather than an array of arguments.
[1]
Avoid use of the shell in the following cases:
- If your Python code must run on platforms other than Unix-like ones, such as Windows.
- If performance is paramount.
- If you find yourself "outsourcing" tasks better handled on the Python side.
If you’re concerned about lack of predictability of the shell environment (as @alastair is):
-
subprocess.callwithshell = Truealways creates non-interactive non-login instances of/bin/sh– note that it is NOT the user’s default shell that is used. -
shdoes NOT read initialization files for non-interactive non-login shells (neither system-wide nor user-specific ones).- Note that even on platforms where
shisbashin disguise,bashwill act this way when invoked assh.
- Note that even on platforms where
-
Every shell instance created with
subprocess.callwithshell = Trueis its own world, and its environment is neither influenced by previous shell instances nor does it influence later ones. -
However, the shell instances created do inherit the environment of the python process itself:
-
If you started your Python program from an interactive shell, then that shell’s environment is inherited. Note that this only pertains to the current working directory and environment variables, and NOT to aliases, shell functions, and shell variables.
-
Generally, that’s a feature, given that Python (CPython) itself is designed to be controllable via environment variables (for 2.x, see https://docs.python.org/2/using/cmdline.html#environment-variables; for 3.x, see https://docs.python.org/3/using/cmdline.html#environment-variables).
-
If needed, you can supply your own environment to the shell via the
envparameter; note, however, that you’ll have to supply the entire environment in that event, potentially including variables such asUSERandHOME, if needed; simple example, defining$PATHexplicitly:subprocess.call('echo $PATH', shell = True, env = { 'PATH': '/sbin:/bin:/usr/bin' })
-