Why does passing variables to subprocess.Popen not work despite passing a list of arguments?

Question

I have a script which calls another Python script by subprocess.Popen. But since I have arguments stored in variable(s)

servers[server]['address']
servers[server]['port']
servers[server]['pass']

I am unable to perform the command

p = subprocess.Popen(
    ["python mytool.py -a ", servers[server]['address'], "-x", servers[server]['port'], "-p", servers[server]['pass'], "some additional command"],
    shell=True,
    stdout=subprocess.PIPE
)

Asked By: GaNi

||

Source

Answer 1

When you call subprocess.Popen you can pass either a string or a list for the command to be run. If you pass a list, the items should be split in a particular way.

In your case, you need to split it something like this:

command = ["python",  "mytool.py", "-a", servers[server]['address'], 
           "-x", servers[server]['port'], 
           "-p", servers[server]['pass'], 
           "some",  "additional", "command"]
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

This is because if you pass in a list, Popen assumes you have already split the command line into words (the values that would end up in sys.argv), so it doesn’t need to.

The way you’re calling it, it will try to run a binary called “python mytool.py -a”, which isn’t what you meant.

The other way to fix it is to join all of the words into a string (which Popen will then split up – see subprocess.list2cmdline). But you’re better off using the list version if possible – it gives simpler control of how the commandline is split up (if arguments have spaces or quotes in them, for example) without having to mess around with quoting quote characters.

Answered By: babbageclunk

Answer 2

You should concatenate the command to a whole string:

p = subprocess.Popen("python mytool.py -a " + servers[server]['address'] + " -x " + servers[server]['port'] + " -p " + servers[server]['pass'] + " some additional command", shell=True, stdout=subprocess.PIPE)

Answered By: ciphor

Answer 3

Your problem in type str for first Popen argument. Replace it to list. Below code can work:

address = servers[server]['address']
port = servers[server]['port']
pass = servers[server]['pass']

command = "python mytool.py -a %s -x %d -p %s some additional command" % (address, port, pass)
p = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
#        it is a list^^^^^^^^^^^^^^^  shell=False

If command arguments get from a trusted source you can construct command and use it with shell=True to such manner:

import pipes as p
command = "python mytool.py -a {} -x {} -p {} some additional command".format(p.quote(address), p.quote(port), p.quote(pass))
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

Note 1: constructed command with shell=Trueis potentially insecure. Use pipes.quote() for reduce injection possibility.
Note 2: pipes.quote() deprecated since python2; for python3 use shlex module.

Answered By: Michael Kazarian

Answer 4

Drop shell=True. The arguments to Popen() are treated differently on Unix if shell=True:

import sys
from subprocess import Popen, PIPE

# populate list of arguments
args = ["mytool.py"]
for opt, optname in zip("-a -x -p".split(), "address port pass".split()):
    args.extend([opt, str(servers[server][optname])])
args.extend("some additional command".split())

# run script
p = Popen([sys.executable or 'python'] + args, stdout=PIPE)
# use p.stdout here...
p.stdout.close()
p.wait()

Note that passing shell=True for commands with external input is a security hazard, as described by a warning in the docs.

Answered By: jfs

Answer 5

Usecase

To run n amount of shell script or python script

Create a run.py file that responsible to pass the argument(integer) to the shell file.

Let say you like to run n(4) qty of shell script or python script that accept 1 argument.

Create a file run.python with the code below.

Below code illustrate

instanceQty = Amount of shell script to run
os.getcwd() = path to your current file
mockScript.sh = shell script that I put on same directory with run.py

To run shell script

# this is python file with name run.py
import subprocess,os
instanceQty = 4
for i in range(0, instanceQty):
    print(os.getcwd())
    subprocess.Popen(f"{os.getcwd()}/mockScript.sh {i}",shell=True,executable='/bin/bash')

To run python script

import subprocess,os,sys
instanceQty = 4
for i in range(0, instanceQty):
    print(os.getcwd())
    subprocess.Popen([sys.executable,f"{os.getcwd()}/mockScript.py",str(i)])

Run this file with

python run.py

permission problem on MacOS

sudo chmod ug+x mockScript.sh

sudo chmod ug+x run.py

All code tested on Python 3.8.1 and MacOs 12.0.1 environment.

Answered By: Tan Hoo Chuan Nickson

Answer 6

You can set the environment by passing

env={whatever_you_want: values} | os.environ

in your subprocess call.

Answered By: Max Bileschi

Why does passing variables to subprocess.Popen not work despite passing a list of arguments?

Question:

Answers: