Something like that:

def runProcess(exe):    
    p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    while(True):
        # returns None while subprocess is running
        retcode = p.poll() 
        line = p.stdout.readline()
        yield line
        if retcode is not None:
            break

Note, that I’m redirecting stderr to stdout, it might not be exactly what you want, but I want error messages also.

This function yields line by line as they come (normally you’d have to wait for subprocess to finish to get the output as a whole).

For your case the usage would be:

for line in runProcess('mysqladmin create test -uroot -pmysqladmin12'.split()):
    print line,

This is way easier, but only works on Unix (including Cygwin) and Python2.7.

import commands
print commands.getstatusoutput('wc -l file')

It returns a tuple with the (return_value, output).

For a solution that works in both Python2 and Python3, use the subprocess module instead:

from subprocess import Popen, PIPE
output = Popen(["date"],stdout=PIPE)
response = output.communicate()
print response

Vartec’s answer doesn’t read all lines, so I made a version that did:

def run_command(command):
    p = subprocess.Popen(command,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')

Usage is the same as the accepted answer:

command = 'mysqladmin create test -uroot -pmysqladmin12'.split()
for line in run_command(command):
    print(line)

Your Mileage May Vary, I attempted @senderle’s spin on Vartec’s solution in Windows on Python 2.6.5, but I was getting errors, and no other solutions worked. My error was: WindowsError: [Error 6] The handle is invalid.

I found that I had to assign PIPE to every handle to get it to return the output I expected – the following worked for me.

import subprocess

def run_command(cmd):
    """given shell command, returns communication tuple of stdout and stderr"""
    return subprocess.Popen(cmd, 
                            stdout=subprocess.PIPE, 
                            stderr=subprocess.PIPE, 
                            stdin=subprocess.PIPE).communicate()

and call like this, ([0] gets the first element of the tuple, stdout):

run_command('tracert 11.1.0.1')[0]

After learning more, I believe I need these pipe arguments because I’m working on a custom system that uses different handles, so I had to directly control all the std’s.

To stop console popups (with Windows), do this:

def run_command(cmd):
    """given shell command, returns communication tuple of stdout and stderr"""
    # instantiate a startupinfo obj:
    startupinfo = subprocess.STARTUPINFO()
    # set the use show window flag, might make conditional on being in Windows:
    startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
    # pass as the startupinfo keyword argument:
    return subprocess.Popen(cmd,
                            stdout=subprocess.PIPE, 
                            stderr=subprocess.PIPE, 
                            stdin=subprocess.PIPE, 
                            startupinfo=startupinfo).communicate()

run_command('tracert 11.1.0.1')

If you need to run a shell command on multiple files, this did the trick for me.

import os
import subprocess

# Define a function for running commands and capturing stdout line by line
# (Modified from Vartec's solution because it wasn't printing all lines)
def runProcess(exe):    
    p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')

# Get all filenames in working directory
for filename in os.listdir('./'):
    # This command will be run on each file
    cmd = 'nm ' + filename

    # Run the command and capture the output line by line.
    for line in runProcess(cmd.split()):
        # Eliminate leading and trailing whitespace
        line.strip()
        # Split the output 
        output = line.split()

        # Filter the output and print relevant lines
        if len(output) > 2:
            if ((output[2] == 'set_program_name')):
                print filename
                print line

Edit: Just saw Max Persson’s solution with J.F. Sebastian’s suggestion. Went ahead and incorporated that.

This is a tricky but super simple solution which works in many situations:

import os
os.system('sample_cmd > tmp')
print(open('tmp', 'r').read())

A temporary file(here is tmp) is created with the output of the command and you can read from it your desired output.

Extra note from the comments:
You can remove the tmp file in the case of one-time job. If you need to do this several times, there is no need to delete the tmp.

os.remove('tmp')

I had a slightly different flavor of the same problem with the following requirements:

1. Capture and return STDOUT messages as they accumulate in the STDOUT buffer (i.e. in realtime).
   • @vartec solved this Pythonically with his use of generators and the ‘yield’
     keyword above
2. Print all STDOUT lines (even if process exits before STDOUT buffer can be fully read)
3. Don’t waste CPU cycles polling the process at high-frequency
4. Check the return code of the subprocess
5. Print STDERR (separate from STDOUT) if we get a non-zero error return code.

I’ve combined and tweaked previous answers to come up with the following:

import subprocess
from time import sleep

def run_command(command):
    p = subprocess.Popen(command,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE,
                         shell=True)
    # Read stdout from subprocess until the buffer is empty !
    for line in iter(p.stdout.readline, b''):
        if line: # Don't print blank lines
            yield line
    # This ensures the process has completed, AND sets the 'returncode' attr
    while p.poll() is None:                                                                                                                                        
        sleep(.1) #Don't waste CPU-cycles
    # Empty STDERR buffer
    err = p.stderr.read()
    if p.returncode != 0:
       # The run_command() function is responsible for logging STDERR 
       print("Error: " + str(err))

This code would be executed the same as previous answers:

for line in run_command(cmd):
    print(line)

python3 offers subprocess.getoutput():

import subprocess
output = subprocess.getoutput("ls -l")
print(output)

You can use following commands to run any shell command. I have used them on ubuntu.

import os
os.popen('your command here').read()

Note: This is deprecated since python 2.6. Now you must use subprocess.Popen. Below is the example

import subprocess

p = subprocess.Popen("Your command", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
print p.split("n")

eg, execute(‘ls -ahl’)
differentiated three/four possible returns and OS platforms:

1. no output, but run successfully
2. output empty line, run successfully
3. run failed
4. output something, run successfully

function below

def execute(cmd, output=True, DEBUG_MODE=False):
"""Executes a bash command.
(cmd, output=True)
output: whether print shell output to screen, only affects screen display, does not affect returned values
return: ...regardless of output=True/False...
        returns shell output as a list with each elment is a line of string (whitespace stripped both sides) from output
        could be 
        [], ie, len()=0 --> no output;    
        [''] --> output empty line;     
        None --> error occured, see below

        if error ocurs, returns None (ie, is None), print out the error message to screen
"""
if not DEBUG_MODE:
    print "Command: " + cmd

    # https://stackoverflow.com/a/40139101/2292993
    def _execute_cmd(cmd):
        if os.name == 'nt' or platform.system() == 'Windows':
            # set stdin, out, err all to PIPE to get results (other than None) after run the Popen() instance
            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
        else:
            # Use bash; the default is sh
            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, executable="/bin/bash")

        # the Popen() instance starts running once instantiated (??)
        # additionally, communicate(), or poll() and wait process to terminate
        # communicate() accepts optional input as stdin to the pipe (requires setting stdin=subprocess.PIPE above), return out, err as tuple
        # if communicate(), the results are buffered in memory

        # Read stdout from subprocess until the buffer is empty !
        # if error occurs, the stdout is '', which means the below loop is essentially skipped
        # A prefix of 'b' or 'B' is ignored in Python 2; 
        # it indicates that the literal should become a bytes literal in Python 3 
        # (e.g. when code is automatically converted with 2to3).
        # return iter(p.stdout.readline, b'')
        for line in iter(p.stdout.readline, b''):
            # # Windows has rn, Unix has n, Old mac has r
            # if line not in ['','n','r','rn']: # Don't print blank lines
                yield line
        while p.poll() is None:                                                                                                                                        
            sleep(.1) #Don't waste CPU-cycles
        # Empty STDERR buffer
        err = p.stderr.read()
        if p.returncode != 0:
            # responsible for logging STDERR 
            print("Error: " + str(err))
            yield None

    out = []
    for line in _execute_cmd(cmd):
        # error did not occur earlier
        if line is not None:
            # trailing comma to avoid a newline (by print itself) being printed
            if output: print line,
            out.append(line.strip())
        else:
            # error occured earlier
            out = None
    return out
else:
    print "Simulation! The command is " + cmd
    print ""

Splitting the initial command for the subprocess might be tricky and cumbersome.

Use shlex.split() to help yourself out.

Sample command

git log -n 5 --since "5 years ago" --until "2 year ago"

The code

from subprocess import check_output
from shlex import split

res = check_output(split('git log -n 5 --since "5 years ago" --until "2 year ago"'))
print(res)
>>> b'commit 7696ab087a163e084d6870bb4e5e4d4198bdc61anAuthor: Artur Barseghyan...'

Without shlex.split() the code would look as follows

res = check_output([
    'git', 
    'log', 
    '-n', 
    '5', 
    '--since', 
    '5 years ago', 
    '--until', 
    '2 year ago'
])
print(res)
>>> b'commit 7696ab087a163e084d6870bb4e5e4d4198bdc61anAuthor: Artur Barseghyan...'

According to @senderle, if you use python3.6 like me:

def sh(cmd, input=""):
    rst = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, input=input.encode("utf-8"))
    assert rst.returncode == 0, rst.stderr.decode("utf-8")
    return rst.stdout.decode("utf-8")

sh("ls -a")

Will act exactly like you run the command in bash

If you use the subprocess python module, you are able to handle the STDOUT, STDERR and return code of command separately. You can see an example for the complete command caller implementation. Of course you can extend it with try..except if you want.

The below function returns the STDOUT, STDERR and Return code so you can handle them in the other script.

import subprocess

def command_caller(command=None)
    sp = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=False)
    out, err = sp.communicate()
    if sp.returncode:
        print(
            "Return code: %(ret_code)s Error message: %(err_msg)s"
            % {"ret_code": sp.returncode, "err_msg": err}
            )
    return sp.returncode, out, err

The output can be redirected to a text file and then read it back.

import subprocess
import os
import tempfile

def execute_to_file(command):
    """
    This function execute the command
    and pass its output to a tempfile then read it back
    It is usefull for process that deploy child process
    """
    temp_file = tempfile.NamedTemporaryFile(delete=False)
    temp_file.close()
    path = temp_file.name
    command = command + " > " + path
    proc = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
    if proc.stderr:
        # if command failed return
        os.unlink(path)
        return
    with open(path, 'r') as f:
        data = f.read()
    os.unlink(path)
    return data

if __name__ == "__main__":
    path = "Somepath"
    command = 'ecls.exe /files ' + path
    print(execute(command))

Here a solution, working if you want to print output while process is running or not.

I added the current working directory also, it was useful to me more than once.

Hoping the solution will help someone :).

import subprocess

def run_command(cmd_and_args, print_constantly=False, cwd=None):
"""Runs a system command.

:param cmd_and_args: the command to run with or without a Pipe (|).
:param print_constantly: If True then the output is logged in continuous until the command ended.
:param cwd: the current working directory (the directory from which you will like to execute the command)
:return: - a tuple containing the return code, the stdout and the stderr of the command
"""
output = []

process = subprocess.Popen(cmd_and_args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd)

while True:
    next_line = process.stdout.readline()
    if next_line:
        output.append(str(next_line))
        if print_constantly:
            print(next_line)
    elif not process.poll():
        break

error = process.communicate()[1]

return process.returncode, 'n'.join(output), error

I would like to suggest simppl as an option for consideration. It is a module that is available via pypi: pip install simppl and was runs on python3.

simppl allows the user to run shell commands and read the output from the screen.

The developers suggest three types of use cases:

1. The simplest usage will look like this:

    from simppl.simple_pipeline import SimplePipeline
    sp = SimplePipeline(start=0, end=100):
    sp.print_and_run('<YOUR_FIRST_OS_COMMAND>')
    sp.print_and_run('<YOUR_SECOND_OS_COMMAND>') ```

2. To run multiple commands concurrently use:

    commands = ['<YOUR_FIRST_OS_COMMAND>', '<YOUR_SECOND_OS_COMMAND>']
    max_number_of_processes = 4
    sp.run_parallel(commands, max_number_of_processes) ```

3. Finally, if your project uses the cli module, you can run directly another command_line_tool as part of a pipeline. The other tool will
   be run from the same process, but it will appear from the logs as
   another command in the pipeline. This enables smoother debugging and
   refactoring of tools calling other tools.

    from example_module import example_tool
    sp.print_and_run_clt(example_tool.run, ['first_number', 'second_nmber'], 
                                 {'-key1': 'val1', '-key2': 'val2'},
                                 {'--flag'}) ```

Note that the printing to STDOUT/STDERR is via python’s logging module.

Here is a complete code to show how simppl works:

import logging
from logging.config import dictConfig

logging_config = dict(
    version = 1,
    formatters = {
        'f': {'format':
              '%(asctime)s %(name)-12s %(levelname)-8s %(message)s'}
        },
    handlers = {
        'h': {'class': 'logging.StreamHandler',
              'formatter': 'f',
              'level': logging.DEBUG}
        },
    root = {
        'handlers': ['h'],
        'level': logging.DEBUG,
        },
)
dictConfig(logging_config)

from simppl.simple_pipeline import SimplePipeline
sp = SimplePipeline(0, 100)
sp.print_and_run('ls')

On Python 3.7+, use subprocess.run and pass capture_output=True:

import subprocess
result = subprocess.run(['echo', 'hello', 'world'], capture_output=True)
print(repr(result.stdout))

This will return bytes:

b'hello worldn'

If you want it to convert the bytes to a string, add text=True:

result = subprocess.run(['echo', 'hello', 'world'], capture_output=True, text=True)
print(repr(result.stdout))

This will read the bytes using your default encoding:

'hello worldn'

If you need to manually specify a different encoding, use encoding="your encoding" instead of text=True:

result = subprocess.run(['echo', 'hello', 'world'], capture_output=True, encoding="utf8")
print(repr(result.stdout))

Here is a simple and flexible solution that works on a variety of OS versions, and both Python 2 and 3, using IPython in shell mode:

from IPython.terminal.embed import InteractiveShellEmbed
my_shell = InteractiveShellEmbed()
result = my_shell.getoutput("echo hello world")
print(result)

Out: ['hello world']

It has a couple of advantages

1. It only requires an IPython install, so you don’t really need to worry about your specific Python or OS version when using it, it comes with Jupyter – which has a wide range of support
2. It takes a simple string by default – so no need to use shell mode arg or string splitting, making it slightly cleaner IMO
3. It also makes it cleaner to easily substitute variables or even entire Python commands in the string itself

To demonstrate:

var = "hello world "
result = my_shell.getoutput("echo {var*2}")
print(result)

Out: ['hello world hello world']

Just wanted to give you an extra option, especially if you already have Jupyter installed

Naturally, if you are in an actual Jupyter notebook as opposed to a .py script you can also always do:

result = !echo hello world
print(result)

To accomplish the same.

For some reason, this one works on Python 2.7 and you only need to import os!

import os 

def bash(command):
    output = os.popen(command).read()
    return output

print_me = bash('ls -l')
print(print_me)

Improvement for better logging.
For better output you can use iterator.
From below, we get better

from subprocess import Popen, getstatusoutput, PIPE
def shell_command(cmd):
    result = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)

    output = iter(result.stdout.readline, b'')
    error = iter(result.stderr.readline, b'')
    print("##### OutPut ###")
    for line in output:
        print(line.decode("utf-8"))
    print("###### Error ########")
    for line in error:
        print(error.decode("utf-8")) # Convert bytes to str

    status, terminal_output = run_command(cmd)
    print(terminal_output)

shell_command("ls") # this will display all the files & folders in directory

Other method using getstatusoutput ( Easy to understand)

from subprocess import Popen, getstatusoutput, PIPE

status_Code, output = getstausoutput(command)
print(output) # this will give the terminal output

# status_code, output = getstatusoutput("ls") # this will print the all files & folder available in the directory

If you want to capture both stdout and stderr and display them as they would appear when a shell command is executed in a interactive terminal and you needed to know the return status of the command, you can do a following hack.

import time, os
cmd_str = "ls -d /bin/nonsense"

tmpfile= "/tmp/results." + str(os.getpid()) + "." + str(time.time()) 
status = os.system( cmd_str + " > " + tmpfile +" 2>&1")
with open(tmpfile, 'r') as f:
  print (f.read())
os.remove(tmpfile)
  
print("status=" + str(status))

Note:
The tmpfile is unique and removed as soon as it is used.
Shell is used to parse the cmd_str, hence don’t use this technique for arbitrary strings because it is unsafe.

The key is to use the function subprocess.check_output

For example, the following function captures stdout and stderr of the process and returns that as well as whether or not the call succeeded. It is Python 2 and 3 compatible:

from subprocess import check_output, CalledProcessError, STDOUT

def system_call(command):
    """ 
    params:
        command: list of strings, ex. `["ls", "-l"]`
    returns: output, success
    """
    try:
        output = check_output(command, stderr=STDOUT).decode()
        success = True 
    except CalledProcessError as e:
        output = e.output.decode()
        success = False
    return output, success

output, success = system_call(["ls", "-l"])

If you want to pass commands as strings rather than arrays, use this version:

from subprocess import check_output, CalledProcessError, STDOUT
import shlex

def system_call(command):
    """ 
    params:
        command: string, ex. `"ls -l"`
    returns: output, success
    """
    command = shlex.split(command)
    try:
        output = check_output(command, stderr=STDOUT).decode()
        success = True 
    except CalledProcessError as e:
        output = e.output.decode()
        success = False
    return output, success

output, success = system_call("ls -l")

This is the code that I use to support multithreading in Jupyter notebook cell and make cell printing the shell outputs, in real time. It make use of bufsize and stderr.

from subprocess import Popen, PIPE, STDOUT

def verbosecmd(command):
    with Popen(
        command, stdout=PIPE, shell=True, stderr=STDOUT, bufsize=0, close_fds=True
    ) as process:
        for line in iter(process.stdout.readline, b""):
            print(line.rstrip().decode("utf-8"))

1-liner (for the obsessed)

def cmd(x): from subprocess import check_output as c; return c(x, shell=True, text=True) # noqa

usage

print(cmd('python -c "print('hello')"')) #recursion
print(cmd('ls -la')) #linux
print(cmd('ipconfig')) #windows

comments

• dont worry about subprocess being imported every time, subsequent calls are just indexed like variable
• it is not according to pep8 formatting standard and looses single line on save if any formatter is configured in vscode, you may ignore using --ignore=E702,E704,E501 by adding in settings or appending # noqa at end (i already added) ref this answer
• Also errors are raised and not passed silently

_{Credits : Icons from flaticon.com}