Calling the "source" command from subprocess.Popen

source is a bash-specific shell built-in (and non-interactive shells are often lightweight dash shells instead of bash). Instead, just call /bin/sh:

foo = subprocess.Popen(["/bin/sh", "the_script.sh"])

You could just run the command in a subshell and use the results to update the current environment.

def shell_source(script):
    """Sometime you want to emulate the action of "source" in bash,
    settings some environment variables. Here is a way to do it."""
    import subprocess, os
    pipe = subprocess.Popen(". %s; env" % script, stdout=subprocess.PIPE, shell=True)
    output = pipe.communicate()[0]
    env = dict((line.split("=", 1) for line in output.splitlines()))
    os.environ.update(env)

If you want to apply source command to some other scripts or executables, then you might create another wrapping script file and call “source” command from it with any further logic you need. In that case, this source command will modify local context where it running – namely in the subprocess that subprocess.Popen creates.

This will not work if you need to modify python context, where your program is running.

A variation on the answer by @xApple since it sometimes is useful to be able to source a shell script (rather than a Python file) to set environment variables, and maybe perform other shell operations, and then propagate that environment to the Python interpreter rather than losing that info when the subshell closes.

The reason for a variation is that the assumption of a one-variable-per-line format of output from “env” isn’t 100% robust: I just had to deal with a variable (a shell function, I think) containing a newline, which screwed up that parsing. So here is a slightly more complex version, which uses Python itself to format the environment dictionary in a robust way:

import subprocess
pipe = subprocess.Popen(". ./shellscript.sh; python -c 'import os; print "newenv = %r" % os.environ'", 
    stdout=subprocess.PIPE, shell=True)
exec(pipe.communicate()[0])
os.environ.update(newenv)

Maybe there is a neater way? This also ensures that the environment parsing isn’t messed up if someone puts an echo statement into the script that’s being sourced. Of course, there’s an exec in here so be careful about non-trusted input… but I think that’s implicit in a discussion about how to source/execute an arbitrary shell script 😉

UPDATE: see @unutbu’s comment on the @xApple answer for an alternative (probably nicer) way to handle newlines in the env output.

There seem to be a lot of answers to this, haven’t read all of them so they may have already pointed it out; but, when calling shell commands like this, you have to pass shell=True to the Popen call. Otherwise, you can call Popen(shlex.split()). make sure to import shlex.

I actually use this function for the purpose of sourcing a file and modifying the current environment.

def set_env(env_file):
    while True:
        source_file = '/tmp/regr.source.%d'%random.randint(0, (2**32)-1)
        if not os.path.isfile(source_file): break
    with open(source_file, 'w') as src_file:
        src_file.write('#!/bin/bashn')
        src_file.write('source %sn'%env_file)
        src_file.write('envn')
    os.chmod(source_file, 0755)
    p = subprocess.Popen(source_file, shell=True,
                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    (out, err) = p.communicate()
    setting = re.compile('^(?P<setting>[^=]*)=')
    value = re.compile('=(?P<value>.*$)')
    env_dict = {}
    for line in out.splitlines():
        if setting.search(line) and value.search(line):
            env_dict[setting.search(line).group('setting')] = value.search(line).group('value')
    for k, v in env_dict.items():
        os.environ[k] = v
    for k, v in env_dict.items():
        try:
            assert(os.getenv(k) == v)
        except AssertionError:
            raise Exception('Unable to modify environment')

Broken Popen("source the_script.sh") is equivalent to Popen(["source the_script.sh"]) that tries unsuccessfully to launch 'source the_script.sh' program. It can’t find it, hence "No such file or directory" error.

Broken Popen("source the_script.sh", shell=True) fails because source is a bash builtin command (type help source in bash) but the default shell is /bin/sh that doesn’t understand it (/bin/sh uses .). Assuming there could be other bash-ism in the_script.sh, it should be run using bash:

foo = Popen("source the_script.sh", shell=True, executable="/bin/bash")

As @IfLoop said, it is not very useful to execute source in a subprocess because it can’t affect parent’s environment.

os.environ.update(env) -based methods fail if the_script.sh executes unset for some variables. os.environ.clear() could be called to reset the environment:

#!/usr/bin/env python2
import os
from pprint import pprint
from subprocess import check_output

os.environ['a'] = 'a'*100
# POSIX: name shall not contain '=', value doesn't contain ''
output = check_output("source the_script.sh; env -0",   shell=True,
                      executable="/bin/bash")
# replace env
os.environ.clear() 
os.environ.update(line.partition('=')[::2] for line in output.split(''))
pprint(dict(os.environ)) #NOTE: only `export`ed envvars here

It uses env -0 and .split('') suggested by @unutbu

To support arbitrary bytes in os.environb, json module could be used (assuming we use Python version where “json.dumps not parsable by json.loads” issue is fixed):

To avoid passing the environment via pipes, the Python code could be changed to invoke itself in the subprocess environment e.g.:

#!/usr/bin/env python2
import os
import sys
from pipes import quote
from pprint import pprint

if "--child" in sys.argv: # executed in the child environment
    pprint(dict(os.environ))
else:
    python, script = quote(sys.executable), quote(sys.argv[0])
    os.execl("/bin/bash", "/bin/bash", "-c",
        "source the_script.sh; %s %s --child" % (python, script))

Update: 2019

"""
    Sometime you want to emulate the action of "source" in bash,
    settings some environment variables. Here is a way to do it.
"""
def shell_source( str_script, lst_filter ):
    #work around to allow variables with new lines
    #example MY_VAR='foon'
    #env -i create clean shell
    #bash -c run bash command
    #set -a optional include if you want to export both shell and enrivonment variables
    #env -0 seperates variables with null char instead of newline
    command = shlex.split(f"env -i bash -c 'set -a && source {str_script} && env -0'")

    pipe = subprocess.Popen( command, stdout=subprocess.PIPE )
    #pipe now outputs as byte, so turn it to utf string before parsing
    output = pipe.communicate()[0].decode('utf-8')
    #There was always a trailing empty line for me so im removing it. Delete this line if this is not happening for you.
    output = output[:-1]

    pre_filter_env = {}
    #split using null char
    for line in output.split('x00'):
        line = line.split( '=', 1)
        pre_filter_env[ line[0]] = line[1]

    post_filter_env = {}
    for str_var in lst_filter:
        post_filter_env[ str_var ] = pre_filter_env[ str_var ]

    os.environ.update( post_filter_env )

Using the answers here I created a solution that fit my needs.

• no need to filter out env variables
• allows for variables with new line characters

def shell_source(script):
    """
    Sometime you want to emulate the action of "source" in bash,
    settings some environment variables. Here is a way to do it.
    """
    
    pipe = subprocess.Popen(". %s && env -0" % script, stdout=subprocess.PIPE, shell=True)
    output = pipe.communicate()[0].decode('utf-8')
    output = output[:-1] # fix for index out for range in 'env[ line[0] ] = line[1]'

    env = {}
    # split using null char
    for line in output.split('x00'):
        line = line.split( '=', 1)
        # print(line)
        env[ line[0] ] = line[1]

    os.environ.update(env)

With this I was able to run commands with the same environment variables without issue:

def runCommand(command):
    """
    Prints and then runs shell command.
    """
    print(f'> running: {command}')
    stream = subprocess.Popen(command, shell=True,env=os.environ)
    (result_data, result_error) = stream.communicate()
    print(f'{result_data}, {result_error}')

Hope this helps anyone in the same position as me