Setting options from environment variables when using argparse

One option is to check whether the environment variable is set, and to modify the calls to add_argument accordingly
e.g.

import argparse
import os

parser=argparse.ArgumentParser()
if 'CVSWEB_URL' in os.environ:
    cvsopt = { 'default': os.environ['CVSWEB_URL'] }
else:
    cvsopt = { 'required': True }
parser.add_argument(
    "-u", "--cvsurl", help="Specify url (overrides CVSWEB_URL environment variable)", 
    **cvsopt)
args=parser.parse_args()

You can set the default= of the argument to a .get() of os.environ with the environment variable you want to grab.

You can also pass a 2nd argument in the .get() call, which is the default value if .get() doesn’t find an environment variable by that name (by default .get() returns None in that case).

import argparse
import os

parser = argparse.ArgumentParser(description='test')
parser.add_argument('--url', default=os.environ.get('URL'))

args = parser.parse_args()
if not args.url:
    exit(parser.print_usage())

Thought I’d post my solution as the original question/answer gave me a lot of help.

My problem is a little different to Russell’s. I’m using OptionParser and instead of an environmental variable for each argument I have just one which simulates the command line.

i.e.

MY_ENVIRONMENT_ARGS = –arg1 “Maltese” –arg2 “Falcon” -r “1930” -h

Solution:

def set_defaults_from_environment(oparser):

    if 'MY_ENVIRONMENT_ARGS' in os.environ:

        environmental_args = os.environ[ 'MY_ENVIRONMENT_ARGS' ].split()

        opts, _ = oparser.parse_args( environmental_args )

        oparser.defaults = opts.__dict__

oparser = optparse.OptionParser()
oparser.add_option('-a', '--arg1', action='store', default="Consider")
oparser.add_option('-b', '--arg2', action='store', default="Phlebas")
oparser.add_option('-r', '--release', action='store', default='1987')
oparser.add_option('-h', '--hardback', action='store_true', default=False)

set_defaults_from_environment(oparser)

options, _ = oparser.parse_args(sys.argv[1:])

Here I don’t throw an error if an argument is not found. But if I wish to I could just do something like

for key in options.__dict__:
    if options.__dict__[key] is None:
        # raise error/log problem/print to console/etc

The topic is quite old, but I had similar problem and I thought I would share my solution with you. Unfortunately custom action solution suggested by @Russell Heilling doesn’t work for me for couple of reasons:

• It prevents me from using predefined actions (like store_true)
• I would rather like it to fallback to default when envvar is not in os.environ (that could be easily fixed)
• I would like to have this behaviour for all of my arguments without specifying action or envvar (which should always be action.dest.upper())

Here’s my solution (in Python 3):

class CustomArgumentParser(argparse.ArgumentParser):
    class _CustomHelpFormatter(argparse.ArgumentDefaultsHelpFormatter):
        def _get_help_string(self, action):
            help = super()._get_help_string(action)
            if action.dest != 'help':
                help += ' [env: {}]'.format(action.dest.upper())
            return help

    def __init__(self, *, formatter_class=_CustomHelpFormatter, **kwargs):
        super().__init__(formatter_class=formatter_class, **kwargs)

    def _add_action(self, action):
        action.default = os.environ.get(action.dest.upper(), action.default)
        return super()._add_action(action)

ConfigArgParse adds support for environment variables to argparse, so you can do things like:

p = configargparse.ArgParser()
p.add('-m', '--moo', help='Path of cow', env_var='MOO_PATH') 
options = p.parse_args()

I usually have to do this for multiple arguments (authentication and API keys).. this is simple and straight forward. Uses **kwargs.

def environ_or_required(key):
    return (
        {'default': os.environ.get(key)} if os.environ.get(key)
        else {'required': True}
    )

parser.add_argument('--thing', **environ_or_required('THING'))

There is an example use-case for ChainMap where you merge together defaults, environment variables and command line arguments.

import os, argparse

defaults = {'color': 'red', 'user': 'guest'}

parser = argparse.ArgumentParser()
parser.add_argument('-u', '--user')
parser.add_argument('-c', '--color')
namespace = parser.parse_args()
command_line_args = {k:v for k, v in vars(namespace).items() if v}

combined = ChainMap(command_line_args, os.environ, defaults)

Came to me from a great talk about beautiful and idiomatic python.

However, I’m not sure how to go about the difference of lower- and uppercase dictionary keys. In the case where both -u foobar is passed as an argument and environment is set to USER=bazbaz, the combined dictionary will look like {'user': 'foobar', 'USER': 'bazbaz'}.

Here’s a relatively simple (looks longer because it’s well-commented) yet complete solution that avoids kludging default by using the namespace argument of parse_args. By default it parses environment variables no differently than command-line arguments though that can easily be changed.

import shlex

# Notes:
#   * Based on https://github.com/python/cpython/blob/
#               15bde92e47e824369ee71e30b07f1624396f5cdc/
#               Lib/argparse.py
#   * Haven't looked into handling "required" for mutually exclusive groups
#   * Probably should make new attributes private even though it's ugly.
class EnvArgParser(argparse.ArgumentParser):
    # env_k:    The keyword to "add_argument" as well as the attribute stored
    #           on matching actions.
    # env_f:    The keyword to "add_argument". Defaults to "env_var_parse" if
    #           not provided.
    # env_i:    Basic container type to identify unfilled arguments.
    env_k = "env_var"
    env_f = "env_var_parse"
    env_i = type("env_i", (object,), {})

    def add_argument(self, *args, **kwargs):
        map_f = (lambda m,k,f=None,d=False:
                    (k, k in m, m.pop(k,f) if d else m.get(k,f)))

        env_k = map_f(kwargs, self.env_k, d=True, f="")
        env_f = map_f(kwargs, self.env_f, d=True, f=self.env_var_parse)

        if env_k[1] and not isinstance(env_k[2], str):
            raise ValueError(f"Parameter '{env_k[0]}' must be a string.")

        if env_f[1] and not env_k[1]:
            raise ValueError(f"Parameter '{env_f[0]}' requires '{env_k[0]}'.")

        if env_f[1] and not callable(env_f[2]):
            raise ValueError(f"Parameter '{env_f[0]}' must be callable.")

        action = super().add_argument(*args, **kwargs)

        if env_k[1] and not action.option_strings:
            raise ValueError(f"Positional parameters may not specify '{env_k[0]}'.")

        # We can get the environment now:
        #   * We need to know now if the keys exist anyway
        #   * os.environ is static
        env_v = map_f(os.environ, env_k[2], f="")

        # Examples:
        # env_k:
        #   ("env_var", True,  "FOO_KEY")
        # env_v:
        #   ("FOO_KEY", False, "")
        #   ("FOO_KEY", True,  "FOO_VALUE")
        #
        # env_k:
        #   ("env_var", False, "")
        # env_v:
        #   (""       , False, "")
        #   ("",        True,  "RIDICULOUS_VALUE")

        # Add the identifier to all valid environment variable actions for
        # later access by i.e. the help formatter.
        if env_k[1]:
            if env_v[1] and action.required:
                action.required = False
            i = self.env_i()
            i.a = action
            i.k = env_k[2]
            i.f = env_f[2]
            i.v = env_v[2]
            i.p = env_v[1]
            setattr(action, env_k[0], i)

        return action

    # Overriding "_parse_known_args" is better than "parse_known_args":
    #   * The namespace will already have been created.
    #   * This method runs in an exception handler.
    def _parse_known_args(self, arg_strings, namespace):
        """precedence: cmd args > env var > preexisting namespace > defaults"""

        for action in self._actions:
            if action.dest is argparse.SUPPRESS:
                continue
            try:
                i = getattr(action, self.env_k)
            except AttributeError:
                continue
            if not i.p:
                continue
            setattr(namespace, action.dest, i)

        namespace, arg_extras = super()._parse_known_args(arg_strings, namespace)

        for k,v in vars(namespace).copy().items():
            # Setting "env_i" on the action is more effective than using an
            # empty unique object() and mapping namespace attributes back to
            # actions.
            if isinstance(v, self.env_i):
                fv = v.f(v.a, v.k, v.v, arg_extras)
                if fv is argparse.SUPPRESS:
                    delattr(namespace, k)
                else:
                    # "_parse_known_args::take_action" checks for action
                    # conflicts. For simplicity we don't.
                    v.a(self, namespace, fv, v.k)

        return (namespace, arg_extras)

    def env_var_parse(self, a, k, v, e):
        # Use shlex, yaml, whatever.
        v = shlex.split(v)

        # From "_parse_known_args::consume_optional".
        n = self._match_argument(a, "A"*len(v))

        # From the main loop of "_parse_known_args". Treat additional
        # environment variable arguments just like additional command-line
        # arguments (which will eventually raise an exception).
        e.extend(v[n:])

        return self._get_values(a, v[:n])


# Derived from "ArgumentDefaultsHelpFormatter".
class EnvArgHelpFormatter(argparse.HelpFormatter):
    """Help message formatter which adds environment variable keys to
    argument help.
    """

    env_k = EnvArgParser.env_k

    # This is supposed to return a %-style format string for "_expand_help".
    # Since %-style strings don't support attribute access we instead expand
    # "env_k" ourselves.
    def _get_help_string(self, a):
        h = super()._get_help_string(a)
        try:
            i = getattr(a, self.env_k)
        except AttributeError:
            return h
        s = f" ({self.env_k}: {i.k})"
        if s not in h:
            h += s
        return h


# An example mix-in.
class DefEnvArgHelpFormatter
        ( EnvArgHelpFormatter
        , argparse.ArgumentDefaultsHelpFormatter
        ):
    pass

Example program:

parser = EnvArgParser
        ( prog="Test Program"
        , formatter_class=DefEnvArgHelpFormatter
        )

parser.add_argument
        ( '--bar'
        , required=True
        , env_var="BAR"
        , type=int
        , nargs="+"
        , default=22
        , help="Help message for bar."
        )

parser.add_argument
        ( 'baz'
        , type=int
        )

args = parser.parse_args()
print(args)

Example program output:

$ BAR="1 2 3 '45  ' 6 7" ./envargparse.py 123
Namespace(bar=[1, 2, 3, 45, 6, 7], baz=123)

$ ./envargparse.py -h
usage: Test Program [-h] --bar BAR [BAR ...] baz

positional arguments:
  baz

optional arguments:
  -h, --help           show this help message and exit
  --bar BAR [BAR ...]  Help message for bar. (default: 22) (env_var: BAR)

You can use OptionParser()

from optparse import OptionParser

def argument_parser(self, parser):
    parser.add_option('--foo', dest="foo", help="foo", default=os.environ.get('foo', None))
    parser.add_option('--bar', dest="bar", help="bar", default=os.environ.get('bar', None))
    return(parser.parse_args())

parser = OptionParser()
(options, args) = argument_parser(parser)
foo = options.foo
bar = options.bar
print("foo: {}".format(foo))
print("bar: {}".format(bar))

shell:

export foo=1
export bar=2
python3 script.py

The Click library handles this explicitly:

import click

@click.command()
@click.argument('src', envvar='SRC', type=click.File('r'))
def echo(src):
    """Print value of SRC environment variable."""
    click.echo(src.read())

And from the command line:

$ export SRC=hello.txt
$ echo
Hello World!

https://click.palletsprojects.com/en/master/arguments/#environment-variables

You can install it with

pip install click

Another option:

    parser = argparse.ArgumentParser()
    env = os.environ
    def add_argument(key, *args, **kwargs):
        if key in env:
            kwargs['default'] = env[key]
        parser.add_argument(*args, **kwargs)

    add_argument('--type', type=str)

Or this one, using os.getenv for setting default value:

parser = argparse.ArgumentParser()
parser.add_argument('--type', type=int, default=os.getenv('type',100))