Which is the best way to allow configuration options be overridden at the command line in Python?
Question:
I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.
Now I would like to use ‘proper’ configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.
I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.
Any ideas how to do this in an easy way? I don’t really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values…
Answers:
I can’t say it’s the best way, but I have an OptionParser class that I made that does just that – acts like optparse.OptionParser with defaults coming from a config file section. You can have it…
class OptionParser(optparse.OptionParser):
def __init__(self, **kwargs):
import sys
import os
config_file = kwargs.pop('config_file',
os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
self.config_section = kwargs.pop('config_section', 'OPTIONS')
self.configParser = ConfigParser()
self.configParser.read(config_file)
optparse.OptionParser.__init__(self, **kwargs)
def add_option(self, *args, **kwargs):
option = optparse.OptionParser.add_option(self, *args, **kwargs)
name = option.get_opt_string()
if name.startswith('--'):
name = name[2:]
if self.configParser.has_option(self.config_section, name):
self.set_default(name, self.configParser.get(self.config_section, name))
Feel free to browse the source. Tests are in a sibling directory.
I’m using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:
subp.set_defaults(**dict(conffile.items(subn)))
This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.
A more complete example is below:
####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost
import ConfigParser
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')
parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')
conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')
for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
subp.set_defaults(**dict(conffile.items(subn)))
print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')
I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:
Default with no user input:
$ ./argparse-partial.py
Option is "default"
Default from configuration file:
$ cat argparse-partial.config
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config
Option is "Hello world!"
Default from configuration file, overridden by commandline:
$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"
argprase-partial.py follows. It is slightly complicated to handle -h for help properly.
import argparse
import ConfigParser
import sys
def main(argv=None):
# Do argv default this way, as doing it in the functional
# declaration sets it at compile time.
if argv is None:
argv = sys.argv
# Parse any conf_file specification
# We make this parser with add_help=False so that
# it doesn't parse -h and print help.
conf_parser = argparse.ArgumentParser(
description=__doc__, # printed with -h/--help
# Don't mess with format of description
formatter_class=argparse.RawDescriptionHelpFormatter,
# Turn off help, so we print all options in response to -h
add_help=False
)
conf_parser.add_argument("-c", "--conf_file",
help="Specify config file", metavar="FILE")
args, remaining_argv = conf_parser.parse_known_args()
defaults = { "option":"default" }
if args.conf_file:
config = ConfigParser.SafeConfigParser()
config.read([args.conf_file])
defaults.update(dict(config.items("Defaults")))
# Parse rest of arguments
# Don't suppress add_help here so it will handle -h
parser = argparse.ArgumentParser(
# Inherit options from config_parser
parents=[conf_parser]
)
parser.set_defaults(**defaults)
parser.add_argument("--option")
args = parser.parse_args(remaining_argv)
print "Option is "{}"".format(args.option)
return(0)
if __name__ == "__main__":
sys.exit(main())
Check out ConfigArgParse – its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.
Try to this way
# encoding: utf-8
import imp
import argparse
class LoadConfigAction(argparse._StoreAction):
NIL = object()
def __init__(self, option_strings, dest, **kwargs):
super(self.__class__, self).__init__(option_strings, dest)
self.help = "Load configuration from file"
def __call__(self, parser, namespace, values, option_string=None):
super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)
config = imp.load_source('config', values)
for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
setattr(namespace, key, getattr(config, key))
Use it:
parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")
And create example config:
# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")
Update: This answer still has issues; for example, it cannot handle required arguments, and requires an awkward config syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent, drop-in replacement.
One issue with the current is that it will not error if the arguments in the config file are invalid. Here’s a version with a different downside: you’ll need to include the -- or - prefix in the keys.
Here’s the python code (Gist link with MIT license):
# Filename: main.py
import argparse
import configparser
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--config_file', help='config file')
args, left_argv = parser.parse_known_args()
if args.config_file:
with open(args.config_file, 'r') as f:
config = configparser.SafeConfigParser()
config.read([args.config_file])
parser.add_argument('--arg1', help='argument 1')
parser.add_argument('--arg2', type=int, help='argument 2')
for k, v in config.items("Defaults"):
parser.parse_args([str(k), str(v)], args)
parser.parse_args(left_argv, args)
print(args)
Here’s an example of a config file:
# Filename: config_correct.conf
[Defaults]
--arg1=Hello!
--arg2=3
Now, running
> python main.py --config_file config_correct.conf --arg1 override
Namespace(arg1='override', arg2=3, config_file='test_argparse.conf')
However, if our config file has an error:
# config_invalid.conf
--arg1=Hello!
--arg2='not an integer!'
Running the script will produce an error, as desired:
> python main.py --config_file config_invalid.conf --arg1 override
usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1]
[--arg2 ARG2]
main.py: error: argument --arg2: invalid int value: 'not an integer!'
The main downside is that this uses parser.parse_args somewhat hackily in order to obtain the error checking from ArgumentParser, but I am not aware of any alternatives to this.
fromfile_prefix_chars
Maybe not the perfect API, but worth knowing about.
main.py
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())
Then:
$ printf -- '-an1n-bn2n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py @opts.txt
Namespace(a='1', b='2')
$ ./main.py @opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 @opts.txt
Namespace(a='1', b='2')
Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars
Tested on Python 3.6.5, Ubuntu 18.04.
You can use ChainMap
A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.
You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.
import os
from collections import ChainMap, defaultdict
options = ChainMap(command_line_options, os.environ, config_file_options,
defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']
print(value, value2)
'optvalue', 'default-value'
parse_args() can take an existing Namespace and merge the existing Namespace with args/options it’s currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:
foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')
ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')
args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')
# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')
I’ve mocked it up for a real config file option. I’m parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.
I have this very simple JSON config file, config.ini:
[DEFAULT]
delimiter = |
and when I run this:
import argparse
import configparser
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')
# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()
# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()
if pre_args.config_file:
config = configparser.ConfigParser()
config.read(pre_args.config_file)
for name, val in config['DEFAULT'].items():
setattr(ConfigNamespace, name, val)
# Parse cmd-line args again, merging with ConfigNamespace,
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)
print(args)
with various cmd-line settings, I get:
./main.py
Namespace(config_file=None, delimiter=',')
./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')
./main.py -c config.ini -d ;
Namespace(config_file='config.ini', delimiter=';')
I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.
Now I would like to use ‘proper’ configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.
I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.
Any ideas how to do this in an easy way? I don’t really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values…
I can’t say it’s the best way, but I have an OptionParser class that I made that does just that – acts like optparse.OptionParser with defaults coming from a config file section. You can have it…
class OptionParser(optparse.OptionParser):
def __init__(self, **kwargs):
import sys
import os
config_file = kwargs.pop('config_file',
os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
self.config_section = kwargs.pop('config_section', 'OPTIONS')
self.configParser = ConfigParser()
self.configParser.read(config_file)
optparse.OptionParser.__init__(self, **kwargs)
def add_option(self, *args, **kwargs):
option = optparse.OptionParser.add_option(self, *args, **kwargs)
name = option.get_opt_string()
if name.startswith('--'):
name = name[2:]
if self.configParser.has_option(self.config_section, name):
self.set_default(name, self.configParser.get(self.config_section, name))
Feel free to browse the source. Tests are in a sibling directory.
I’m using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:
subp.set_defaults(**dict(conffile.items(subn)))
This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.
A more complete example is below:
####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost
import ConfigParser
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')
parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')
conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')
for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
subp.set_defaults(**dict(conffile.items(subn)))
print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')
I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:
Default with no user input:
$ ./argparse-partial.py
Option is "default"
Default from configuration file:
$ cat argparse-partial.config
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config
Option is "Hello world!"
Default from configuration file, overridden by commandline:
$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"
argprase-partial.py follows. It is slightly complicated to handle -h for help properly.
import argparse
import ConfigParser
import sys
def main(argv=None):
# Do argv default this way, as doing it in the functional
# declaration sets it at compile time.
if argv is None:
argv = sys.argv
# Parse any conf_file specification
# We make this parser with add_help=False so that
# it doesn't parse -h and print help.
conf_parser = argparse.ArgumentParser(
description=__doc__, # printed with -h/--help
# Don't mess with format of description
formatter_class=argparse.RawDescriptionHelpFormatter,
# Turn off help, so we print all options in response to -h
add_help=False
)
conf_parser.add_argument("-c", "--conf_file",
help="Specify config file", metavar="FILE")
args, remaining_argv = conf_parser.parse_known_args()
defaults = { "option":"default" }
if args.conf_file:
config = ConfigParser.SafeConfigParser()
config.read([args.conf_file])
defaults.update(dict(config.items("Defaults")))
# Parse rest of arguments
# Don't suppress add_help here so it will handle -h
parser = argparse.ArgumentParser(
# Inherit options from config_parser
parents=[conf_parser]
)
parser.set_defaults(**defaults)
parser.add_argument("--option")
args = parser.parse_args(remaining_argv)
print "Option is "{}"".format(args.option)
return(0)
if __name__ == "__main__":
sys.exit(main())
Check out ConfigArgParse – its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.
Try to this way
# encoding: utf-8
import imp
import argparse
class LoadConfigAction(argparse._StoreAction):
NIL = object()
def __init__(self, option_strings, dest, **kwargs):
super(self.__class__, self).__init__(option_strings, dest)
self.help = "Load configuration from file"
def __call__(self, parser, namespace, values, option_string=None):
super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)
config = imp.load_source('config', values)
for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
setattr(namespace, key, getattr(config, key))
Use it:
parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")
And create example config:
# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")
Update: This answer still has issues; for example, it cannot handle required arguments, and requires an awkward config syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent, drop-in replacement.
One issue with the current is that it will not error if the arguments in the config file are invalid. Here’s a version with a different downside: you’ll need to include the -- or - prefix in the keys.
Here’s the python code (Gist link with MIT license):
# Filename: main.py
import argparse
import configparser
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--config_file', help='config file')
args, left_argv = parser.parse_known_args()
if args.config_file:
with open(args.config_file, 'r') as f:
config = configparser.SafeConfigParser()
config.read([args.config_file])
parser.add_argument('--arg1', help='argument 1')
parser.add_argument('--arg2', type=int, help='argument 2')
for k, v in config.items("Defaults"):
parser.parse_args([str(k), str(v)], args)
parser.parse_args(left_argv, args)
print(args)
Here’s an example of a config file:
# Filename: config_correct.conf
[Defaults]
--arg1=Hello!
--arg2=3
Now, running
> python main.py --config_file config_correct.conf --arg1 override
Namespace(arg1='override', arg2=3, config_file='test_argparse.conf')
However, if our config file has an error:
# config_invalid.conf
--arg1=Hello!
--arg2='not an integer!'
Running the script will produce an error, as desired:
> python main.py --config_file config_invalid.conf --arg1 override
usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1]
[--arg2 ARG2]
main.py: error: argument --arg2: invalid int value: 'not an integer!'
The main downside is that this uses parser.parse_args somewhat hackily in order to obtain the error checking from ArgumentParser, but I am not aware of any alternatives to this.
fromfile_prefix_chars
Maybe not the perfect API, but worth knowing about.
main.py
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())
Then:
$ printf -- '-an1n-bn2n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py @opts.txt
Namespace(a='1', b='2')
$ ./main.py @opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 @opts.txt
Namespace(a='1', b='2')
Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars
Tested on Python 3.6.5, Ubuntu 18.04.
You can use ChainMap
A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.
You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.
import os
from collections import ChainMap, defaultdict
options = ChainMap(command_line_options, os.environ, config_file_options,
defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']
print(value, value2)
'optvalue', 'default-value'
parse_args() can take an existing Namespace and merge the existing Namespace with args/options it’s currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:
foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')
ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')
args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')
# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')
I’ve mocked it up for a real config file option. I’m parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.
I have this very simple JSON config file, config.ini:
[DEFAULT]
delimiter = |
and when I run this:
import argparse
import configparser
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')
# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()
# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()
if pre_args.config_file:
config = configparser.ConfigParser()
config.read(pre_args.config_file)
for name, val in config['DEFAULT'].items():
setattr(ConfigNamespace, name, val)
# Parse cmd-line args again, merging with ConfigNamespace,
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)
print(args)
with various cmd-line settings, I get:
./main.py
Namespace(config_file=None, delimiter=',')
./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')
./main.py -c config.ini -d ;
Namespace(config_file='config.ini', delimiter=';')