Python configuration file: Any file format recommendation? INI format still appropriate? Seems quite old school
Question:
I need to store configurations (key/value) for a Python application and I am searching for the best way to store these configurations in a file.
I run into Python’s ConfigParser and I wondered if the INI file format is really still appropriate nowadays?!
Does there exist a more up-to-date format or is INI still the recommended way to go? (XML, JSON, …)
Please share your opinions/recommendations…
Answers:
This entirely depends on your requirements. If (as you say) all you need is key/value pairs, ini files (or other “plain” config files) will perfectly suit you. No, they are not outdated, as they are still in use.
XML/JSON is perfect if you have hierarchical structures and also want to use more sophisticated methods (e.g: XML file validation, namespaces, etc.).
It depends on how the config file will be used.
One of the advantages of INI files is that they are really easy to read and understand. It’s much easier to make a mistake in JSON or XML file if you edit config by hand. PHP still uses INI files.
However, if you your config is not meant to be edited by hand, go with any format you like, because INI is not the easiest one to parse.
Consider using plain Python files as configuration files.
An example (config.py):
# use normal python comments
value1 = 32
value2 = "A string value"
value3 = ["lists", "are", "handy"]
value4 = {"and": "so", "are": "dictionaries"}
In your program, load the config file using exec (docs):
from pathlib import Path
if __name__ == "__main__":
config = {}
exec(Path("config.py").read_text(encoding="utf8"), {}, config)
print(config["value1"])
print(config["value4"])
I like this approach, for the following reasons:
- In the simple case, the format is as easy to author as an INI-style config file. It also shares an important characteristic with INI files: it is very suitable for version control (this is less true for XML and maybe also for JSON)
- I like the flexibility that comes with having the config file in an actual programming language.
The approach is widely used, a few examples:
- A Django site’s settings lives inside settings.py. Django does not use
execfile, it uses import to read/execute settings.py AFAIK, but the end result is the same: the code inside the settings file is executed.
- The bash shell reads and executes
~/.bashrc on startup.
- The Python interpreter imports
site.py on startup.
Dictionaries are pretty popular as well. Basically a hash table.
{"one": 1, "two": 2} is an example, kind of looks like json.
Then you can call it up like mydict["one"], which would return 1.
Then you can use shelve to save the dictionary to a file:
mydict = shelve.open(filename)
# then you can call it from there, like
mydict["one"]
So, it’s somewhat easier then an ini file. You can add stuff just like a list or change options pretty easily and then once you close it, it will write it back out.
Heres a simple example of what i mean:
import shelve
def main():
mydict = shelve.open("testfile")
mydict["newKey"] = value("some comment", 5)
print(mydict["newKey"].value)
print(mydict["newKey"].comment)
mydict.close()
class value():
def __init__(self, comment, value):
self.comment = comment
self.value = value
if __name__ == '__main__':
main()
For completeness, you can also use a shell-style configuration format with the help of the “shlex” module. If you have a fixed set of configuration parameters then you can combine it with the “optparse” module.
from optparse import OptionParser
_o = OptionParser("%prog [options] configfiles...")
_o.add_option("--hostname", metavar="HOSTNAME", default="10.0.0.1")
_o.add_option("--username", metavar="USERNAME", default="admin")
_o.add_option("--password", metavar="PASSWORD", default="admin")
import shlex
def parse(filename, defaults):
opt, args = _o.parse_args(shlex.split(open(filename).read()), defaults)
return opt
if __name__ == "__main__":
import sys
values, args = _o.parse_args()
for arg in args:
values = parse(arg, values)
values, args = _o.parse_args(values = values)
for name in _o.defaults:
print name, "=", getattr(values, name)
The example shows how you can chain ini files for having a default set of values and user-defined redefinitions. So assume you have two files containing
file1.ini:
--hostname 10.2.3.4
--password admin-sc
file2.ini:
--username "foo bar"
--password "special key"
Then you can run ./configtest.py file1.ini file2.ini --password other and the resulting values will have hostname as 10.2.3.4 and username as “foo bar” and password as “other”. This variation for configuration settings comes in handy if you do already have an optparse-definition for your program parameters -> just reuse it and you can chain the values from the command line with the values from a config file and possibly some global config settings.
As an incentive, your configuration parameters are always documented and mistyped configuration parameters will come up early as an error, just as you can use the optparse-instance to precheck your default settings file (schema check). As a downside, there are no comments allowed in the ini and configuration elements are not easily substructured.Still your parser is essentially a one-liner.
I ran into same problem more than once. I love the idea of python file(s) as settings files, it’s just simple, elegant and all fellow python devs like it. Same time not really convinced with execfile idea.
So I went ahead and created Converge.
It supports some advanced options but at it’s heart it simple python module as settings file.
It’s as simple as
- creating
default_settings.py, prod_settings.py
pip install convergeimport settings
I need to store configurations (key/value) for a Python application and I am searching for the best way to store these configurations in a file.
I run into Python’s ConfigParser and I wondered if the INI file format is really still appropriate nowadays?!
Does there exist a more up-to-date format or is INI still the recommended way to go? (XML, JSON, …)
Please share your opinions/recommendations…
This entirely depends on your requirements. If (as you say) all you need is key/value pairs, ini files (or other “plain” config files) will perfectly suit you. No, they are not outdated, as they are still in use.
XML/JSON is perfect if you have hierarchical structures and also want to use more sophisticated methods (e.g: XML file validation, namespaces, etc.).
It depends on how the config file will be used.
One of the advantages of INI files is that they are really easy to read and understand. It’s much easier to make a mistake in JSON or XML file if you edit config by hand. PHP still uses INI files.
However, if you your config is not meant to be edited by hand, go with any format you like, because INI is not the easiest one to parse.
Consider using plain Python files as configuration files.
An example (config.py):
# use normal python comments
value1 = 32
value2 = "A string value"
value3 = ["lists", "are", "handy"]
value4 = {"and": "so", "are": "dictionaries"}
In your program, load the config file using exec (docs):
from pathlib import Path
if __name__ == "__main__":
config = {}
exec(Path("config.py").read_text(encoding="utf8"), {}, config)
print(config["value1"])
print(config["value4"])
I like this approach, for the following reasons:
- In the simple case, the format is as easy to author as an INI-style config file. It also shares an important characteristic with INI files: it is very suitable for version control (this is less true for XML and maybe also for JSON)
- I like the flexibility that comes with having the config file in an actual programming language.
The approach is widely used, a few examples:
- A Django site’s settings lives inside settings.py. Django does not use
execfile, it usesimportto read/executesettings.pyAFAIK, but the end result is the same: the code inside the settings file is executed. - The bash shell reads and executes
~/.bashrcon startup. - The Python interpreter imports
site.pyon startup.
Dictionaries are pretty popular as well. Basically a hash table.
{"one": 1, "two": 2} is an example, kind of looks like json.
Then you can call it up like mydict["one"], which would return 1.
Then you can use shelve to save the dictionary to a file:
mydict = shelve.open(filename)
# then you can call it from there, like
mydict["one"]
So, it’s somewhat easier then an ini file. You can add stuff just like a list or change options pretty easily and then once you close it, it will write it back out.
Heres a simple example of what i mean:
import shelve
def main():
mydict = shelve.open("testfile")
mydict["newKey"] = value("some comment", 5)
print(mydict["newKey"].value)
print(mydict["newKey"].comment)
mydict.close()
class value():
def __init__(self, comment, value):
self.comment = comment
self.value = value
if __name__ == '__main__':
main()
For completeness, you can also use a shell-style configuration format with the help of the “shlex” module. If you have a fixed set of configuration parameters then you can combine it with the “optparse” module.
from optparse import OptionParser
_o = OptionParser("%prog [options] configfiles...")
_o.add_option("--hostname", metavar="HOSTNAME", default="10.0.0.1")
_o.add_option("--username", metavar="USERNAME", default="admin")
_o.add_option("--password", metavar="PASSWORD", default="admin")
import shlex
def parse(filename, defaults):
opt, args = _o.parse_args(shlex.split(open(filename).read()), defaults)
return opt
if __name__ == "__main__":
import sys
values, args = _o.parse_args()
for arg in args:
values = parse(arg, values)
values, args = _o.parse_args(values = values)
for name in _o.defaults:
print name, "=", getattr(values, name)
The example shows how you can chain ini files for having a default set of values and user-defined redefinitions. So assume you have two files containing
file1.ini:
--hostname 10.2.3.4
--password admin-sc
file2.ini:
--username "foo bar"
--password "special key"
Then you can run ./configtest.py file1.ini file2.ini --password other and the resulting values will have hostname as 10.2.3.4 and username as “foo bar” and password as “other”. This variation for configuration settings comes in handy if you do already have an optparse-definition for your program parameters -> just reuse it and you can chain the values from the command line with the values from a config file and possibly some global config settings.
As an incentive, your configuration parameters are always documented and mistyped configuration parameters will come up early as an error, just as you can use the optparse-instance to precheck your default settings file (schema check). As a downside, there are no comments allowed in the ini and configuration elements are not easily substructured.Still your parser is essentially a one-liner.
I ran into same problem more than once. I love the idea of python file(s) as settings files, it’s just simple, elegant and all fellow python devs like it. Same time not really convinced with execfile idea.
So I went ahead and created Converge.
It supports some advanced options but at it’s heart it simple python module as settings file.
It’s as simple as
- creating
default_settings.py,prod_settings.py pip install convergeimport settings