Override the {…} notation so i get an OrderedDict() instead of a dict()?
Question:
Update: dicts retaining insertion order is guaranteed for Python 3.7+
I want to use a .py file like a config file.
So using the {...} notation I can create a dictionary using strings as keys but the definition order is lost in a standard python dictionary.
My question: is it possible to override the {...} notation so that I get an OrderedDict() instead of a dict()?
I was hoping that simply overriding dict constructor with OrderedDict (dict = OrderedDict) would work, but it doesn’t.
Eg:
dict = OrderedDict
dictname = {
'B key': 'value1',
'A key': 'value2',
'C key': 'value3'
}
print dictname.items()
Output:
[('B key', 'value1'), ('A key', 'value2'), ('C key', 'value3')]
Answers:
OrderedDict is not “standard python syntax”, however, an ordered set of key-value pairs (in standard python syntax) is simply:
[('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')]
To explicitly get an OrderedDict:
OrderedDict([('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')])
Another alternative, is to sort dictname.items(), if that’s all you need:
sorted(dictname.items())
What you are asking for is impossible, but if a config file in JSON syntax is sufficient you can do something similar with the json module:
>>> import json, collections
>>> d = json.JSONDecoder(object_pairs_hook = collections.OrderedDict)
>>> d.decode('{"a":5,"b":6}')
OrderedDict([(u'a', 5), (u'b', 6)])
To literally get what you are asking for, you have to fiddle with the syntax tree of your file. I don’t think it is advisable to do so, but I couldn’t resist the temptation to try. So here we go.
First, we create a module with a function my_execfile() that works like the built-in execfile(), except that all occurrences of dictionary displays, e.g. {3: 4, "a": 2} are replaced by explicit calls to the dict() constructor, e.g. dict([(3, 4), ('a', 2)]). (Of course we could directly replace them by calls to collections.OrderedDict(), but we don’t want to be too intrusive.) Here’s the code:
import ast
class DictDisplayTransformer(ast.NodeTransformer):
def visit_Dict(self, node):
self.generic_visit(node)
list_node = ast.List(
[ast.copy_location(ast.Tuple(list(x), ast.Load()), x[0])
for x in zip(node.keys, node.values)],
ast.Load())
name_node = ast.Name("dict", ast.Load())
new_node = ast.Call(ast.copy_location(name_node, node),
[ast.copy_location(list_node, node)],
[], None, None)
return ast.copy_location(new_node, node)
def my_execfile(filename, globals=None, locals=None):
if globals is None:
globals = {}
if locals is None:
locals = globals
node = ast.parse(open(filename).read())
transformed = DictDisplayTransformer().visit(node)
exec compile(transformed, filename, "exec") in globals, locals
With this modification in place, we can modify the behaviour of dictionary displays by overwriting dict. Here is an example:
# test.py
from collections import OrderedDict
print {3: 4, "a": 2}
dict = OrderedDict
print {3: 4, "a": 2}
Now we can run this file using my_execfile("test.py"), yielding the output
{'a': 2, 3: 4}
OrderedDict([(3, 4), ('a', 2)])
Note that for simplicity, the above code doesn’t touch dictionary comprehensions, which should be transformed to generator expressions passed to the dict() constructor. You’d need to add a visit_DictComp() method to the DictDisplayTransformer class. Given the above example code, this should be straight-forward.
Again, I don’t recommend this kind of messing around with the language semantics. Did you have a look into the ConfigParser module?
If what you are looking for is a way to get easy-to-use initialization syntax – consider creating a subclass of OrderedDict and adding operators to it that update the dict, for example:
from collections import OrderedDict
class OrderedMap(OrderedDict):
def __add__(self,other):
self.update(other)
return self
d = OrderedMap()+{1:2}+{4:3}+{"key":"value"}
d will be- OrderedMap([(1, 2), (4, 3), (‘key’,’value’)])
Another possible syntactic-sugar example using the slicing syntax:
class OrderedMap(OrderedDict):
def __getitem__(self, index):
if isinstance(index, slice):
self[index.start] = index.stop
return self
else:
return OrderedDict.__getitem__(self, index)
d = OrderedMap()[1:2][6:4][4:7]["a":"H"]
The one solution I found is to patch python itself, making the dict object remember the order of insertion.
This then works for all kind of syntaxes:
x = {'a': 1, 'b':2, 'c':3 }
y = dict(a=1, b=2, c=3)
etc.
I have taken the ordereddict C implementation from https://pypi.python.org/pypi/ruamel.ordereddict/ and merged back into the main python code.
If you do not mind re-building the python interpreter, here is a patch for Python 2.7.8:
https://github.com/fwyzard/cpython/compare/2.7.8…ordereddict-2.7.8.diff
.A
Here’s a hack that almost gives you the syntax you want:
class _OrderedDictMaker(object):
def __getitem__(self, keys):
if not isinstance(keys, tuple):
keys = (keys,)
assert all(isinstance(key, slice) for key in keys)
return OrderedDict([(k.start, k.stop) for k in keys])
ordereddict = _OrderedDictMaker()
from nastyhacks import ordereddict
menu = ordereddict[
"about" : "about",
"login" : "login",
'signup': "signup"
]
Edit: Someone else discovered this independently, and has published the odictliteral package on PyPI that provides a slightly more thorough implementation – use that package instead
As of python 3.6, all dictionaries will be ordered by default. For now, this is an implementation detail of dict and should not be relied upon, but it will likely become standard after v3.6.
Insertion order is always preserved in the new dict implementation:
>>>x = {'a': 1, 'b':2, 'c':3 }
>>>list(x.keys())
['a', 'b', 'c']
As of python 3.6 **kwargs order [PEP468] and class attribute order [PEP520] are preserved. The new compact, ordered dictionary implementation is used to implement the ordering for both of these.
Update: dicts retaining insertion order is guaranteed for Python 3.7+
I want to use a .py file like a config file.
So using the {...} notation I can create a dictionary using strings as keys but the definition order is lost in a standard python dictionary.
My question: is it possible to override the {...} notation so that I get an OrderedDict() instead of a dict()?
I was hoping that simply overriding dict constructor with OrderedDict (dict = OrderedDict) would work, but it doesn’t.
Eg:
dict = OrderedDict
dictname = {
'B key': 'value1',
'A key': 'value2',
'C key': 'value3'
}
print dictname.items()
Output:
[('B key', 'value1'), ('A key', 'value2'), ('C key', 'value3')]
OrderedDict is not “standard python syntax”, however, an ordered set of key-value pairs (in standard python syntax) is simply:
[('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')]
To explicitly get an OrderedDict:
OrderedDict([('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')])
Another alternative, is to sort dictname.items(), if that’s all you need:
sorted(dictname.items())
What you are asking for is impossible, but if a config file in JSON syntax is sufficient you can do something similar with the json module:
>>> import json, collections
>>> d = json.JSONDecoder(object_pairs_hook = collections.OrderedDict)
>>> d.decode('{"a":5,"b":6}')
OrderedDict([(u'a', 5), (u'b', 6)])
To literally get what you are asking for, you have to fiddle with the syntax tree of your file. I don’t think it is advisable to do so, but I couldn’t resist the temptation to try. So here we go.
First, we create a module with a function my_execfile() that works like the built-in execfile(), except that all occurrences of dictionary displays, e.g. {3: 4, "a": 2} are replaced by explicit calls to the dict() constructor, e.g. dict([(3, 4), ('a', 2)]). (Of course we could directly replace them by calls to collections.OrderedDict(), but we don’t want to be too intrusive.) Here’s the code:
import ast
class DictDisplayTransformer(ast.NodeTransformer):
def visit_Dict(self, node):
self.generic_visit(node)
list_node = ast.List(
[ast.copy_location(ast.Tuple(list(x), ast.Load()), x[0])
for x in zip(node.keys, node.values)],
ast.Load())
name_node = ast.Name("dict", ast.Load())
new_node = ast.Call(ast.copy_location(name_node, node),
[ast.copy_location(list_node, node)],
[], None, None)
return ast.copy_location(new_node, node)
def my_execfile(filename, globals=None, locals=None):
if globals is None:
globals = {}
if locals is None:
locals = globals
node = ast.parse(open(filename).read())
transformed = DictDisplayTransformer().visit(node)
exec compile(transformed, filename, "exec") in globals, locals
With this modification in place, we can modify the behaviour of dictionary displays by overwriting dict. Here is an example:
# test.py
from collections import OrderedDict
print {3: 4, "a": 2}
dict = OrderedDict
print {3: 4, "a": 2}
Now we can run this file using my_execfile("test.py"), yielding the output
{'a': 2, 3: 4}
OrderedDict([(3, 4), ('a', 2)])
Note that for simplicity, the above code doesn’t touch dictionary comprehensions, which should be transformed to generator expressions passed to the dict() constructor. You’d need to add a visit_DictComp() method to the DictDisplayTransformer class. Given the above example code, this should be straight-forward.
Again, I don’t recommend this kind of messing around with the language semantics. Did you have a look into the ConfigParser module?
If what you are looking for is a way to get easy-to-use initialization syntax – consider creating a subclass of OrderedDict and adding operators to it that update the dict, for example:
from collections import OrderedDict
class OrderedMap(OrderedDict):
def __add__(self,other):
self.update(other)
return self
d = OrderedMap()+{1:2}+{4:3}+{"key":"value"}
d will be- OrderedMap([(1, 2), (4, 3), (‘key’,’value’)])
Another possible syntactic-sugar example using the slicing syntax:
class OrderedMap(OrderedDict):
def __getitem__(self, index):
if isinstance(index, slice):
self[index.start] = index.stop
return self
else:
return OrderedDict.__getitem__(self, index)
d = OrderedMap()[1:2][6:4][4:7]["a":"H"]
The one solution I found is to patch python itself, making the dict object remember the order of insertion.
This then works for all kind of syntaxes:
x = {'a': 1, 'b':2, 'c':3 }
y = dict(a=1, b=2, c=3)
etc.
I have taken the ordereddict C implementation from https://pypi.python.org/pypi/ruamel.ordereddict/ and merged back into the main python code.
If you do not mind re-building the python interpreter, here is a patch for Python 2.7.8:
https://github.com/fwyzard/cpython/compare/2.7.8…ordereddict-2.7.8.diff
.A
Here’s a hack that almost gives you the syntax you want:
class _OrderedDictMaker(object):
def __getitem__(self, keys):
if not isinstance(keys, tuple):
keys = (keys,)
assert all(isinstance(key, slice) for key in keys)
return OrderedDict([(k.start, k.stop) for k in keys])
ordereddict = _OrderedDictMaker()
from nastyhacks import ordereddict
menu = ordereddict[
"about" : "about",
"login" : "login",
'signup': "signup"
]
Edit: Someone else discovered this independently, and has published the odictliteral package on PyPI that provides a slightly more thorough implementation – use that package instead
As of python 3.6, all dictionaries will be ordered by default. For now, this is an implementation detail of dict and should not be relied upon, but it will likely become standard after v3.6.
Insertion order is always preserved in the new dict implementation:
>>>x = {'a': 1, 'b':2, 'c':3 }
>>>list(x.keys())
['a', 'b', 'c']
As of python 3.6 **kwargs order [PEP468] and class attribute order [PEP520] are preserved. The new compact, ordered dictionary implementation is used to implement the ordering for both of these.