Is the order of a Python dictionary guaranteed over iterations?
Question:
I’m currently implementing a complex microbial food-web in Python using SciPy.integrate.ode. I need the ability to easily add species and reactions to the system, so I have to code up something quite general. My scheme looks something like this:
class Reaction(object):
def __init__(self):
#stuff common to all reactions
def __getReactionRate(self, **kwargs):
raise NotImplementedError
... Reaction subclasses that
... implement specific types of reactions
class Species(object):
def __init__(self, reactionsDict):
self.reactionsDict = reactionsDict
#reactionsDict looks like {'ReactionName':reactionObject, ...}
#stuff common to all species
def sumOverAllReactionsForThisSpecies(self, **kwargs):
#loop over all the reactions and return the
#cumulative change in the concentrations of all solutes
...Species subclasses where for each species
... are defined and passed to the superclass constructor
class FermentationChamber(object):
def __init__(self, speciesList, timeToSolve, *args):
#do initialization
def step(self):
#loop over each species, which in turn loops
#over each reaction inside it and return a
#cumulative dictionary of total change for each
#solute in the whole system
if __name__==__main__:
f = FermentationChamber(...)
o = ode(...) #initialize ode solver
while o.successful() and o.t<timeToSolve:
o.integrate()
#process o.t and o.y (o.t contains the time points
#and o.y contains the solution matrix)
So, the question is, when I iterate over the dictionaries in Species.sumOverAllReactionsForThisSpecies() and FermentationChamber.step(), is the iteration order of the dictionaries guaranteed to be the same if no elements are added or removed from the dictionaries between the first and the last iteration? That is, can I assume that the order of the numpy array created at each iteration from the dictionary will not vary? For example, if a dictionary has the format {‘Glucose’:10, ‘Fructose’:12}, if an Array created from this dictionary will always have the same order (it doesn’t matter what that order is, as long as it’s deterministic).
Sorry for the mega-post, I just wanted to let you know where I’m coming from.
Answers:
Yes, the same order is guaranteed if it is not modified.
See the docs here.
Edit:
Regarding if changing the value (but not adding/removing a key) will affect the order, this is what the comments in the C-source says:
/* CAUTION: PyDict_SetItem() must guarantee that it won't resize the
* dictionary if it's merely replacing the value for an existing key.
* This means that it's safe to loop over a dictionary with PyDict_Next()
* and occasionally replace a value -- but you can't insert new keys or
* remove them.
*/
It seems that its not an implementation detail, but a requirement of the language.
Provided no modifications are made to the dictionary, the answer is yes. See the docs here.
However, dictionaries are unordered by nature in Python. In general, it’s not the best practice to rely on dictionaries for sensitive sorted data.
An example of an a more robust solution would be Django’s SortedDict data structure.
If you want the order to be consistent, I would do something to force a particular order. Although you might be able to convince yourself that the order is guaranteed, and you might be right, it seems fragile to me, and it will be mysterious to other developers.
For example, you emphasize always in your question. Is it important that it be the same order in Python 2.5 and 2.6? 2.6 and 3.1? CPython and Jython? I wouldn’t count on those.
I also would recommend not relying on the fact the dictionaries order is non-random.
If you want a built in solution to sorting you dictionary read http://www.python.org/dev/peps/pep-0265/
Here is the most relevant material:
This PEP is rejected because the need for it has been largely
fulfilled by Py2.4’s sorted() builtin function:
>>> sorted(d.iteritems(), key=itemgetter(1), reverse=True)
[('b', 23), ('d', 17), ('c', 5), ('a', 2), ('e', 1)]
or for just the keys:
>>> sorted(d, key=d.__getitem__, reverse=True)
['b', 'd', 'c', 'a', 'e']
Also, Python 2.5's heapq.nlargest() function addresses the common use
case of finding only a few of the highest valued items:
>>> nlargest(2, d.iteritems(), itemgetter(1))
[('b', 23), ('d', 17)]
Python 3.1 has a collections.OrderedDict class that can be used for this purpose. It’s very efficient, too: “Big-O running times for all methods are the same as for regular dictionaries.”
The code for OrderedDict itself is compatible with Python 2.x, though some inherited methods (from the _abcoll module) do use Python 3-only features. However, they can be modified to 2.x code with minimal effort.
It depends on the Python version.
Python 3.7+
Dictionary iteration order is guaranteed to be in order of insertion.
Python 3.6
Dictionary iteration order happens to be in order of insertion in CPython implementation, but it is not a documented guarantee of the language.
Prior versions
Keys and values are iterated over in an arbitrary order which is non-random, varies across Python implementations, and depends on the dictionary’s history of insertions and deletions. If keys, values and items views are iterated over with no intervening modifications to the dictionary, the order of items will directly correspond. https://docs.python.org/2/library/stdtypes.html#dictionary-view-objects
The -R option
Python 2.6 added the -R option as (insufficient, it turned out) protection against hash flooding attacks. In Python 2 turning this on affected dictionary iteration order (the properties specified above were still maintained, but the specific iteration order would be different from one execution of the program to the next). For this reason, the option was off by default.
In Python 3, the -R option is on by default since Python 3.3, which adds nondeterminism to dict iteration order, as every time Python interpreter is run, the seed value for hash computation is generated randomly. This situation lasts until CPython 3.6 which changed dict implementation in a way so that the hash values of entries do not influence iteration order.
Source
-
Changed in version 3.7: Dictionary order is guaranteed to be insertion order. This behavior was an implementation detail of CPython from 3.6. https://docs.python.org/3.8/library/stdtypes.html
-
What’s New In Python 3.6: The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5). https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-compactdict
I’m currently implementing a complex microbial food-web in Python using SciPy.integrate.ode. I need the ability to easily add species and reactions to the system, so I have to code up something quite general. My scheme looks something like this:
class Reaction(object):
def __init__(self):
#stuff common to all reactions
def __getReactionRate(self, **kwargs):
raise NotImplementedError
... Reaction subclasses that
... implement specific types of reactions
class Species(object):
def __init__(self, reactionsDict):
self.reactionsDict = reactionsDict
#reactionsDict looks like {'ReactionName':reactionObject, ...}
#stuff common to all species
def sumOverAllReactionsForThisSpecies(self, **kwargs):
#loop over all the reactions and return the
#cumulative change in the concentrations of all solutes
...Species subclasses where for each species
... are defined and passed to the superclass constructor
class FermentationChamber(object):
def __init__(self, speciesList, timeToSolve, *args):
#do initialization
def step(self):
#loop over each species, which in turn loops
#over each reaction inside it and return a
#cumulative dictionary of total change for each
#solute in the whole system
if __name__==__main__:
f = FermentationChamber(...)
o = ode(...) #initialize ode solver
while o.successful() and o.t<timeToSolve:
o.integrate()
#process o.t and o.y (o.t contains the time points
#and o.y contains the solution matrix)
So, the question is, when I iterate over the dictionaries in Species.sumOverAllReactionsForThisSpecies() and FermentationChamber.step(), is the iteration order of the dictionaries guaranteed to be the same if no elements are added or removed from the dictionaries between the first and the last iteration? That is, can I assume that the order of the numpy array created at each iteration from the dictionary will not vary? For example, if a dictionary has the format {‘Glucose’:10, ‘Fructose’:12}, if an Array created from this dictionary will always have the same order (it doesn’t matter what that order is, as long as it’s deterministic).
Sorry for the mega-post, I just wanted to let you know where I’m coming from.
Yes, the same order is guaranteed if it is not modified.
See the docs here.
Edit:
Regarding if changing the value (but not adding/removing a key) will affect the order, this is what the comments in the C-source says:
/* CAUTION: PyDict_SetItem() must guarantee that it won't resize the
* dictionary if it's merely replacing the value for an existing key.
* This means that it's safe to loop over a dictionary with PyDict_Next()
* and occasionally replace a value -- but you can't insert new keys or
* remove them.
*/
It seems that its not an implementation detail, but a requirement of the language.
Provided no modifications are made to the dictionary, the answer is yes. See the docs here.
However, dictionaries are unordered by nature in Python. In general, it’s not the best practice to rely on dictionaries for sensitive sorted data.
An example of an a more robust solution would be Django’s SortedDict data structure.
If you want the order to be consistent, I would do something to force a particular order. Although you might be able to convince yourself that the order is guaranteed, and you might be right, it seems fragile to me, and it will be mysterious to other developers.
For example, you emphasize always in your question. Is it important that it be the same order in Python 2.5 and 2.6? 2.6 and 3.1? CPython and Jython? I wouldn’t count on those.
I also would recommend not relying on the fact the dictionaries order is non-random.
If you want a built in solution to sorting you dictionary read http://www.python.org/dev/peps/pep-0265/
Here is the most relevant material:
This PEP is rejected because the need for it has been largely
fulfilled by Py2.4’s sorted() builtin function:
>>> sorted(d.iteritems(), key=itemgetter(1), reverse=True)
[('b', 23), ('d', 17), ('c', 5), ('a', 2), ('e', 1)]
or for just the keys:
>>> sorted(d, key=d.__getitem__, reverse=True)
['b', 'd', 'c', 'a', 'e']
Also, Python 2.5's heapq.nlargest() function addresses the common use
case of finding only a few of the highest valued items:
>>> nlargest(2, d.iteritems(), itemgetter(1))
[('b', 23), ('d', 17)]
Python 3.1 has a collections.OrderedDict class that can be used for this purpose. It’s very efficient, too: “Big-O running times for all methods are the same as for regular dictionaries.”
The code for OrderedDict itself is compatible with Python 2.x, though some inherited methods (from the _abcoll module) do use Python 3-only features. However, they can be modified to 2.x code with minimal effort.
It depends on the Python version.
Python 3.7+
Dictionary iteration order is guaranteed to be in order of insertion.
Python 3.6
Dictionary iteration order happens to be in order of insertion in CPython implementation, but it is not a documented guarantee of the language.
Prior versions
Keys and values are iterated over in an arbitrary order which is non-random, varies across Python implementations, and depends on the dictionary’s history of insertions and deletions. If keys, values and items views are iterated over with no intervening modifications to the dictionary, the order of items will directly correspond. https://docs.python.org/2/library/stdtypes.html#dictionary-view-objects
The -R option
Python 2.6 added the -R option as (insufficient, it turned out) protection against hash flooding attacks. In Python 2 turning this on affected dictionary iteration order (the properties specified above were still maintained, but the specific iteration order would be different from one execution of the program to the next). For this reason, the option was off by default.
In Python 3, the -R option is on by default since Python 3.3, which adds nondeterminism to dict iteration order, as every time Python interpreter is run, the seed value for hash computation is generated randomly. This situation lasts until CPython 3.6 which changed dict implementation in a way so that the hash values of entries do not influence iteration order.
Source
Changed in version 3.7: Dictionary order is guaranteed to be insertion order. This behavior was an implementation detail of CPython from 3.6. https://docs.python.org/3.8/library/stdtypes.html
What’s New In Python 3.6: The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5). https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-compactdict