hash function in Python 3.3 returns different results between sessions
Question:
I’ve implemented a BloomFilter in python 3.3, and got different results every session. Drilling down this weird behavior got me to the internal hash() function – it returns different hash values for the same string every session.
Example:
>>> hash("235")
-310569535015251310
—– opening a new python console —–
>>> hash("235")
-1900164331622581997
Why is this happening?
Why is this useful?
Answers:
Python uses a random hash seed to prevent attackers from tar-pitting your application by sending you keys designed to collide. See the original vulnerability disclosure. By offsetting the hash with a random seed (set once at startup) attackers can no longer predict what keys will collide.
You can set a fixed seed or disable the feature by setting the PYTHONHASHSEED environment variable; the default is random but you can set it to a fixed positive integer value, with 0 disabling the feature altogether.
Python versions 2.7 and 3.2 have the feature disabled by default (use the -R switch or set PYTHONHASHSEED=random to enable it); it is enabled by default in Python 3.3 and up.
If you were relying on the order of keys in a Python set, then don’t. Python uses a hash table to implement these types and their order depends on the insertion and deletion history as well as the random hash seed. Note that in Python 3.5 and older, this applies to dictionaries, too.
Also see the object.__hash__() special method documentation:
Note: By default, the __hash__() values of str, bytes and datetime objects are “salted” with an unpredictable random value. Although they remain constant within an individual Python process, they are not predictable between repeated invocations of Python.
This is intended to provide protection against a denial-of-service caused by carefully-chosen inputs that exploit the worst case performance of a dict insertion, O(n^2) complexity. See http://www.ocert.org/advisories/ocert-2011-003.html for details.
Changing hash values affects the iteration order of dicts, sets and other mappings. Python has never made guarantees about this ordering (and it typically varies between 32-bit and 64-bit builds).
See also PYTHONHASHSEED.
If you need a stable hash implementation, you probably want to look at the hashlib module; this implements cryptographic hash functions. The pybloom project uses this approach.
Since the offset consists of a prefix and a suffix (start value and final XORed value, respectively) you cannot just store the offset, unfortunately. On the plus side, this does mean that attackers cannot easily determine the offset with timing attacks either.
Hash randomisation is turned on by default in Python 3. This is a security feature:
Hash randomization is intended to provide protection against a denial-of-service caused by carefully-chosen inputs that exploit the worst case performance of a dict construction
In previous versions from 2.6.8, you could switch it on at the command line with -R, or the PYTHONHASHSEED environment option.
You can switch it off by setting PYTHONHASHSEED to zero.
This behavior of hash() tripped me up when trying to compare records saved in a database between sessions.
The PYTHONHASHSEED solution was too complicated because I needed my program to work reliably, independent of environment variable settings.
So I created my simple has function that hashes strings (it’s easy to convert anything to strings) and produces a 32 bit positive integer as the hash. It’s not a cryptographically safe hash but it’s good enough for quick comparisons.
def myHash(text:str):
hash=0
for ch in text:
hash = ( hash*281 ^ ord(ch)*997) & 0xFFFFFFFF
return hash
The numbers in the multiplications are just arbitrarily chosen prime numbers in order to mix up the bits.
If you want the hash to be a hex string, you can replace the last line with:
return hex(hash)[2:].upper().zfill(8)
I’ve implemented a BloomFilter in python 3.3, and got different results every session. Drilling down this weird behavior got me to the internal hash() function – it returns different hash values for the same string every session.
Example:
>>> hash("235")
-310569535015251310
—– opening a new python console —–
>>> hash("235")
-1900164331622581997
Why is this happening?
Why is this useful?
Python uses a random hash seed to prevent attackers from tar-pitting your application by sending you keys designed to collide. See the original vulnerability disclosure. By offsetting the hash with a random seed (set once at startup) attackers can no longer predict what keys will collide.
You can set a fixed seed or disable the feature by setting the PYTHONHASHSEED environment variable; the default is random but you can set it to a fixed positive integer value, with 0 disabling the feature altogether.
Python versions 2.7 and 3.2 have the feature disabled by default (use the -R switch or set PYTHONHASHSEED=random to enable it); it is enabled by default in Python 3.3 and up.
If you were relying on the order of keys in a Python set, then don’t. Python uses a hash table to implement these types and their order depends on the insertion and deletion history as well as the random hash seed. Note that in Python 3.5 and older, this applies to dictionaries, too.
Also see the object.__hash__() special method documentation:
Note: By default, the
__hash__()values of str, bytes and datetime objects are “salted” with an unpredictable random value. Although they remain constant within an individual Python process, they are not predictable between repeated invocations of Python.This is intended to provide protection against a denial-of-service caused by carefully-chosen inputs that exploit the worst case performance of a dict insertion, O(n^2) complexity. See http://www.ocert.org/advisories/ocert-2011-003.html for details.
Changing hash values affects the iteration order of dicts, sets and other mappings. Python has never made guarantees about this ordering (and it typically varies between 32-bit and 64-bit builds).
See also
PYTHONHASHSEED.
If you need a stable hash implementation, you probably want to look at the hashlib module; this implements cryptographic hash functions. The pybloom project uses this approach.
Since the offset consists of a prefix and a suffix (start value and final XORed value, respectively) you cannot just store the offset, unfortunately. On the plus side, this does mean that attackers cannot easily determine the offset with timing attacks either.
Hash randomisation is turned on by default in Python 3. This is a security feature:
Hash randomization is intended to provide protection against a denial-of-service caused by carefully-chosen inputs that exploit the worst case performance of a dict construction
In previous versions from 2.6.8, you could switch it on at the command line with -R, or the PYTHONHASHSEED environment option.
You can switch it off by setting PYTHONHASHSEED to zero.
This behavior of hash() tripped me up when trying to compare records saved in a database between sessions.
The PYTHONHASHSEED solution was too complicated because I needed my program to work reliably, independent of environment variable settings.
So I created my simple has function that hashes strings (it’s easy to convert anything to strings) and produces a 32 bit positive integer as the hash. It’s not a cryptographically safe hash but it’s good enough for quick comparisons.
def myHash(text:str):
hash=0
for ch in text:
hash = ( hash*281 ^ ord(ch)*997) & 0xFFFFFFFF
return hash
The numbers in the multiplications are just arbitrarily chosen prime numbers in order to mix up the bits.
If you want the hash to be a hex string, you can replace the last line with:
return hex(hash)[2:].upper().zfill(8)