Why compare two strings via calculating xor of their characters?
Question:
Some time ago I found this function (unfortunately, I don’t remember from where it came from, most likely from some Python framework) that compares two strings and returns a bool value. It’s quite simple to understand what’s going on here.
Finding xor between char returns 1 (True) if they do not match.
def cmp_strings(str1, str2):
return len(str1) == len(str2) and sum(ord(x)^ord(y) for x, y in zip(str1, str2)) == 0
But why is this function used? Isn’t it the same as str1==str2?
Answers:
It takes a similar amount of time to compare any strings that have the same length. It’s used for security when the strings are sensitive. Usually it’s used to compare password hashes.
If == is used, Python stops comparing characters when the first one not matching is found. This is bad for hashes because it could reveal how close a hash was to matching. This would help an attacker to brute force a password.
This is how hmac.compare_digest works.
It appears to be doing a correlation (XOR sum) character-wise between the strings, given they are of the same length. It could be required in situations where you need to know ‘similarity’ and not equality. Maybe that was the plan. The author might have wanted to extend this function further.
The security issue that is being addressed by XOR comparison is known as a Timing Attack. …This is where you observe how much time it takes the Compare function to succeed|fail, and use that knowledge to gain an advantage over the system.
There are 95 printable ASCII characters. If you have an 8 character password, there are 95^8 (6,634,204,312,890,625) possible combinations …If the correct password is the last one in your list, and you can try 1 billion passwords per second, it will take you about 77 days to Brute Force the password …That’s too long – so we need a shortcut!
There are an infinite number of ways to store a string – and probably a dozen in popular use {length-prefixed, nul-terminated, …}{Unicode, UTF-8, ASCII, ,…}. For this working example, I will use the ubiquitous ‘NUL-terminated array of bytes using ASCII encoding’ …IE. "ABC" will be stored as "ABC"NUL, or {65, 66, 67, 0} …but whatever storage/encoding standard you use, the problem is essentially the same.
Syntactically, there are as many ways to compare two strings as there are languages, eg. if str1 == str2 or if (strcmp(str1, str2) == 0) etc. …but when you look at how they work internally, they are all pretty-much the same. Here is some simple (but realisitic) pseudo-code to perform a classic (non-security) string compare:
index = 0
LOOP FOREVER {
IF ( (str1[index] == 0) AND (str2[index] == 0) ) THEN return 'same'
IF (str1[index] != str2[index]) THEN return 'different'
index = index + 1
}
Assuming the secret password is "BY3"NUL …Let’s try some passwords, and notice how many operations the Compare function has to do to establish success|fail.
1. "A"NUL ... returns 'different' when 1st char is checked (A) [zero chars are correct]
2. "B"NUL ... returns 'different' when 2nd char is checked (NUL) [first char must be correct]
3. "BX"NUL ... returns 'different' when 2nd char is checked (X) [first char must be correct]
4. "BY"NUL ... returns 'different' when 3rd char is checked (NUL) [first two chars must be correct]
5. "BY1"NUL ... returns 'different' when 3rd char is checked (1) [first two chars must be correct]
6. "BY2"NUL ... returns 'different' when 3rd char is checked (2) [first two chars must be correct]
7. "BY3"NUL ... returns 'same' when the 4th character is checked (NUL) [all three chars are correct]
You can see that guess 1 fails the 1st time around the loop, guesses 2 & 3 fail the 2nd time around the loop …guesses 4, 5, 6 fail the 3rd time around the loop …and guess 7 succeeds the 4th time around the loop.
By observing how much time it takes the Compare function to fail, we can tell which character is wrong! This means we can actually guess the password one character at a time.
Again, let’s assume an 8 character password made up of the 95 printable characters, and our last guess will be correct …Because we can now guess the password one character at a time, it will take 95*8 (760) guesses. At 1 billion guesses per second, it will take about 0.7 milliseconds to find the password [it takes about 100mS to blink] …which is a significant advantage over 77 days …For a laugh work out the advantage for a 20 character password (95^20 vs 95 * 20).
So how do we stop an attacker from using a Timing Attack? [Spoiler: XOR]
The first thing we need to do is to make both strings the same length; and secondly, we must ALWAYS check EVERY character before returning ‘same’ or ‘different’ …This is surprisingly difficult to do without introducing a new Timing Attack. But rather than show you lots of ways to get it wrong, let’s see a way to do it right.
Passwords should (where possible) be stored as Hashes …{DES, MD5, SHA-1, …} have now been shown to have cryptographic flaws, {SHA-256, SHA-3, Whirlpool, …} are still in good favour [Oct 2021] …You may know that ALL Hashes (generated by a given algorithm) are the same length …So if we Hash the guess and compare the Guess-Hash against the Stored-Hash, we have solved the first problem – the ‘strings’ (array of bytes) we need to compare are now ALWAYS the same length.
Secondly. How to make sure our Compare function ALWAYS takes the same amount of time to reach its decision …There are probably a lot of ways to do this, but the most common solution is to use XOR like this:
result = 0
index = 0
LOOP WHILE (index < hashLength) {
result = result OR ( secretHash[index] XOR guessHash[index] )
index = index + 1
}
IF result == 0 THEN return 'same' ELSE return 'different'
And this way ALL calls to the compare function take the same length of time to run …No more Timing Attack!
Footnote:
For readers not familiar with Boolean Logic – go and read up; but the essence here is:
If A and B are the same, (A XOR B) gives a result of 0
If A and B are different, (A XOR B) gives a non-0 result
If A and B are both 0, (A OR B) gives a result of 0
If either A or B are non-0, (A OR B) gives a non-0 result
So (looking at the second code block) the first time the XOR returns non-0 (different), the result becomes non-0 (different) and can never return to 0 (same).
A search for "cve timing attack" will provide you with a list of real-life examples.
Some time ago I found this function (unfortunately, I don’t remember from where it came from, most likely from some Python framework) that compares two strings and returns a bool value. It’s quite simple to understand what’s going on here.
Finding xor between char returns 1 (True) if they do not match.
def cmp_strings(str1, str2):
return len(str1) == len(str2) and sum(ord(x)^ord(y) for x, y in zip(str1, str2)) == 0
But why is this function used? Isn’t it the same as str1==str2?
It takes a similar amount of time to compare any strings that have the same length. It’s used for security when the strings are sensitive. Usually it’s used to compare password hashes.
If == is used, Python stops comparing characters when the first one not matching is found. This is bad for hashes because it could reveal how close a hash was to matching. This would help an attacker to brute force a password.
This is how hmac.compare_digest works.
It appears to be doing a correlation (XOR sum) character-wise between the strings, given they are of the same length. It could be required in situations where you need to know ‘similarity’ and not equality. Maybe that was the plan. The author might have wanted to extend this function further.
The security issue that is being addressed by XOR comparison is known as a Timing Attack. …This is where you observe how much time it takes the Compare function to succeed|fail, and use that knowledge to gain an advantage over the system.
There are 95 printable ASCII characters. If you have an 8 character password, there are 95^8 (6,634,204,312,890,625) possible combinations …If the correct password is the last one in your list, and you can try 1 billion passwords per second, it will take you about 77 days to Brute Force the password …That’s too long – so we need a shortcut!
There are an infinite number of ways to store a string – and probably a dozen in popular use {length-prefixed, nul-terminated, …}{Unicode, UTF-8, ASCII, ,…}. For this working example, I will use the ubiquitous ‘NUL-terminated array of bytes using ASCII encoding’ …IE. "ABC" will be stored as "ABC"NUL, or {65, 66, 67, 0} …but whatever storage/encoding standard you use, the problem is essentially the same.
Syntactically, there are as many ways to compare two strings as there are languages, eg. if str1 == str2 or if (strcmp(str1, str2) == 0) etc. …but when you look at how they work internally, they are all pretty-much the same. Here is some simple (but realisitic) pseudo-code to perform a classic (non-security) string compare:
index = 0
LOOP FOREVER {
IF ( (str1[index] == 0) AND (str2[index] == 0) ) THEN return 'same'
IF (str1[index] != str2[index]) THEN return 'different'
index = index + 1
}
Assuming the secret password is "BY3"NUL …Let’s try some passwords, and notice how many operations the Compare function has to do to establish success|fail.
1. "A"NUL ... returns 'different' when 1st char is checked (A) [zero chars are correct]
2. "B"NUL ... returns 'different' when 2nd char is checked (NUL) [first char must be correct]
3. "BX"NUL ... returns 'different' when 2nd char is checked (X) [first char must be correct]
4. "BY"NUL ... returns 'different' when 3rd char is checked (NUL) [first two chars must be correct]
5. "BY1"NUL ... returns 'different' when 3rd char is checked (1) [first two chars must be correct]
6. "BY2"NUL ... returns 'different' when 3rd char is checked (2) [first two chars must be correct]
7. "BY3"NUL ... returns 'same' when the 4th character is checked (NUL) [all three chars are correct]
You can see that guess 1 fails the 1st time around the loop, guesses 2 & 3 fail the 2nd time around the loop …guesses 4, 5, 6 fail the 3rd time around the loop …and guess 7 succeeds the 4th time around the loop.
By observing how much time it takes the Compare function to fail, we can tell which character is wrong! This means we can actually guess the password one character at a time.
Again, let’s assume an 8 character password made up of the 95 printable characters, and our last guess will be correct …Because we can now guess the password one character at a time, it will take 95*8 (760) guesses. At 1 billion guesses per second, it will take about 0.7 milliseconds to find the password [it takes about 100mS to blink] …which is a significant advantage over 77 days …For a laugh work out the advantage for a 20 character password (95^20 vs 95 * 20).
So how do we stop an attacker from using a Timing Attack? [Spoiler: XOR]
The first thing we need to do is to make both strings the same length; and secondly, we must ALWAYS check EVERY character before returning ‘same’ or ‘different’ …This is surprisingly difficult to do without introducing a new Timing Attack. But rather than show you lots of ways to get it wrong, let’s see a way to do it right.
Passwords should (where possible) be stored as Hashes …{DES, MD5, SHA-1, …} have now been shown to have cryptographic flaws, {SHA-256, SHA-3, Whirlpool, …} are still in good favour [Oct 2021] …You may know that ALL Hashes (generated by a given algorithm) are the same length …So if we Hash the guess and compare the Guess-Hash against the Stored-Hash, we have solved the first problem – the ‘strings’ (array of bytes) we need to compare are now ALWAYS the same length.
Secondly. How to make sure our Compare function ALWAYS takes the same amount of time to reach its decision …There are probably a lot of ways to do this, but the most common solution is to use XOR like this:
result = 0
index = 0
LOOP WHILE (index < hashLength) {
result = result OR ( secretHash[index] XOR guessHash[index] )
index = index + 1
}
IF result == 0 THEN return 'same' ELSE return 'different'
And this way ALL calls to the compare function take the same length of time to run …No more Timing Attack!
Footnote:
For readers not familiar with Boolean Logic – go and read up; but the essence here is:
If A and B are the same, (A XOR B) gives a result of 0
If A and B are different, (A XOR B) gives a non-0 result
If A and B are both 0, (A OR B) gives a result of 0
If either A or B are non-0, (A OR B) gives a non-0 result
So (looking at the second code block) the first time the XOR returns non-0 (different), the result becomes non-0 (different) and can never return to 0 (same).
A search for "cve timing attack" will provide you with a list of real-life examples.