Is the shortcircuit behaviour of Python's any/all explicit?
Question:
Prompted by the discussion here
The docs suggest some equivalent code for the behaviour of all
and any
Should the behaviour of the equivalent code be considered part of the definition, or can an implementation implement them in a non-shortcircuit manner?
Here is the relevant excerpt from cpython/Lib/test/test_builtin.py
def test_all(self):
self.assertEqual(all([2, 4, 6]), True)
self.assertEqual(all([2, None, 6]), False)
self.assertRaises(RuntimeError, all, [2, TestFailingBool(), 6])
self.assertRaises(RuntimeError, all, TestFailingIter())
self.assertRaises(TypeError, all, 10) # Non-iterable
self.assertRaises(TypeError, all) # No args
self.assertRaises(TypeError, all, [2, 4, 6], []) # Too many args
self.assertEqual(all([]), True) # Empty iterator
S = [50, 60]
self.assertEqual(all(x > 42 for x in S), True)
S = [50, 40, 60]
self.assertEqual(all(x > 42 for x in S), False)
def test_any(self):
self.assertEqual(any([None, None, None]), False)
self.assertEqual(any([None, 4, None]), True)
self.assertRaises(RuntimeError, any, [None, TestFailingBool(), 6])
self.assertRaises(RuntimeError, all, TestFailingIter())
self.assertRaises(TypeError, any, 10) # Non-iterable
self.assertRaises(TypeError, any) # No args
self.assertRaises(TypeError, any, [2, 4, 6], []) # Too many args
self.assertEqual(any([]), False) # Empty iterator
S = [40, 60, 30]
self.assertEqual(any(x > 42 for x in S), True)
S = [10, 20, 30]
self.assertEqual(any(x > 42 for x in S), False)
It doesn’t do anything to enforce the shortcircuit behaviour
Answers:
The docs say
“Return True if any element of the iterable is true. If the iterable is empty, return False. EQUIVALENT TO:” (emphasis mine) …
def any(iterable):
for element in iterable:
if element:
return True
return False
If any
didn’t short circuit, it wouldn’t be EQUIVALENT to the posted code since the posted code clearly short circuits. You could consume more of a generator than you want to for example. In light of that, I say that the short circuiting behavior is guaranteed.
The exact same argument could be made for all
.
It HAS to short circuit, since it could be given an unbound iterable. If it did not short circuit then this would never terminate:
any(x == 10 for x in itertools.count())
The behaviour is guaranteed. I’ve contributed a patch, which was accepted and merged recently, so if you grab the latest sources you will see that the short-circuiting behaviour is now explicitly enforced.
git clone https://github.com/python/cpython.git
grep Short-circuit cpython/Lib/test/test_builtin.py
In case you landed here wondering why your any
/all
calls do not seem to short circuit:
They do short-circuit, but using a list comprehension can sometimes be a gotcha, making it seem like you are overriding the short-circuiting behavior:
def hi():
print('hi')
return True
>>> any(hi() for num in [1, 2, 3, 4])
hi
>>> any([hi() for num in [1, 2, 3, 4]])
hi
hi
hi
hi
The list comprehension in the second call here executes before any()
does: the entire list gets built before any()
starts to look at the built list’s four elements. So it appears that any()
isn’t short-circuiting, but in reality, after a few steps, it is.
Note: This does not answer the OP’s very different question
Prompted by the discussion here
The docs suggest some equivalent code for the behaviour of all
and any
Should the behaviour of the equivalent code be considered part of the definition, or can an implementation implement them in a non-shortcircuit manner?
Here is the relevant excerpt from cpython/Lib/test/test_builtin.py
def test_all(self):
self.assertEqual(all([2, 4, 6]), True)
self.assertEqual(all([2, None, 6]), False)
self.assertRaises(RuntimeError, all, [2, TestFailingBool(), 6])
self.assertRaises(RuntimeError, all, TestFailingIter())
self.assertRaises(TypeError, all, 10) # Non-iterable
self.assertRaises(TypeError, all) # No args
self.assertRaises(TypeError, all, [2, 4, 6], []) # Too many args
self.assertEqual(all([]), True) # Empty iterator
S = [50, 60]
self.assertEqual(all(x > 42 for x in S), True)
S = [50, 40, 60]
self.assertEqual(all(x > 42 for x in S), False)
def test_any(self):
self.assertEqual(any([None, None, None]), False)
self.assertEqual(any([None, 4, None]), True)
self.assertRaises(RuntimeError, any, [None, TestFailingBool(), 6])
self.assertRaises(RuntimeError, all, TestFailingIter())
self.assertRaises(TypeError, any, 10) # Non-iterable
self.assertRaises(TypeError, any) # No args
self.assertRaises(TypeError, any, [2, 4, 6], []) # Too many args
self.assertEqual(any([]), False) # Empty iterator
S = [40, 60, 30]
self.assertEqual(any(x > 42 for x in S), True)
S = [10, 20, 30]
self.assertEqual(any(x > 42 for x in S), False)
It doesn’t do anything to enforce the shortcircuit behaviour
The docs say
“Return True if any element of the iterable is true. If the iterable is empty, return False. EQUIVALENT TO:” (emphasis mine) …
def any(iterable):
for element in iterable:
if element:
return True
return False
If any
didn’t short circuit, it wouldn’t be EQUIVALENT to the posted code since the posted code clearly short circuits. You could consume more of a generator than you want to for example. In light of that, I say that the short circuiting behavior is guaranteed.
The exact same argument could be made for all
.
It HAS to short circuit, since it could be given an unbound iterable. If it did not short circuit then this would never terminate:
any(x == 10 for x in itertools.count())
The behaviour is guaranteed. I’ve contributed a patch, which was accepted and merged recently, so if you grab the latest sources you will see that the short-circuiting behaviour is now explicitly enforced.
git clone https://github.com/python/cpython.git
grep Short-circuit cpython/Lib/test/test_builtin.py
In case you landed here wondering why your any
/all
calls do not seem to short circuit:
They do short-circuit, but using a list comprehension can sometimes be a gotcha, making it seem like you are overriding the short-circuiting behavior:
def hi():
print('hi')
return True
>>> any(hi() for num in [1, 2, 3, 4])
hi
>>> any([hi() for num in [1, 2, 3, 4]])
hi
hi
hi
hi
The list comprehension in the second call here executes before any()
does: the entire list gets built before any()
starts to look at the built list’s four elements. So it appears that any()
isn’t short-circuiting, but in reality, after a few steps, it is.
Note: This does not answer the OP’s very different question