Python AND operator on two boolean lists – how?
Question:
I have two boolean lists, e.g.,
x=[True,True,False,False]
y=[True,False,True,False]
I want to AND these lists together, with the expected output:
xy=[True,False,False,False]
I thought that expression x and y would work, but came to discover that it does not: in fact, (x and y) != (y and x)
Output of x and y: [True,False,True,False]
Output of y and x: [True,True,False,False]
Using list comprehension does have correct output. Whew!
xy = [x[i] and y[i] for i in range(len(x)]
Mind you I could not find any reference that told me the AND operator would work as I tried with x and y. But it’s easy to try things in Python.
Can someone explain to me what is happening with x and y?
And here is a simple test program:
import random
random.seed()
n = 10
x = [random.random() > 0.5 for i in range(n)]
y = [random.random() > 0.5 for i in range(n)]
# Next two methods look sensible, but do not work
a = x and y
z = y and x
# Next: apparently only the list comprehension method is correct
xy = [x[i] and y[i] for i in range(n)]
print 'x : %s'%str(x)
print 'y : %s'%str(y)
print 'x and y : %s'%str(a)
print 'y and x : %s'%str(z)
print '[x and y]: %s'%str(xy)
Answers:
This should do what you want:
xy = [a and b for a, b in zip(x, y)]
The reason x and y returns y and y and x returns x is because boolean operators in python return the last value checked that determines the true-ness of the expression. Non-empty list‘s evaluate to True, and since and requires both operands to evaluate True, the last operand checked is the second operand. Contrast with x or y, which would return x because it doesn’t need to check y to determine the true-ness of the expression.
You can use the zip function
x=[True,True,False,False]
y=[True,False,True,False]
z=[a and b for a,b in zip(x,y)]
and simply returns either the first or the second operand, based on their truth value. If the first operand is considered false, it is returned, otherwise the other operand is returned.
Lists are considered true when not empty, so both lists are considered true. Their contents don’t play a role here.
Because both lists are not empty, x and y simply returns the second list object; only if x was empty would it be returned instead:
>>> [True, False] and ['foo', 'bar']
['foo', 'bar']
>>> [] and ['foo', 'bar']
[]
See the Truth value testing section in the Python documentation:
Any object can be tested for truth value, for use in an if or while condition or as operand of the Boolean operations below. The following values are considered false:
[…]
- any empty sequence, for example,
'', (), [].
[…]
All other values are considered true — so objects of many types are always true.
(emphasis mine), and the Boolean operations section right below that:
x and y
if x is false, then x, else y
This is a short-circuit operator, so it only evaluates the second argument if the first one is True.
You indeed need to test the values contained in the lists explicitly. You can do so with a list comprehension, as you discovered. You can rewrite it with the zip() function to pair up the values:
[a and b for a, b in zip(x, y)]
and is not necessarily a Boolean operator; it returns one of its two arguments, regardless of their type. If the first argument is false-ish (False, numeric zero, or an empty string/container), it returns that argument. Otherwise, it returns the second argument.
In your case, both x and y are non-empty lists, so the first argument is always true-ish, meaning x and y returns y and y and x returns x.
You could use numpy:
>>> import numpy as np
>>> x=np.array([True,True,False,False])
>>> y=np.array([True,False,True,False])
>>> x & y
array([ True, False, False, False], dtype=bool)
Numpy allows numerical and logical operations on arrays such as:
>>> z=np.array([1,2,3,4])
>>> z+1
array([2, 3, 4, 5])
You can perform bitwise and with the & operator.
Instead of a list comprehension, you can use numpy to generate the boolean array directly like so:
>>> np.random.random(10)>.5
array([ True, True, True, False, False, True, True, False, False, False], dtype=bool)
Instead of using
[a and b for a, b in zip(x, y)]
one could just use the possibility of numpy to multiply bool-values:
(np.array(x)*np.array(y))
>> array([ True, False, False, False], dtype=bool)
Or do I overlook a special case?
In addition to what @Martijn Pieters has answered, I would just add the following code to explain and and or operations in action.
and returns the first falsy value encountered else the last evaluated argument.
Similarly or returns the first truthy value encountered else the last evaluated argument.
nl1 = [3,3,3,3,0,0,0,0]
nl2 = [2,2,0,0,2,2,0,0]
nl3 = [1,0,1,0,1,0,1,0]
and_list = [a and b and c for a,b,c in zip(nl1,nl2,nl3)]
or_list = [a or b or c for a,b,c in zip(nl1,nl2,nl3)]
Values are
and_list = [1, 0, 0, 0, 0, 0, 0, 0]
or_list = [3, 3, 3, 3, 2, 2, 1, 0]
Thanks for the answer @Martijn Pieters and @Tony.
I dig into the timing of the various options we have to make the AND of two lists and I would like to share my results, because I found them interesting.
Despite liking a lot the pythonic way [a and b for a,b in zip(x,y) ], turns out really slow.
I compare with a integer product of arrays (1*(array of bool)) * (1*(array of bool)) and it turns out to be more than 10x faster
import time
import numpy as np
array_to_filter = np.linspace(1,1000000,1000000) # 1 million of integers :-)
value_limit = 100
cycles = 100
# METHOD #1: [a and b for a,b in zip(x,y) ]
t0=time.clock()
for jj in range(cycles):
x = array_to_filter<np.max(array_to_filter)-value_limit # filter the values > MAX-value_limit
y = array_to_filter>value_limit # filter the values < value_limit
z= [a and b for a,b in zip(x,y) ] # AND
filtered = array_to_filter[z]
print('METHOD #1 = %.2f s' % ( (time.clock()-t0)))
# METHOD 1*(array of bool) AND 1*(array of bool)
t0=time.clock()
for jj in range(cycles):
x = 1*(array_to_filter<np.max(array_to_filter)-value_limit) # filter the values > MAX-value_limit
y = 1*(array_to_filter>value_limit) # filter the values < value_limit
z = x*y # AND
z = z.astype(bool) # convert back to array of bool
filtered = array_to_filter[z]
print('METHOD #2 = %.2f s' % ( (time.clock()-t0)))
The results are
METHOD #1 = 15.36 s
METHOD #2 = 1.85 s
The speed is almost affected equally by the size of the array or by the number of cycles.
I hope I helped someone code to be faster. 🙂
Here is a simple solution:
np.logical_and(x,y)
To generalize on the zip approach, use all and any for any number of lists.
all for AND:
[all(i) for i in zip(a, b, c)] # zip all lists
and any for OR:
[any(i) for i in zip(a, b, c)]
I have two boolean lists, e.g.,
x=[True,True,False,False]
y=[True,False,True,False]
I want to AND these lists together, with the expected output:
xy=[True,False,False,False]
I thought that expression x and y would work, but came to discover that it does not: in fact, (x and y) != (y and x)
Output of x and y: [True,False,True,False]
Output of y and x: [True,True,False,False]
Using list comprehension does have correct output. Whew!
xy = [x[i] and y[i] for i in range(len(x)]
Mind you I could not find any reference that told me the AND operator would work as I tried with x and y. But it’s easy to try things in Python.
Can someone explain to me what is happening with x and y?
And here is a simple test program:
import random
random.seed()
n = 10
x = [random.random() > 0.5 for i in range(n)]
y = [random.random() > 0.5 for i in range(n)]
# Next two methods look sensible, but do not work
a = x and y
z = y and x
# Next: apparently only the list comprehension method is correct
xy = [x[i] and y[i] for i in range(n)]
print 'x : %s'%str(x)
print 'y : %s'%str(y)
print 'x and y : %s'%str(a)
print 'y and x : %s'%str(z)
print '[x and y]: %s'%str(xy)
This should do what you want:
xy = [a and b for a, b in zip(x, y)]
The reason x and y returns y and y and x returns x is because boolean operators in python return the last value checked that determines the true-ness of the expression. Non-empty list‘s evaluate to True, and since and requires both operands to evaluate True, the last operand checked is the second operand. Contrast with x or y, which would return x because it doesn’t need to check y to determine the true-ness of the expression.
You can use the zip function
x=[True,True,False,False]
y=[True,False,True,False]
z=[a and b for a,b in zip(x,y)]
and simply returns either the first or the second operand, based on their truth value. If the first operand is considered false, it is returned, otherwise the other operand is returned.
Lists are considered true when not empty, so both lists are considered true. Their contents don’t play a role here.
Because both lists are not empty, x and y simply returns the second list object; only if x was empty would it be returned instead:
>>> [True, False] and ['foo', 'bar']
['foo', 'bar']
>>> [] and ['foo', 'bar']
[]
See the Truth value testing section in the Python documentation:
Any object can be tested for truth value, for use in an
iforwhilecondition or as operand of the Boolean operations below. The following values are considered false:[…]
- any empty sequence, for example,
'',(),[].[…]
All other values are considered true — so objects of many types are always true.
(emphasis mine), and the Boolean operations section right below that:
x and y
if x is false, then x, else yThis is a short-circuit operator, so it only evaluates the second argument if the first one is
True.
You indeed need to test the values contained in the lists explicitly. You can do so with a list comprehension, as you discovered. You can rewrite it with the zip() function to pair up the values:
[a and b for a, b in zip(x, y)]
and is not necessarily a Boolean operator; it returns one of its two arguments, regardless of their type. If the first argument is false-ish (False, numeric zero, or an empty string/container), it returns that argument. Otherwise, it returns the second argument.
In your case, both x and y are non-empty lists, so the first argument is always true-ish, meaning x and y returns y and y and x returns x.
You could use numpy:
>>> import numpy as np
>>> x=np.array([True,True,False,False])
>>> y=np.array([True,False,True,False])
>>> x & y
array([ True, False, False, False], dtype=bool)
Numpy allows numerical and logical operations on arrays such as:
>>> z=np.array([1,2,3,4])
>>> z+1
array([2, 3, 4, 5])
You can perform bitwise and with the & operator.
Instead of a list comprehension, you can use numpy to generate the boolean array directly like so:
>>> np.random.random(10)>.5
array([ True, True, True, False, False, True, True, False, False, False], dtype=bool)
Instead of using
[a and b for a, b in zip(x, y)]
one could just use the possibility of numpy to multiply bool-values:
(np.array(x)*np.array(y))
>> array([ True, False, False, False], dtype=bool)
Or do I overlook a special case?
In addition to what @Martijn Pieters has answered, I would just add the following code to explain and and or operations in action.
and returns the first falsy value encountered else the last evaluated argument.
Similarly or returns the first truthy value encountered else the last evaluated argument.
nl1 = [3,3,3,3,0,0,0,0]
nl2 = [2,2,0,0,2,2,0,0]
nl3 = [1,0,1,0,1,0,1,0]
and_list = [a and b and c for a,b,c in zip(nl1,nl2,nl3)]
or_list = [a or b or c for a,b,c in zip(nl1,nl2,nl3)]
Values are
and_list = [1, 0, 0, 0, 0, 0, 0, 0]
or_list = [3, 3, 3, 3, 2, 2, 1, 0]
Thanks for the answer @Martijn Pieters and @Tony.
I dig into the timing of the various options we have to make the AND of two lists and I would like to share my results, because I found them interesting.
Despite liking a lot the pythonic way [a and b for a,b in zip(x,y) ], turns out really slow.
I compare with a integer product of arrays (1*(array of bool)) * (1*(array of bool)) and it turns out to be more than 10x faster
import time
import numpy as np
array_to_filter = np.linspace(1,1000000,1000000) # 1 million of integers :-)
value_limit = 100
cycles = 100
# METHOD #1: [a and b for a,b in zip(x,y) ]
t0=time.clock()
for jj in range(cycles):
x = array_to_filter<np.max(array_to_filter)-value_limit # filter the values > MAX-value_limit
y = array_to_filter>value_limit # filter the values < value_limit
z= [a and b for a,b in zip(x,y) ] # AND
filtered = array_to_filter[z]
print('METHOD #1 = %.2f s' % ( (time.clock()-t0)))
# METHOD 1*(array of bool) AND 1*(array of bool)
t0=time.clock()
for jj in range(cycles):
x = 1*(array_to_filter<np.max(array_to_filter)-value_limit) # filter the values > MAX-value_limit
y = 1*(array_to_filter>value_limit) # filter the values < value_limit
z = x*y # AND
z = z.astype(bool) # convert back to array of bool
filtered = array_to_filter[z]
print('METHOD #2 = %.2f s' % ( (time.clock()-t0)))
The results are
METHOD #1 = 15.36 s
METHOD #2 = 1.85 s
The speed is almost affected equally by the size of the array or by the number of cycles.
I hope I helped someone code to be faster. 🙂
Here is a simple solution:
np.logical_and(x,y)
To generalize on the zip approach, use all and any for any number of lists.
all for AND:
[all(i) for i in zip(a, b, c)] # zip all lists
and any for OR:
[any(i) for i in zip(a, b, c)]