Fastest way to search a list in python
Question:
When you do something like "test" in a where a is a list does python do a sequential search on the list or does it create a hash table representation to optimize the lookup? In the application I need this for I’ll be doing a lot of lookups on the list so would it be best to do something like b = set(a) and then "test" in b? Also note that the list of values I’ll have won’t have duplicate data and I don’t actually care about the order it’s in; I just need to be able to check for the existence of a value.
Answers:
Also note that the list of values I’ll have won’t have duplicate data and I don’t actually care about the order it’s in; I just need to be able to check for the existence of a value.
Don’t use a list, use a set() instead. It has exactly the properties you want, including a blazing fast in test.
I’ve seen speedups of 20x and higher in places (mostly heavy number crunching) where one list was changed for a set.
"test" in a with a list a will do a linear search. Setting up a hash table on the fly would be much more expensive than a linear search. "test" in b on the other hand will do an amoirtised O(1) hash look-up.
In the case you describe, there doesn’t seem to be a reason to use a list over a set.
I think it would be better to go with the set implementation. I know for a fact that sets have O(1) lookup time. I think lists take O(n) lookup time. But even if lists are also O(1) lookup, you lose nothing with switching to sets.
Further, sets don’t allow duplicate values. This will make your program slightly more memory efficient as well
List and tuples seems to have the same time, and using “in” is slow for large data:
>>> t = list(range(0, 1000000))
>>> a=time.time();x = [b in t for b in range(100234,101234)];print(time.time()-a)
1.66235494614
>>> t = tuple(range(0, 1000000))
>>> a=time.time();x = [b in t for b in range(100234,101234)];print(time.time()-a)
1.6594209671
Here is much better solution: Most efficient way for a lookup/search in a huge list (python)
It’s super fast:
>>> from bisect import bisect_left
>>> t = list(range(0, 1000000))
>>> a=time.time();x = [t[bisect_left(t,b)]==b for b in range(100234,101234)];print(time.time()-a)
0.0054759979248
When you do something like "test" in a where a is a list does python do a sequential search on the list or does it create a hash table representation to optimize the lookup? In the application I need this for I’ll be doing a lot of lookups on the list so would it be best to do something like b = set(a) and then "test" in b? Also note that the list of values I’ll have won’t have duplicate data and I don’t actually care about the order it’s in; I just need to be able to check for the existence of a value.
Also note that the list of values I’ll have won’t have duplicate data and I don’t actually care about the order it’s in; I just need to be able to check for the existence of a value.
Don’t use a list, use a set() instead. It has exactly the properties you want, including a blazing fast in test.
I’ve seen speedups of 20x and higher in places (mostly heavy number crunching) where one list was changed for a set.
"test" in a with a list a will do a linear search. Setting up a hash table on the fly would be much more expensive than a linear search. "test" in b on the other hand will do an amoirtised O(1) hash look-up.
In the case you describe, there doesn’t seem to be a reason to use a list over a set.
I think it would be better to go with the set implementation. I know for a fact that sets have O(1) lookup time. I think lists take O(n) lookup time. But even if lists are also O(1) lookup, you lose nothing with switching to sets.
Further, sets don’t allow duplicate values. This will make your program slightly more memory efficient as well
List and tuples seems to have the same time, and using “in” is slow for large data:
>>> t = list(range(0, 1000000))
>>> a=time.time();x = [b in t for b in range(100234,101234)];print(time.time()-a)
1.66235494614
>>> t = tuple(range(0, 1000000))
>>> a=time.time();x = [b in t for b in range(100234,101234)];print(time.time()-a)
1.6594209671
Here is much better solution: Most efficient way for a lookup/search in a huge list (python)
It’s super fast:
>>> from bisect import bisect_left
>>> t = list(range(0, 1000000))
>>> a=time.time();x = [t[bisect_left(t,b)]==b for b in range(100234,101234)];print(time.time()-a)
0.0054759979248