Case-insensitive string startswith in Python
Question:
Here is how I check whether mystring begins with some string:
>>> mystring.lower().startswith("he")
True
The problem is that mystring is very long (thousands of characters), so the lower() operation takes a lot of time.
QUESTION: Is there a more efficient way?
My unsuccessful attempt:
>>> import re;
>>> mystring.startswith("he", re.I)
False
Answers:
You could use a regular expression as follows:
In [33]: bool(re.match('he', 'Hello', re.I))
Out[33]: True
In [34]: bool(re.match('el', 'Hello', re.I))
Out[34]: False
On a 2000-character string this is about 20x times faster than lower():
In [38]: s = 'A' * 2000
In [39]: %timeit s.lower().startswith('he')
10000 loops, best of 3: 41.3 us per loop
In [40]: %timeit bool(re.match('el', s, re.I))
100000 loops, best of 3: 2.06 us per loop
If you are matching the same prefix repeatedly, pre-compiling the regex can make a large difference:
In [41]: p = re.compile('he', re.I)
In [42]: %timeit p.match(s)
1000000 loops, best of 3: 351 ns per loop
For short prefixes, slicing the prefix out of the string before converting it to lowercase could be even faster:
In [43]: %timeit s[:2].lower() == 'he'
1000000 loops, best of 3: 287 ns per loop
Relative timings of these approaches will of course depend on the length of the prefix. On my machine the breakeven point seems to be about six characters, which is when the pre-compiled regex becomes the fastest method.
In my experiments, checking every character separately could be even faster:
In [44]: %timeit (s[0] == 'h' or s[0] == 'H') and (s[1] == 'e' or s[1] == 'E')
1000000 loops, best of 3: 189 ns per loop
However, this method only works for prefixes that are known when you’re writing the code, and doesn’t lend itself to longer prefixes.
How about this:
prefix = 'he'
if myVeryLongStr[:len(prefix)].lower() == prefix.lower()
Depending on the performance of .lower(), if prefix was small enough it might be faster to check equality multiple times:
s = 'A' * 2000
prefix = 'he'
ch0 = s[0]
ch1 = s[1]
substr = ch0 == 'h' or ch0 == 'H' and ch1 == 'e' or ch1 == 'E'
Timing (using the same string as NPE):
>>> timeit.timeit("ch0 = s[0]; ch1 = s[1]; ch0 == 'h' or ch0 == 'H' and ch1 == 'e' or ch1 == 'E'", "s = 'A' * 2000")
0.2509511683747405
= 0.25 us per loop
Compared to existing method:
>>> timeit.timeit("s.lower().startswith('he')", "s = 'A' * 2000", number=10000)
0.6162763703208611
= 61.63 us per loop
(This is horrible, of course, but if the code is extremely performance critical then it might be worth it)
None of the given answers is actually correct, as soon as you consider anything outside the ASCII range.
For example in a case insensitive comparison ß should be considered equal to SS if you’re following Unicode’s case mapping rules.
To get correct results the easiest solution is to install Python’s regex module which follows the standard:
import re
import regex
# enable new improved engine instead of backwards compatible v0
regex.DEFAULT_VERSION = regex.VERSION1
print(re.match('ß', 'SS', re.IGNORECASE)) # none
print(regex.match('ß', 'SS', regex.IGNORECASE)) # matches
Another simple solution is to pass a tuple to startswith() for all the cases needed to match e.g. .startswith(('case1', 'case2', ..)).
For example:
>>> 'Hello'.startswith(('He', 'HE'))
True
>>> 'HEllo'.startswith(('He', 'HE'))
True
>>>
In Python 3.8, the fastest solution involves slicing and comparing the prefix, as suggested in this answer:
def startswith(a_source: str, a_prefix: str) -> bool:
source_prefix = a_source[:len(a_prefix)]
return source_prefix.casefold() == a_prefix.casefold()
The second fastest solution uses ctypes (e.g., _wcsicmp.) Note: This is a Windows example.
import ctypes.util
libc_name = ctypes.util.find_library('msvcrt')
libc = ctypes.CDLL(libc_name)
libc._wcsicmp.argtypes = (ctypes.c_wchar_p, ctypes.c_wchar_p)
def startswith(a_source: str, a_prefix: str) -> bool:
source_prefix = a_source[:len(a_prefix)]
return libc._wcsicmp(source_prefix, a_prefix) == 0
The compiled re solution is the third fastest solution, including the cost of compilation. That solution is even slower if the regex module is used for full Unicode support, as suggested in this answer. Each successive match costs around the same as each of the ctypes calls.
lower() and casefold() are expensive because these functions create new Unicode strings by iterating over each character in the source strings, regardless of case, and mapping them accordingly. (See: How is the built-in function str.lower() implemented?) The time spent in that loop increases with each character, so if you’re dealing with short prefixes and long strings, call these functions on only the prefixes.
Another option:
import re
o = re.search('(?i)^we', 'Wednesday')
print(o != None)
Here is how I check whether mystring begins with some string:
>>> mystring.lower().startswith("he")
True
The problem is that mystring is very long (thousands of characters), so the lower() operation takes a lot of time.
QUESTION: Is there a more efficient way?
My unsuccessful attempt:
>>> import re;
>>> mystring.startswith("he", re.I)
False
You could use a regular expression as follows:
In [33]: bool(re.match('he', 'Hello', re.I))
Out[33]: True
In [34]: bool(re.match('el', 'Hello', re.I))
Out[34]: False
On a 2000-character string this is about 20x times faster than lower():
In [38]: s = 'A' * 2000
In [39]: %timeit s.lower().startswith('he')
10000 loops, best of 3: 41.3 us per loop
In [40]: %timeit bool(re.match('el', s, re.I))
100000 loops, best of 3: 2.06 us per loop
If you are matching the same prefix repeatedly, pre-compiling the regex can make a large difference:
In [41]: p = re.compile('he', re.I)
In [42]: %timeit p.match(s)
1000000 loops, best of 3: 351 ns per loop
For short prefixes, slicing the prefix out of the string before converting it to lowercase could be even faster:
In [43]: %timeit s[:2].lower() == 'he'
1000000 loops, best of 3: 287 ns per loop
Relative timings of these approaches will of course depend on the length of the prefix. On my machine the breakeven point seems to be about six characters, which is when the pre-compiled regex becomes the fastest method.
In my experiments, checking every character separately could be even faster:
In [44]: %timeit (s[0] == 'h' or s[0] == 'H') and (s[1] == 'e' or s[1] == 'E')
1000000 loops, best of 3: 189 ns per loop
However, this method only works for prefixes that are known when you’re writing the code, and doesn’t lend itself to longer prefixes.
How about this:
prefix = 'he'
if myVeryLongStr[:len(prefix)].lower() == prefix.lower()
Depending on the performance of .lower(), if prefix was small enough it might be faster to check equality multiple times:
s = 'A' * 2000
prefix = 'he'
ch0 = s[0]
ch1 = s[1]
substr = ch0 == 'h' or ch0 == 'H' and ch1 == 'e' or ch1 == 'E'
Timing (using the same string as NPE):
>>> timeit.timeit("ch0 = s[0]; ch1 = s[1]; ch0 == 'h' or ch0 == 'H' and ch1 == 'e' or ch1 == 'E'", "s = 'A' * 2000")
0.2509511683747405
= 0.25 us per loop
Compared to existing method:
>>> timeit.timeit("s.lower().startswith('he')", "s = 'A' * 2000", number=10000)
0.6162763703208611
= 61.63 us per loop
(This is horrible, of course, but if the code is extremely performance critical then it might be worth it)
None of the given answers is actually correct, as soon as you consider anything outside the ASCII range.
For example in a case insensitive comparison ß should be considered equal to SS if you’re following Unicode’s case mapping rules.
To get correct results the easiest solution is to install Python’s regex module which follows the standard:
import re
import regex
# enable new improved engine instead of backwards compatible v0
regex.DEFAULT_VERSION = regex.VERSION1
print(re.match('ß', 'SS', re.IGNORECASE)) # none
print(regex.match('ß', 'SS', regex.IGNORECASE)) # matches
Another simple solution is to pass a tuple to startswith() for all the cases needed to match e.g. .startswith(('case1', 'case2', ..)).
For example:
>>> 'Hello'.startswith(('He', 'HE'))
True
>>> 'HEllo'.startswith(('He', 'HE'))
True
>>>
In Python 3.8, the fastest solution involves slicing and comparing the prefix, as suggested in this answer:
def startswith(a_source: str, a_prefix: str) -> bool:
source_prefix = a_source[:len(a_prefix)]
return source_prefix.casefold() == a_prefix.casefold()
The second fastest solution uses ctypes (e.g., _wcsicmp.) Note: This is a Windows example.
import ctypes.util
libc_name = ctypes.util.find_library('msvcrt')
libc = ctypes.CDLL(libc_name)
libc._wcsicmp.argtypes = (ctypes.c_wchar_p, ctypes.c_wchar_p)
def startswith(a_source: str, a_prefix: str) -> bool:
source_prefix = a_source[:len(a_prefix)]
return libc._wcsicmp(source_prefix, a_prefix) == 0
The compiled re solution is the third fastest solution, including the cost of compilation. That solution is even slower if the regex module is used for full Unicode support, as suggested in this answer. Each successive match costs around the same as each of the ctypes calls.
lower() and casefold() are expensive because these functions create new Unicode strings by iterating over each character in the source strings, regardless of case, and mapping them accordingly. (See: How is the built-in function str.lower() implemented?) The time spent in that loop increases with each character, so if you’re dealing with short prefixes and long strings, call these functions on only the prefixes.
Another option:
import re
o = re.search('(?i)^we', 'Wednesday')
print(o != None)