Python: splitting string by all space characters
Question:
To split strings by spaces in python, one usually uses split method of the string without parameters:
>>> 'atb cnd'.split()
['a', 'b', 'c', 'd']
But yesterday I ran across a string that used ZERO WIDTH SPACE between words as well. Having turned my new knowledge in a short black magic performance (among JavaScript folks), I would like to ask how to better split by all whitespace characters, since the split is not enough:
>>> u'au200bc d'.split()
[u'au200bc', u'd']
UPD1
it seems the solution suggested by sth gererally works but depends on some OS settings or Python compilation options. It would be nice to know the reason for sure (and if the setting can be switched on in Windows).
UPD2
cptphil found a great link that makes everything clear:
So I contacted the Unicode Technical Committee about the issue and received a promptly received a response back. They pointed that the ZWSP was, once upon a time considered white space but that was changed in Unicode 4.0.1
A quotation from unicode site:
Changing U+200B Zero Width Space from Zs to Cf (2003.10.27)
There have been persistent problems with usage of the U+200B Zero Width Space (ZWSP). The function of this character is to allow a line break at positions where it normally would not be allowed, and is thus functionally a format character with a general category of Cf. This behavior is well documented in the Unicode Standard, and the character not considered a Whitespace character in the Unicode Character Database. However, for historical reasons the general category is still Zs (Space Separator), which causes the character to be misused. ZWSP is also the only Zs character that is not Whitespace. The general category can cause misinterpretation of rule D13 Base character as allowing ZWSP as a base for combining marks.
The proposal is to change the general category of U+200B from Zs to Cf.
Resolution: Closed. The general category of U+200B will be changed from Zs to Cf in Unicode version 4.0.1.
The change was then reflected in Python. The result of u'u200B'.isspace() in Python 2.5.4 and 2.6.5 is True, in Python 2.7.1 it is already False.
For other space characters regular split is enough:
>>> u'au200Ac'.split()
[u'a', u'c']
And if that is not enough for you, add characters one by one as Gabi Purcaru suggests below.
Answers:
You can use the ‘re’ module and pass a separator to ‘split’: http://docs.python.org/library/re.html#re.split
You can use a regular expression with enabled Unicode matching:
>>> re.split(r'(?u)s', u'au200bc d')
[u'a', u'c', u'd']
You can use re.split, like this:
import re
re.split(u's|u200b', your_string)
Edit
It turns out that u200b is not technically defined as whitespace , and so python does not recognize it as matching s even with the unicode flag on. So it must be treated as an non-whitespace character.
http://en.wikipedia.org/wiki/Whitespace_character#Unicode
http://bugs.python.org/issue13391
import re
re.split(ur"[u200bs]+", "some string", flags=re.UNICODE)
Can you use something like this?
re.split(r's+', your_string, re.UNICODE)
Just use split:
>>> u'u200b'.isspace()
True
To split strings by spaces in python, one usually uses split method of the string without parameters:
>>> 'atb cnd'.split()
['a', 'b', 'c', 'd']
But yesterday I ran across a string that used ZERO WIDTH SPACE between words as well. Having turned my new knowledge in a short black magic performance (among JavaScript folks), I would like to ask how to better split by all whitespace characters, since the split is not enough:
>>> u'au200bc d'.split()
[u'au200bc', u'd']
UPD1
it seems the solution suggested by sth gererally works but depends on some OS settings or Python compilation options. It would be nice to know the reason for sure (and if the setting can be switched on in Windows).
UPD2
cptphil found a great link that makes everything clear:
So I contacted the Unicode Technical Committee about the issue and received a promptly received a response back. They pointed that the ZWSP was, once upon a time considered white space but that was changed in Unicode 4.0.1
A quotation from unicode site:
Changing U+200B Zero Width Space from Zs to Cf (2003.10.27)
There have been persistent problems with usage of the U+200B Zero Width Space (ZWSP). The function of this character is to allow a line break at positions where it normally would not be allowed, and is thus functionally a format character with a general category of Cf. This behavior is well documented in the Unicode Standard, and the character not considered a Whitespace character in the Unicode Character Database. However, for historical reasons the general category is still Zs (Space Separator), which causes the character to be misused. ZWSP is also the only Zs character that is not Whitespace. The general category can cause misinterpretation of rule D13 Base character as allowing ZWSP as a base for combining marks.
The proposal is to change the general category of U+200B from Zs to Cf.
Resolution: Closed. The general category of U+200B will be changed from Zs to Cf in Unicode version 4.0.1.
The change was then reflected in Python. The result of u'u200B'.isspace() in Python 2.5.4 and 2.6.5 is True, in Python 2.7.1 it is already False.
For other space characters regular split is enough:
>>> u'au200Ac'.split()
[u'a', u'c']
And if that is not enough for you, add characters one by one as Gabi Purcaru suggests below.
You can use the ‘re’ module and pass a separator to ‘split’: http://docs.python.org/library/re.html#re.split
You can use a regular expression with enabled Unicode matching:
>>> re.split(r'(?u)s', u'au200bc d')
[u'a', u'c', u'd']
You can use re.split, like this:
import re
re.split(u's|u200b', your_string)
Edit
It turns out that u200b is not technically defined as whitespace , and so python does not recognize it as matching s even with the unicode flag on. So it must be treated as an non-whitespace character.
http://en.wikipedia.org/wiki/Whitespace_character#Unicode
http://bugs.python.org/issue13391
import re
re.split(ur"[u200bs]+", "some string", flags=re.UNICODE)
Can you use something like this?
re.split(r's+', your_string, re.UNICODE)
Just use split:
>>> u'u200b'.isspace()
True