Python: splitting string by all space characters


To split strings by spaces in python, one usually uses split method of the string without parameters:

>>> 'atb cnd'.split()
['a', 'b', 'c', 'd']

But yesterday I ran across a string that used ZERO WIDTH SPACE between words as well. Having turned my new knowledge in a short black magic performance (among JavaScript folks), I would like to ask how to better split by all whitespace characters, since the split is not enough:

>>> u'au200bc d'.split()
[u'au200bc', u'd']


it seems the solution suggested by sth gererally works but depends on some OS settings or Python compilation options. It would be nice to know the reason for sure (and if the setting can be switched on in Windows).

cptphil found a great link that makes everything clear:

So I contacted the Unicode Technical Committee about the issue and received a promptly received a response back. They pointed that the ZWSP was, once upon a time considered white space but that was changed in Unicode 4.0.1

A quotation from unicode site:

Changing U+200B Zero Width Space from Zs to Cf (2003.10.27)

There have been persistent problems with usage of the U+200B Zero Width Space (ZWSP). The function of this character is to allow a line break at positions where it normally would not be allowed, and is thus functionally a format character with a general category of Cf. This behavior is well documented in the Unicode Standard, and the character not considered a Whitespace character in the Unicode Character Database. However, for historical reasons the general category is still Zs (Space Separator), which causes the character to be misused. ZWSP is also the only Zs character that is not Whitespace. The general category can cause misinterpretation of rule D13 Base character as allowing ZWSP as a base for combining marks.

The proposal is to change the general category of U+200B from Zs to Cf.

Resolution: Closed. The general category of U+200B will be changed from Zs to Cf in Unicode version 4.0.1.

The change was then reflected in Python. The result of u'u200B'.isspace() in Python 2.5.4 and 2.6.5 is True, in Python 2.7.1 it is already False.

For other space characters regular split is enough:

>>> u'au200Ac'.split()
[u'a', u'c']

And if that is not enough for you, add characters one by one as Gabi Purcaru suggests below.

Asked By: newtover



You can use the ‘re’ module and pass a separator to ‘split’:

Answered By: Savino Sguera

You can use a regular expression with enabled Unicode matching:

>>> re.split(r'(?u)s', u'au200bc d')
[u'a', u'c', u'd']
Answered By: sth

You can use re.split, like this:

import re
re.split(u's|u200b', your_string)
Answered By: Gabi Purcaru


It turns out that u200b is not technically defined as whitespace , and so python does not recognize it as matching s even with the unicode flag on. So it must be treated as an non-whitespace character.

import re

re.split(ur"[u200bs]+", "some string", flags=re.UNICODE)
Answered By: philofinfinitejest

Can you use something like this?

re.split(r's+', your_string, re.UNICODE)
Answered By: Ian Clelland

Just use split:

>>> u'u200b'.isspace()
Answered By: PaulMcG
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.