Python remove anything that is not a letter or number
Question:
I’m having a little trouble with Python regular expressions.
What is a good way to remove all characters in a string that are not letters or numbers?
Thanks!
Answers:
In the char set matching rule [...]
you can specify ^
as first char to mean “not in”
import re
re.sub("[^0-9a-zA-Z]", # Anything except 0..9, a..z and A..Z
"", # replaced with nothing
"this is a test!!") # in this string
--> 'thisisatest'
'W'
is the same as [^A-Za-z0-9_]
plus accented chars from your locale.
>>> re.sub('W', '', 'text 1, 2, 3...')
'text123'
Maybe you want to keep the spaces or have all the words (and numbers):
>>> re.findall('w+', 'my. text, --without-- (punctuation) 123')
['my', 'text', 'without', 'punctuation', '123']
you can use predefined regex in python : W
corresponds to the set [^a-zA-Z0-9_]
. Then,
import re
s = 'Hello dutrow 123'
re.sub('W', '', s)
--> 'Hellodutrow123'
There are other ways also you may consider e.g. simply loop thru string and skip unwanted chars e.g. assuming you want to delete all ascii chars which are not letter or digits
>>> newstring = [c for c in "a!1#b$2c%3tnx" if c in string.letters + string.digits]
>>> "".join(newstring)
'a1b2c3x'
or use string.translate to map one char to other or delete some chars e.g.
>>> todelete = [ chr(i) for i in range(256) if chr(i) not in string.letters + string.digits ]
>>> todelete = "".join(todelete)
>>> "a!1#b$2c%3tnx".translate(None, todelete)
'a1b2c3x'
this way you need to calculate todelete
list once or todelete
can be hard-coded once and use it everywhere you need to convert string
You need to be more specific:
- What about Unicode “letters”? ie, those with diacriticals.
- What about white space? (I assume this is what you DO want to delete along with punctuation)
- When you say “letters” do you mean
A-Z
and a-z
in ASCII only?
- When you say “numbers” do you mean
0-9
only? What about decimals, separators and exponents?
It gets complex quickly…
A great place to start is an interactive regex site, such as RegExr
You can also get Python specific Python Regex Tool
Also you can try to use isalpha and isnumeric methods the following way:
text = 'base, sample test;'
getVals = lambda x: (c for c in text if c.isalpha() or c.isnumeric())
map(lambda word: ' '.join(getVals(word)): text.split(' '))
[w]
matches (alphanumeric or underscore).
[W]
matches (not (alphanumeric or underscore)), which is equivalent to (not alphanumeric and not underscore)
You need [W_]
to remove ALL non-alphanumerics.
When using re.sub(), it will be much more efficient if you reduce the number of substitutions (expensive) by matching using [W_]+
instead of doing it one at a time.
Now all you need is to define alphanumerics:
str
object, only ASCII A-Za-z0-9:
re.sub(r'[W_]+', '', s)
str
object, only locale-defined alphanumerics:
re.sub(r'[W_]+', '', s, flags=re.LOCALE)
unicode
object, all alphanumerics:
re.sub(ur'[W_]+', u'', s, flags=re.UNICODE)
Examples for str
object:
>>> import re, locale
>>> sall = ''.join(chr(i) for i in xrange(256))
>>> len(sall)
256
>>> re.sub('[W_]+', '', sall)
'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
>>> re.sub('[W_]+', '', sall, flags=re.LOCALE)
'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
>>> locale.setlocale(locale.LC_ALL, '')
'English_Australia.1252'
>>> re.sub('[W_]+', '', sall, flags=re.LOCALE)
'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzx83x8ax8cx8e
x9ax9cx9ex9fxaaxb2xb3xb5xb9xbaxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9
xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd8xd9xdaxdbxdcxddxde
xdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2
xf3xf4xf5xf6xf8xf9xfaxfbxfcxfdxfexff'
# above output wrapped at column 80
Unicode example:
>>> re.sub(ur'[W_]+', u'', u'a_b A_Z x80xFF u0404', flags=re.UNICODE)
u'abAZxffu0404'
I’m having a little trouble with Python regular expressions.
What is a good way to remove all characters in a string that are not letters or numbers?
Thanks!
In the char set matching rule [...]
you can specify ^
as first char to mean “not in”
import re
re.sub("[^0-9a-zA-Z]", # Anything except 0..9, a..z and A..Z
"", # replaced with nothing
"this is a test!!") # in this string
--> 'thisisatest'
'W'
is the same as [^A-Za-z0-9_]
plus accented chars from your locale.
>>> re.sub('W', '', 'text 1, 2, 3...')
'text123'
Maybe you want to keep the spaces or have all the words (and numbers):
>>> re.findall('w+', 'my. text, --without-- (punctuation) 123')
['my', 'text', 'without', 'punctuation', '123']
you can use predefined regex in python : W
corresponds to the set [^a-zA-Z0-9_]
. Then,
import re
s = 'Hello dutrow 123'
re.sub('W', '', s)
--> 'Hellodutrow123'
There are other ways also you may consider e.g. simply loop thru string and skip unwanted chars e.g. assuming you want to delete all ascii chars which are not letter or digits
>>> newstring = [c for c in "a!1#b$2c%3tnx" if c in string.letters + string.digits]
>>> "".join(newstring)
'a1b2c3x'
or use string.translate to map one char to other or delete some chars e.g.
>>> todelete = [ chr(i) for i in range(256) if chr(i) not in string.letters + string.digits ]
>>> todelete = "".join(todelete)
>>> "a!1#b$2c%3tnx".translate(None, todelete)
'a1b2c3x'
this way you need to calculate todelete
list once or todelete
can be hard-coded once and use it everywhere you need to convert string
You need to be more specific:
- What about Unicode “letters”? ie, those with diacriticals.
- What about white space? (I assume this is what you DO want to delete along with punctuation)
- When you say “letters” do you mean
A-Z
anda-z
in ASCII only? - When you say “numbers” do you mean
0-9
only? What about decimals, separators and exponents?
It gets complex quickly…
A great place to start is an interactive regex site, such as RegExr
You can also get Python specific Python Regex Tool
Also you can try to use isalpha and isnumeric methods the following way:
text = 'base, sample test;'
getVals = lambda x: (c for c in text if c.isalpha() or c.isnumeric())
map(lambda word: ' '.join(getVals(word)): text.split(' '))
[w]
matches (alphanumeric or underscore).
[W]
matches (not (alphanumeric or underscore)), which is equivalent to (not alphanumeric and not underscore)
You need [W_]
to remove ALL non-alphanumerics.
When using re.sub(), it will be much more efficient if you reduce the number of substitutions (expensive) by matching using [W_]+
instead of doing it one at a time.
Now all you need is to define alphanumerics:
str
object, only ASCII A-Za-z0-9:
re.sub(r'[W_]+', '', s)
str
object, only locale-defined alphanumerics:
re.sub(r'[W_]+', '', s, flags=re.LOCALE)
unicode
object, all alphanumerics:
re.sub(ur'[W_]+', u'', s, flags=re.UNICODE)
Examples for str
object:
>>> import re, locale
>>> sall = ''.join(chr(i) for i in xrange(256))
>>> len(sall)
256
>>> re.sub('[W_]+', '', sall)
'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
>>> re.sub('[W_]+', '', sall, flags=re.LOCALE)
'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
>>> locale.setlocale(locale.LC_ALL, '')
'English_Australia.1252'
>>> re.sub('[W_]+', '', sall, flags=re.LOCALE)
'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzx83x8ax8cx8e
x9ax9cx9ex9fxaaxb2xb3xb5xb9xbaxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9
xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd8xd9xdaxdbxdcxddxde
xdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2
xf3xf4xf5xf6xf8xf9xfaxfbxfcxfdxfexff'
# above output wrapped at column 80
Unicode example:
>>> re.sub(ur'[W_]+', u'', u'a_b A_Z x80xFF u0404', flags=re.UNICODE)
u'abAZxffu0404'