How to replace unicode characters in string with something else python?
Question:
I have a string that I got from reading a HTML webpage with bullets that have a symbol like “•” because of the bulleted list. Note that the text is an HTML source from a webpage using Python 2.7’s urllib2.read(webaddress)
.
I know the unicode character for the bullet character as U+2022
, but how do I actually replace that unicode character with something else?
I tried doing
str.replace("•", "something")
but it does not appear to work… how do I do this?
Answers:
import re
regex = re.compile("u'2022'",re.UNICODE)
newstring = re.sub(regex, something, yourstring, <optional flags>)
-
Decode the string to Unicode. Assuming it’s UTF-8-encoded:
str.decode("utf-8")
-
Call the replace
method and be sure to pass it a Unicode string as its first argument:
str.decode("utf-8").replace(u"u2022", "*")
-
Encode back to UTF-8, if needed:
str.decode("utf-8").replace(u"u2022", "*").encode("utf-8")
(Fortunately, Python 3 puts a stop to this mess. Step 3 should really only be performed just prior to I/O. Also, mind you that calling a string str
shadows the built-in type str
.)
Encode string as unicode.
>>> special = u"u2022"
>>> abc = u'ABC•def'
>>> abc.replace(special,'X')
u'ABCXdef'
Funny the answer is hidden in among the answers.
str.replace("•", "something")
would work if you use the right semantics.
str.replace(u"u2022","something")
works wonders 😉 , thnx to RParadox for the hint.
If you want to remove all u character. Code below for you
def replace_unicode_character(self, content: str):
content = content.encode('utf-8')
if "\x80" in str(content):
count_unicode = 0
i = 0
while i < len(content):
if "\x" in str(content[i:i + 1]):
if count_unicode % 3 == 0:
content = content[:i] + b'x80x80x80' + content[i + 3:]
i += 2
count_unicode += 1
i += 1
content = content.replace(b'x80x80x80', b'')
return content.decode('utf-8')
Try this one.
you will get the output in a normal string
str.encode().decode('unicode-escape')
and after that, you can perform any replacement.
str.replace('•','something')
str1 = "This is Pythonu500cPool"
Encode the string to ASCII and replace all the utf-8 characters with ‘?’.
str1 = str1.encode("ascii", "replace")
Decode the byte stream to string.
str1 = str1.decode(encoding="utf-8", errors="ignore")
Replace the question mark with the desired character.
str1 = str1.replace("?"," ")
I have a string that I got from reading a HTML webpage with bullets that have a symbol like “•” because of the bulleted list. Note that the text is an HTML source from a webpage using Python 2.7’s urllib2.read(webaddress)
.
I know the unicode character for the bullet character as U+2022
, but how do I actually replace that unicode character with something else?
I tried doing
str.replace("•", "something")
but it does not appear to work… how do I do this?
import re
regex = re.compile("u'2022'",re.UNICODE)
newstring = re.sub(regex, something, yourstring, <optional flags>)
-
Decode the string to Unicode. Assuming it’s UTF-8-encoded:
str.decode("utf-8")
-
Call the
replace
method and be sure to pass it a Unicode string as its first argument:str.decode("utf-8").replace(u"u2022", "*")
-
Encode back to UTF-8, if needed:
str.decode("utf-8").replace(u"u2022", "*").encode("utf-8")
(Fortunately, Python 3 puts a stop to this mess. Step 3 should really only be performed just prior to I/O. Also, mind you that calling a string str
shadows the built-in type str
.)
Encode string as unicode.
>>> special = u"u2022"
>>> abc = u'ABC•def'
>>> abc.replace(special,'X')
u'ABCXdef'
Funny the answer is hidden in among the answers.
str.replace("•", "something")
would work if you use the right semantics.
str.replace(u"u2022","something")
works wonders 😉 , thnx to RParadox for the hint.
If you want to remove all u character. Code below for you
def replace_unicode_character(self, content: str):
content = content.encode('utf-8')
if "\x80" in str(content):
count_unicode = 0
i = 0
while i < len(content):
if "\x" in str(content[i:i + 1]):
if count_unicode % 3 == 0:
content = content[:i] + b'x80x80x80' + content[i + 3:]
i += 2
count_unicode += 1
i += 1
content = content.replace(b'x80x80x80', b'')
return content.decode('utf-8')
Try this one.
you will get the output in a normal string
str.encode().decode('unicode-escape')
and after that, you can perform any replacement.
str.replace('•','something')
str1 = "This is Pythonu500cPool"
Encode the string to ASCII and replace all the utf-8 characters with ‘?’.
str1 = str1.encode("ascii", "replace")
Decode the byte stream to string.
str1 = str1.decode(encoding="utf-8", errors="ignore")
Replace the question mark with the desired character.
str1 = str1.replace("?"," ")