Delete every non utf-8 symbols from string

Question

I have a big amount of files and parser. What I Have to do is strip all non utf-8 symbols and put data in mongodb.
Currently I have code like this.

with open(fname, "r") as fp:
    for line in fp:
        line = line.strip()
        line = line.decode('utf-8', 'ignore')
        line = line.encode('utf-8', 'ignore')

somehow I still get an error

bson.errors.InvalidStringData: strings in documents must be valid UTF-8: 
1/b62010montecassianomcirxe2x86x90ta0xe2x86x90008923304320733/290066010401040101506055soccorin

I don’t get it. Is there some simple way to do it?

UPD: seems like Python and Mongo don’t agree about definition of Utf-8 Valid string.

Asked By: Darth Kotik

||

Source

Answer 1

Try below code line instead of last two lines. Hope it helps:

line=line.decode('utf-8','ignore').encode("utf-8")

Answered By: Irshad Bhat

Answer 2

Example to handle no utf-8 characters

import string

test=u"nnnnnnnnnnnnnnHi <<First Name>>nthis is filler text xa325 more filler.nadditilnal filler.nnyet morexa0still morexa0filler.nnxa0nnnnnmorennnnnnnnnnnnnnnnnnnfiller.x03ntttttt    almost there nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnthe endnnnnnnnnnnnnn"

print ''.join(x for x in test if x in string.printable)

Answered By: Hafiz Muhammad Shafiq

Answer 3

For python 3, as mentioned in a comment in this thread, you can do:

line = bytes(line, 'utf-8').decode('utf-8', 'ignore')

The ‘ignore’ parameter prevents an error from being raised if any characters are unable to be decoded.

If your line is already a bytes object (e.g. b'my string') then you just need to decode it with decode('utf-8', 'ignore').

Answered By: Alex

Answer 4

with open(fname, "r") as fp:
for line in fp:
    line = line.strip()
    line = line.decode('cp1252').encode('utf-8')

Answered By: Willem

Delete every non utf-8 symbols from string

Question:

Answers: