How to filter (or replace) unicode characters that would take more than 3 bytes in UTF-8?

Encode as UTF-16, then reencode as UTF-8.

>>> t = u' '
>>> e = t.encode('utf-16le')
>>> ''.join(unichr(x).encode('utf-8') for x in struct.unpack('<' + 'H' * (len(e) // 2), e))
'xedxa0xb5xedxb0x9fxedxa0xb5xedxb0xa8xedxa0xb5xedxb0xa8'

Note that you can’t encode after joining, since the surrogate pairs may be decoded before reencoding.

EDIT:

MySQL (at least 5.1.47) has no problem dealing with surrogate pairs:

mysql> create table utf8test (t character(128)) collate utf8_general_ci;
Query OK, 0 rows affected (0.12 sec)

  ...

>>> cxn = MySQLdb.connect(..., charset='utf8')
>>> csr = cxn.cursor()
>>> t = u' '
>>> e = t.encode('utf-16le')
>>> v = ''.join(unichr(x).encode('utf-8') for x in struct.unpack('<' + 'H' * (len(e) // 2), e))
>>> v
'xedxa0xb5xedxb0x9fxedxa0xb5xedxb0xa8xedxa0xb5xedxb0xa8'
>>> csr.execute('insert into utf8test (t) values (%s)', (v,))
1L
>>> csr.execute('select * from utf8test')
1L
>>> r = csr.fetchone()
>>> r
(u'ud835udc1fud835udc28ud835udc28',)
>>> print r[0]
 

I’m guessing it’s not the fastest, but quite straightforward (“pythonic” 🙂 :

def max3bytes(unicode_string):
    return u''.join(uc if uc <= u'uffff' else u'ufffd' for uc in unicode_string)

NB: this code does not take into account the fact that Unicode has surrogate characters in the ranges U+D800-U+DFFF.

And just for the fun of it, an itertools monstrosity 🙂

import itertools as it, operator as op

def max3bytes(unicode_string):

    # sequence of pairs of (char_in_string, u'N{REPLACEMENT CHARACTER}')
    pairs= it.izip(unicode_string, it.repeat(u'ufffd'))

    # is the argument less than or equal to 65535?
    selector= ft.partial(op.le, 65535)

    # using the character ordinals, return 0 or 1 based on `selector`
    indexer= it.imap(selector, it.imap(ord, unicode_string))

    # now pick the correct item for all pairs
    return u''.join(it.imap(tuple.__getitem__, pairs, indexer))

According to the MySQL 5.1 documentation: “The ucs2 and utf8 character sets do not support supplementary characters that lie outside the BMP.” This indicates that there might be a problem with surrogate pairs.

Note that the Unicode standard 5.2 chapter 3 actually forbids encoding a surrogate pair as two 3-byte UTF-8 sequences instead of one 4-byte UTF-8 sequence … see for example page 93 “””Because surrogate code points are not Unicode scalar values, any UTF-8 byte sequence that would otherwise map to code points D800..DFFF is ill-formed.””” However this proscription is as far as I know largely unknown or ignored.

It may well be a good idea to check what MySQL does with surrogate pairs. If they are not to be retained, this code will provide a simple-enough check:

all(uc < u'ud800' or u'ue000' <= uc <= u'uffff' for uc in unicode_string)

and this code will replace any “nasties” with uufffd:

u''.join(
    uc if uc < u'ud800' or u'ue000' <= uc <= u'uffff' else u'ufffd'
    for uc in unicode_string
    )

You may skip the decoding and encoding steps and directly detect the value of the first byte (8-bit string) of each character. According to UTF-8:

#1-byte characters have the following format: 0xxxxxxx
#2-byte characters have the following format: 110xxxxx 10xxxxxx
#3-byte characters have the following format: 1110xxxx 10xxxxxx 10xxxxxx
#4-byte characters have the following format: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

According to that, you only need to check the value of only the first byte of each character to filter out 4-byte characters:

def filter_4byte_chars(s):
    i = 0
    j = len(s)
    # you need to convert
    # the immutable string
    # to a mutable list first
    s = list(s)
    while i < j:
        # get the value of this byte
        k = ord(s[i])
        # this is a 1-byte character, skip to the next byte
        if k <= 127:
            i += 1
        # this is a 2-byte character, skip ahead by 2 bytes
        elif k < 224:
            i += 2
        # this is a 3-byte character, skip ahead by 3 bytes
        elif k < 240:
            i += 3
        # this is a 4-byte character, remove it and update
        # the length of the string we need to check
        else:
            s[i:i+4] = []
            j -= 4
    return ''.join(s)

Skipping the decoding and encoding parts will save you some time and for smaller strings that mostly have 1-byte characters this could even be faster than the regular expression filtering.

This does more than filtering out just 3+ byte UTF-8 unicode characters. It removes unicode but tries to do that in a gentle way and replace it with relevant ASCII characters if possible. It can be a blessing in the future if you don’t have for example a dozen of various unicode apostrophes and unicode quotation marks in your text (usually coming from Apple handhelds) but only the regular ASCII apostrophes and quotations.

unicodedata.normalize("NFKD", sentence).encode("ascii", "ignore")

This is robust, I use it with some more guards:

import unicodedata

def neutralize_unicode(value):
    """
    Taking care of special characters as gently as possible

    Args:
        value (string): input string, can contain unicode characters

    Returns:
        :obj:`string` where the unicode characters are replaced with standard
        ASCII counterparts (for example en-dash and em-dash with regular dash,
        apostrophe and quotation variations with the standard ones) or taken
        out if there's no substitute.
    """
    if not value or not isinstance(value, basestring):
        return value

    if isinstance(value, str):
        return value

    return unicodedata.normalize("NFKD", value).encode("ascii", "ignore")

This is Python 2 BTW.