Fast way to filter illegal xml unicode chars in python?

Question

The XML specification lists a bunch of Unicode characters that are either illegal or “discouraged”. Given a string, how can I remove all illegal characters from it?

I came up with the following regular expression, but it’s a bit of a mouthful.

illegal_xml_re = re.compile(u'[x00-x08x0b-x1fx7f-x84x86-x9fud800-udfffufdd0-ufddfufffe-uffff]')
clean = illegal_xml_re.sub('', dirty)

(Python 2.5 doesn’t know about Unicode chars above 0xFFFF, so no need to filter those.)

Asked By: itsadok

||

Source

Answer 1

You could also use unicode’s translate method to delete selected codepoints. However, the mapping you have is pretty big (2128 codepoints) and that might make it much slower than just using a regex:

ranges = [(0, 8), (0xb, 0x1f), (0x7f, 0x84), (0x86, 0x9f), (0xd800, 0xdfff), (0xfdd0, 0xfddf), (0xfffe, 0xffff)]
# fromkeys creates  the wanted (codepoint -> None) mapping
nukemap = dict.fromkeys(r for start, end in ranges for r in range(start, end+1))
clean = dirty.translate(nukemap)

Answered By: u0b34a0f6ae

Answer 2

Recently we (Trac XmlRpcPlugin maintainers) have been notified of the fact that the regular expression above strips surrogate pairs on Python narrow builds (see th:comment:13:ticket:11050) . An alternative approach consists in using the following regex (see th:changeset:13729) .

_illegal_unichrs = [(0x00, 0x08), (0x0B, 0x0C), (0x0E, 0x1F), 
                        (0x7F, 0x84), (0x86, 0x9F), 
                        (0xFDD0, 0xFDDF), (0xFFFE, 0xFFFF)] 
if sys.maxunicode >= 0x10000:  # not narrow build 
        _illegal_unichrs.extend([(0x1FFFE, 0x1FFFF), (0x2FFFE, 0x2FFFF), 
                                 (0x3FFFE, 0x3FFFF), (0x4FFFE, 0x4FFFF), 
                                 (0x5FFFE, 0x5FFFF), (0x6FFFE, 0x6FFFF), 
                                 (0x7FFFE, 0x7FFFF), (0x8FFFE, 0x8FFFF), 
                                 (0x9FFFE, 0x9FFFF), (0xAFFFE, 0xAFFFF), 
                                 (0xBFFFE, 0xBFFFF), (0xCFFFE, 0xCFFFF), 
                                 (0xDFFFE, 0xDFFFF), (0xEFFFE, 0xEFFFF), 
                                 (0xFFFFE, 0xFFFFF), (0x10FFFE, 0x10FFFF)]) 

_illegal_ranges = ["%s-%s" % (unichr(low), unichr(high)) 
                   for (low, high) in _illegal_unichrs] 
_illegal_xml_chars_RE = re.compile(u'[%s]' % u''.join(_illegal_ranges))

p.s. See this post on surrogates explaining what they are for .

Update so as to not to match (replace) 0x0D which is a valid XML character.

Answered By: Olemis Lang

Answer 3

Here’s an updated version of Olemis Lang’s answer for Python 3:

import re
import sys

illegal_unichrs = [(0x00, 0x08), (0x0B, 0x0C), (0x0E, 0x1F),
                   (0x7F, 0x84), (0x86, 0x9F),
                   (0xFDD0, 0xFDDF), (0xFFFE, 0xFFFF)]
if sys.maxunicode >= 0x10000:  # not narrow build
    illegal_unichrs.extend([(0x1FFFE, 0x1FFFF), (0x2FFFE, 0x2FFFF),
                            (0x3FFFE, 0x3FFFF), (0x4FFFE, 0x4FFFF),
                            (0x5FFFE, 0x5FFFF), (0x6FFFE, 0x6FFFF),
                            (0x7FFFE, 0x7FFFF), (0x8FFFE, 0x8FFFF),
                            (0x9FFFE, 0x9FFFF), (0xAFFFE, 0xAFFFF),
                            (0xBFFFE, 0xBFFFF), (0xCFFFE, 0xCFFFF),
                            (0xDFFFE, 0xDFFFF), (0xEFFFE, 0xEFFFF),
                            (0xFFFFE, 0xFFFFF), (0x10FFFE, 0x10FFFF)])

illegal_ranges = [fr'{chr(low)}-{chr(high)}' for (low, high) in illegal_unichrs]
xml_illegal_character_regex = '[' + ''.join(illegal_ranges) + ']'
illegal_xml_chars_re = re.compile(xml_illegal_character_regex)
# filtered_string = illegal_xml_chars_re.sub('', original_string)

Answered By: Megacier

Fast way to filter illegal xml unicode chars in python?

Question:

Answers: