What's the easiest way to escape HTML in Python?
Question:
cgi.escape seems like one possible choice. Does it work well? Is there something that is considered better?
Answers:
html.escape is the correct answer now, it used to be cgi.escape in python before 3.2. It escapes:
< to <> to >& to &
That is enough for all HTML.
EDIT: If you have non-ascii chars you also want to escape, for inclusion in another encoded document that uses a different encoding, like Craig says, just use:
data.encode('ascii', 'xmlcharrefreplace')
Don’t forget to decode data to unicode first, using whatever encoding it was encoded.
However in my experience that kind of encoding is useless if you just work with unicode all the time from start. Just encode at the end to the encoding specified in the document header (utf-8 for maximum compatibility).
Example:
>>> cgi.escape(u'<a>bá</a>').encode('ascii', 'xmlcharrefreplace')
'<a>bá</a>
Also worth of note (thanks Greg) is the extra quote parameter cgi.escape takes. With it set to True, cgi.escape also escapes double quote chars (") so you can use the resulting value in a XML/HTML attribute.
EDIT: Note that cgi.escape has been deprecated in Python 3.2 in favor of html.escape, which does the same except that quote defaults to True.
cgi.escape should be good to escape HTML in the limited sense of escaping the HTML tags and character entities.
But you might have to also consider encoding issues: if the HTML you want to quote has non-ASCII characters in a particular encoding, then you would also have to take care that you represent those sensibly when quoting. Perhaps you could convert them to entities. Otherwise you should ensure that the correct encoding translations are done between the “source” HTML and the page it’s embedded in, to avoid corrupting the non-ASCII characters.
In Python 3.2 a new html module was introduced, which is used for escaping reserved characters from HTML markup.
It has one function escape():
>>> import html
>>> html.escape('x > 2 && x < 7 single quote: ' double quote: "')
'x > 2 && x < 7 single quote: ' double quote: "'
If you wish to escape HTML in a URL:
This is probably NOT what the OP wanted (the question doesn’t clearly indicate in which context the escaping is meant to be used), but Python’s native library urllib has a method to escape HTML entities that need to be included in a URL safely.
The following is an example:
#!/usr/bin/python
from urllib import quote
x = '+<>^&'
print quote(x) # prints '%2B%3C%3E%5E%26'
cgi.escape extended
This version improves cgi.escape. It also preserves whitespace and newlines. Returns a unicode string.
def escape_html(text):
"""escape strings for display in HTML"""
return cgi.escape(text, quote=True).
replace(u'n', u'<br />').
replace(u't', u' ').
replace(u' ', u' ')
for example
>>> escape_html('<foo>nfoot"bar"')
u'<foo><br />foo "bar"'
For legacy code in Python 2.7, can do it via BeautifulSoup4:
>>> bs4.dammit import EntitySubstitution
>>> esub = EntitySubstitution()
>>> esub.substitute_html("r&d")
'r&d'
Not the easiest way, but still straightforward. The main difference from cgi.escape module – it still will work properly if you already have & in your text. As you see from comments to it:
cgi.escape version
def escape(s, quote=None):
'''Replace special characters "&", "<" and ">" to HTML-safe sequences.
If the optional flag quote is true, the quotation mark character (")
is also translated.'''
s = s.replace("&", "&") # Must be done first!
s = s.replace("<", "<")
s = s.replace(">", ">")
if quote:
s = s.replace('"', """)
return s
regex version
QUOTE_PATTERN = r"""([&<>"'])(?!(amp|lt|gt|quot|#39);)"""
def escape(word):
"""
Replaces special characters <>&"' to HTML-safe sequences.
With attention to already escaped characters.
"""
replace_with = {
'<': '>',
'>': '<',
'&': '&',
'"': '"', # should be escaped in attributes
"'": ''' # should be escaped in attributes
}
quote_pattern = re.compile(QUOTE_PATTERN)
return re.sub(quote_pattern, lambda x: replace_with[x.group(0)], word)
There is also the excellent markupsafe package.
>>> from markupsafe import Markup, escape
>>> escape("<script>alert(document.cookie);</script>")
Markup(u'<script>alert(document.cookie);</script>')
The markupsafe package is well engineered, and probably the most versatile and Pythonic way to go about escaping, IMHO, because:
- the return (
Markup) is a class derived from unicode (i.e. isinstance(escape('str'), unicode) == True
- it properly handles unicode input
- it works in Python (2.6, 2.7, 3.3, and pypy)
- it respects custom methods of objects (i.e. objects with a
__html__ property) and template overloads (__html_format__).
No libraries, pure python, safely escapes text into html text:
text.replace('&', '&').replace('>', '>').replace('<', '<'
).replace(''',''').replace('"','"').encode('ascii', 'xmlcharrefreplace')
cgi.escape seems like one possible choice. Does it work well? Is there something that is considered better?
html.escape is the correct answer now, it used to be cgi.escape in python before 3.2. It escapes:
<to<>to>&to&
That is enough for all HTML.
EDIT: If you have non-ascii chars you also want to escape, for inclusion in another encoded document that uses a different encoding, like Craig says, just use:
data.encode('ascii', 'xmlcharrefreplace')
Don’t forget to decode data to unicode first, using whatever encoding it was encoded.
However in my experience that kind of encoding is useless if you just work with unicode all the time from start. Just encode at the end to the encoding specified in the document header (utf-8 for maximum compatibility).
Example:
>>> cgi.escape(u'<a>bá</a>').encode('ascii', 'xmlcharrefreplace')
'<a>bá</a>
Also worth of note (thanks Greg) is the extra quote parameter cgi.escape takes. With it set to True, cgi.escape also escapes double quote chars (") so you can use the resulting value in a XML/HTML attribute.
EDIT: Note that cgi.escape has been deprecated in Python 3.2 in favor of html.escape, which does the same except that quote defaults to True.
cgi.escape should be good to escape HTML in the limited sense of escaping the HTML tags and character entities.
But you might have to also consider encoding issues: if the HTML you want to quote has non-ASCII characters in a particular encoding, then you would also have to take care that you represent those sensibly when quoting. Perhaps you could convert them to entities. Otherwise you should ensure that the correct encoding translations are done between the “source” HTML and the page it’s embedded in, to avoid corrupting the non-ASCII characters.
In Python 3.2 a new html module was introduced, which is used for escaping reserved characters from HTML markup.
It has one function escape():
>>> import html
>>> html.escape('x > 2 && x < 7 single quote: ' double quote: "')
'x > 2 && x < 7 single quote: ' double quote: "'
If you wish to escape HTML in a URL:
This is probably NOT what the OP wanted (the question doesn’t clearly indicate in which context the escaping is meant to be used), but Python’s native library urllib has a method to escape HTML entities that need to be included in a URL safely.
The following is an example:
#!/usr/bin/python
from urllib import quote
x = '+<>^&'
print quote(x) # prints '%2B%3C%3E%5E%26'
cgi.escape extended
This version improves cgi.escape. It also preserves whitespace and newlines. Returns a unicode string.
def escape_html(text):
"""escape strings for display in HTML"""
return cgi.escape(text, quote=True).
replace(u'n', u'<br />').
replace(u't', u' ').
replace(u' ', u' ')
for example
>>> escape_html('<foo>nfoot"bar"')
u'<foo><br />foo "bar"'
For legacy code in Python 2.7, can do it via BeautifulSoup4:
>>> bs4.dammit import EntitySubstitution
>>> esub = EntitySubstitution()
>>> esub.substitute_html("r&d")
'r&d'
Not the easiest way, but still straightforward. The main difference from cgi.escape module – it still will work properly if you already have & in your text. As you see from comments to it:
cgi.escape version
def escape(s, quote=None):
'''Replace special characters "&", "<" and ">" to HTML-safe sequences.
If the optional flag quote is true, the quotation mark character (")
is also translated.'''
s = s.replace("&", "&") # Must be done first!
s = s.replace("<", "<")
s = s.replace(">", ">")
if quote:
s = s.replace('"', """)
return s
regex version
QUOTE_PATTERN = r"""([&<>"'])(?!(amp|lt|gt|quot|#39);)"""
def escape(word):
"""
Replaces special characters <>&"' to HTML-safe sequences.
With attention to already escaped characters.
"""
replace_with = {
'<': '>',
'>': '<',
'&': '&',
'"': '"', # should be escaped in attributes
"'": ''' # should be escaped in attributes
}
quote_pattern = re.compile(QUOTE_PATTERN)
return re.sub(quote_pattern, lambda x: replace_with[x.group(0)], word)
There is also the excellent markupsafe package.
>>> from markupsafe import Markup, escape
>>> escape("<script>alert(document.cookie);</script>")
Markup(u'<script>alert(document.cookie);</script>')
The markupsafe package is well engineered, and probably the most versatile and Pythonic way to go about escaping, IMHO, because:
- the return (
Markup) is a class derived from unicode (i.e.isinstance(escape('str'), unicode) == True - it properly handles unicode input
- it works in Python (2.6, 2.7, 3.3, and pypy)
- it respects custom methods of objects (i.e. objects with a
__html__property) and template overloads (__html_format__).
No libraries, pure python, safely escapes text into html text:
text.replace('&', '&').replace('>', '>').replace('<', '<'
).replace(''',''').replace('"','"').encode('ascii', 'xmlcharrefreplace')