Python bottle requests and unicode
Question:
I’m building a small RESTful API with bottle in python and am currently experiencing an issue with character encodings when working with the request object.
Hitting up http://server.com/api?q=äöü and looking at request.query['q'] on the server gets me “äöü”, which is obviously not what I’m looking for.
Same goes for a POST request containing the form-urlencoded key q with the value äöü. request.forms.get('q') contains “äöü”.
What’s going on here? I don’t really have the option of decoding these elements with a different encoding or do I? Is there a general option for bottle to store these in unicode?
Thanks.
Answers:
request.query['q'] and forms.get('q') return the raw byte value submitted by the web browser. The value äöü, submitted by a browser as UTF-8 encoded bytes, is 'xc3xa4xc3xb6xc3xbc'.
If you print that byte string, and the place you’re printing it to interprets it as ISO-8859-1, or the similar Windows code page 1252, you will get äöü. If you are debugging by printing to a Windows command prompt, or a file displayed by Notepad, that’s why.
If you use the alternative direct property access request.query.q or forms.q Bottle will give you Unicode strings instead, decoded from the byte version using UTF-8. It’s usually best to work with these Unicode strings wherever you can. (Though still you may have trouble printing them to console. The Windows command prompt is notoriously terrible at coping with non-ASCII characters, and as such is a bad place to be debugging Unicode issues.)
in this case, to convert it ,I did like this search_field.encode(“ISO-8859-1”).decode(“utf-8”)
There’s also the alternative of using def getunicode(self, name, default=None, encoding=None)
Instead of using
request.forms.get('some_form_field_name')
try
request.forms.getunicode('some_form_field_name')
That’ll handle non-latin characters.
I’m building a small RESTful API with bottle in python and am currently experiencing an issue with character encodings when working with the request object.
Hitting up http://server.com/api?q=äöü and looking at request.query['q'] on the server gets me “äöü”, which is obviously not what I’m looking for.
Same goes for a POST request containing the form-urlencoded key q with the value äöü. request.forms.get('q') contains “äöü”.
What’s going on here? I don’t really have the option of decoding these elements with a different encoding or do I? Is there a general option for bottle to store these in unicode?
Thanks.
request.query['q'] and forms.get('q') return the raw byte value submitted by the web browser. The value äöü, submitted by a browser as UTF-8 encoded bytes, is 'xc3xa4xc3xb6xc3xbc'.
If you print that byte string, and the place you’re printing it to interprets it as ISO-8859-1, or the similar Windows code page 1252, you will get äöü. If you are debugging by printing to a Windows command prompt, or a file displayed by Notepad, that’s why.
If you use the alternative direct property access request.query.q or forms.q Bottle will give you Unicode strings instead, decoded from the byte version using UTF-8. It’s usually best to work with these Unicode strings wherever you can. (Though still you may have trouble printing them to console. The Windows command prompt is notoriously terrible at coping with non-ASCII characters, and as such is a bad place to be debugging Unicode issues.)
in this case, to convert it ,I did like this search_field.encode(“ISO-8859-1”).decode(“utf-8”)
There’s also the alternative of using def getunicode(self, name, default=None, encoding=None)
Instead of using
request.forms.get('some_form_field_name')
try
request.forms.getunicode('some_form_field_name')
That’ll handle non-latin characters.