Python dictionary : removing u' chars
Question:
How do I remove u chars from the following dictionary?
{u'name': u'A', u'primary_key': 1}
This data is coming from Mongo Database find() query
so that it looks like
{'name': 'A', 'primary_key': 1}
Answers:
The u characters that you are seeing simply mean that they are unicode strings.
If you do not want them to be unicode, you can encode them as something else, such as ASCII.
>>> s = u'hi!'
>>> s
u'hi'
>>> s2 = s.encode('ascii')
>>> s2
'hi'
As sven mentions in his comment, the u is an indication of the types represented in mongodb (actually it’s because json is defined to use unicode).
This fact should be totally transparent to you, in fact you can use str and unicode values interchangeably in the dicts.
>>> 'foo' in {u'foo': 5}
True
>>> {u'foo': 5}['foo']
5
>>>
You cannot simply remove the u from the strings, as it symbolizes, that the strings are in Unicode.
One solution is to use the encode function:
old_strings = {u'name':u'A', u'primary_key':1}
newstrings = {}
for k in old_strings.keys():
newtsrings[k] = old_strings[k].encode('ascii','ignore')
This would just ignore non ascii characters.
You need to let psycopg2 encode your strings, not try to insert Python-syntax strings into your queries raw — you are putting yourself in danger of a SQL injection problem if some of the strings contain characters that SQL will interpret as ending the string. You should pass parameters to psycopg2 like this:
cursor.execute('INSERT INTO person (name, town) VALUES (%s %s)', (name, town))
Because psycopg2 knows SQL syntax very, very well, it will leave off the u characters as it gets your name and town strings and quotes and escapes them in exactly the way that this SQL statement needs.
Some databases such as Sqlite3 let you define converter and adapter functions so you can retrieve text as str rather than unicode. Unfortunately, MongoDB doesn’t provide this option for any of the commonly needed types such as str, decimal or datetime:
- http://api.mongodb.org/python/current/tutorial.html#a-note-on-unicode-strings
- http://api.mongodb.org/python/current/faq.html#how-can-i-store-decimal-decimal-instances
- http://api.mongodb.org/python/current/faq.html#how-can-i-save-a-datetime-date-instance
Having eliminated Mongo options, that leaves writing Python code to do the conversion after the data is retrieved. You could write a recursive function that traverses the result to convert each field.
As a quick-and-dirty alternative, here is a little hack that may be of use:
>>> import json, ast
>>> r = {u'name': u'A', u'primary_key': 1}
>>> ast.literal_eval(json.dumps(r))
{'name': 'A', 'primary_key': 1}
If you simply want to convert the dict to json data string you can do:
>>> from bson.json_util import dumps
>>> data = {u'name': u'A', u'primary_key': 1}
>>> dumps(data)
'{"name": "A", "primary_key": 1}'
Similar to what has already been posted, but using the json library.
>>> import json
>>> udata = {u'name': u'A', u'primary_key': 1}
>>> json.dumps(udata)
'{"name": "A", "primary_key": 1}'
How do I remove u chars from the following dictionary?
{u'name': u'A', u'primary_key': 1}
This data is coming from Mongo Database find() query
so that it looks like
{'name': 'A', 'primary_key': 1}
The u characters that you are seeing simply mean that they are unicode strings.
If you do not want them to be unicode, you can encode them as something else, such as ASCII.
>>> s = u'hi!'
>>> s
u'hi'
>>> s2 = s.encode('ascii')
>>> s2
'hi'
As sven mentions in his comment, the u is an indication of the types represented in mongodb (actually it’s because json is defined to use unicode).
This fact should be totally transparent to you, in fact you can use str and unicode values interchangeably in the dicts.
>>> 'foo' in {u'foo': 5}
True
>>> {u'foo': 5}['foo']
5
>>>
You cannot simply remove the u from the strings, as it symbolizes, that the strings are in Unicode.
One solution is to use the encode function:
old_strings = {u'name':u'A', u'primary_key':1}
newstrings = {}
for k in old_strings.keys():
newtsrings[k] = old_strings[k].encode('ascii','ignore')
This would just ignore non ascii characters.
You need to let psycopg2 encode your strings, not try to insert Python-syntax strings into your queries raw — you are putting yourself in danger of a SQL injection problem if some of the strings contain characters that SQL will interpret as ending the string. You should pass parameters to psycopg2 like this:
cursor.execute('INSERT INTO person (name, town) VALUES (%s %s)', (name, town))
Because psycopg2 knows SQL syntax very, very well, it will leave off the u characters as it gets your name and town strings and quotes and escapes them in exactly the way that this SQL statement needs.
Some databases such as Sqlite3 let you define converter and adapter functions so you can retrieve text as str rather than unicode. Unfortunately, MongoDB doesn’t provide this option for any of the commonly needed types such as str, decimal or datetime:
- http://api.mongodb.org/python/current/tutorial.html#a-note-on-unicode-strings
- http://api.mongodb.org/python/current/faq.html#how-can-i-store-decimal-decimal-instances
- http://api.mongodb.org/python/current/faq.html#how-can-i-save-a-datetime-date-instance
Having eliminated Mongo options, that leaves writing Python code to do the conversion after the data is retrieved. You could write a recursive function that traverses the result to convert each field.
As a quick-and-dirty alternative, here is a little hack that may be of use:
>>> import json, ast
>>> r = {u'name': u'A', u'primary_key': 1}
>>> ast.literal_eval(json.dumps(r))
{'name': 'A', 'primary_key': 1}
If you simply want to convert the dict to json data string you can do:
>>> from bson.json_util import dumps
>>> data = {u'name': u'A', u'primary_key': 1}
>>> dumps(data)
'{"name": "A", "primary_key": 1}'
Similar to what has already been posted, but using the json library.
>>> import json
>>> udata = {u'name': u'A', u'primary_key': 1}
>>> json.dumps(udata)
'{"name": "A", "primary_key": 1}'