How do I automatically fix an invalid JSON string?
Question:
From the 2gis API I got the following JSON string.
{
"api_version": "1.3",
"response_code": "200",
"id": "3237490513229753",
"lon": "38.969916127827",
"lat": "45.069889625267",
"page_url": null,
"name": "ATB",
"firm_group": {
"id": "3237499103085728",
"count": "1"
},
"city_name": "Krasnodar",
"city_id": "3237585002430511",
"address": "Turgeneva, 172/1",
"create_time": "2008-07-22 10:02:04 07",
"modification_time": "2013-08-09 20:04:36 07",
"see_also": [
{
"id": "3237491513434577",
"lon": 38.973110606808,
"lat": 45.029031222211,
"name": "Advance",
"hash": "5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e",
"ads": {
"sponsored_article": {
"title": "Center "ADVANCE"",
"text": "Business.English."
},
"warning": null
}
}
]
}
But Python doesn’t recognize it:
json.loads(firm_str)
Expecting , delimiter: line 1 column 3646 (char 3645)
It looks like a problem with quotes in:
“title”: “Center “ADVANCE””
How can I fix it automatically in Python?
Answers:
If this is exactly what the API is returning then there is a problem with their API. This is invalid JSON. Especially around this area:
"ads": {
"sponsored_article": {
"title": "Образовательный центр "ADVANCE"", <-- here
"text": "Бизнес.Риторика.Английский язык.Подготовка к школе.Подготовка к ЕГЭ."
},
"warning": null
}
The double quotes around ADVANCE are not escaped. You can tell by using something like http://jsonlint.com/ to validate it.
This is a problem with the " not being escaped, the data is bad at the source if this is what you are getting. They need to fix it.
Parse error on line 4:
...азовательный центр "ADVANCE"",
-----------------------^
Expecting '}', ':', ',', ']'
This fixes the problem:
"title": "Образовательный центр "ADVANCE"",
You need to escape double quotes in JSON strings, as follows:
"title": "Образовательный центр "ADVANCE"",
To fix it programmatically, the simplest way would be to modify your JSON parser so you have some context for the error, then attempt to repair it.
The answer by @Michael gave me an idea… not a very pretty idea, but it seems to work, at least on your example: Try to parse the JSON string, and if it fails, look for the character where it failed in the exception string1 and replace that character.
while True:
try:
result = json.loads(s) # try to parse...
break # parsing worked -> exit loop
except Exception as e:
# "Expecting , delimiter: line 34 column 54 (char 1158)"
# position of unexpected character after '"'
unexp = int(re.findall(r'(char (d+))', str(e))[0])
# position of unescaped '"' before that
unesc = s.rfind(r'"', 0, unexp)
s = s[:unesc] + r'"' + s[unesc+1:]
# position of correspondig closing '"' (+2 for inserted '')
closg = s.find(r'"', unesc + 2)
s = s[:closg] + r'"' + s[closg+1:]
print result
You may want to add some additional checks to prevent this from ending in an infinite loop (e.g., at max as many repetitions as there are characters in the string). Also, this will still not work if an incorrect " is actually followed by a comma, as pointed out by @gnibbler.
Update: This seems to work pretty well now (though still not perfect), even if the unescaped " is followed by a comma, or closing bracket, as in this case it will likely get a complaint about a syntax error after that (expected property name, etc.) and trace back to the last ". It also automatically escapes the corresponding closing " (assuming there is one).
1) The exception’s str is "Expecting , delimiter: line XXX column YYY (char ZZZ)", where ZZZ is the position in the string where the error occurred. Note, though, that this message may depend on the version of Python, the json module, the OS, or the locale, and thus this solution may have to be adapted accordingly.
The only real and definitive solution is to ask 2gis to fix their API.
In the meantime it is possible to fix the badly encoded JSON escaping double quotes inside strings. If every key-value pair is followed by a newline (as it seems to be from the posted data) the following function will do the job:
def fixjson(badjson):
s = badjson
idx = 0
while True:
try:
start = s.index( '": "', idx) + 4
end1 = s.index( '",n',idx)
end2 = s.index( '"n', idx)
if end1 < end2:
end = end1
else:
end = end2
content = s[start:end]
content = content.replace('"', '\"')
s = s[:start] + content + s[end:]
idx = start + len(content) + 6
except:
return s
Please, note that some assumtions made:
The function attemps to escape double quotes characters inside value string belonging to key-value pairs.
It is assumed that the text to be escaped begins after the sequence
": "
and ends before the sequence
",n
or
"n
Passing the posted JSON to the function results in this returned value
{
"api_version": "1.3",
"response_code": "200",
"id": "3237490513229753",
"lon": "38.969916127827",
"lat": "45.069889625267",
"page_url": null,
"name": "ATB",
"firm_group": {
"id": "3237499103085728",
"count": "1"
},
"city_name": "Krasnodar",
"city_id": "3237585002430511",
"address": "Turgeneva, 172/1",
"create_time": "2008-07-22 10:02:04 07",
"modification_time": "2013-08-09 20:04:36 07",
"see_also": [
{
"id": "3237491513434577",
"lon": 38.973110606808,
"lat": 45.029031222211,
"name": "Advance",
"hash": "5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e",
"ads": {
"sponsored_article": {
"title": "Center "ADVANCE"",
"text": "Business.English."
},
"warning": null
}
}
]
}
Keep in mind you can easily customize the function if your needs are not fully satisfied.
The above Idea is good but I had problem with that. My json Sting consisted only one additional double quote in it.
So, I made a fix to the above given code.
The jsonStr was
{
"api_version": "1.3",
"response_code": "200",
"id": "3237490513229753",
"lon": "38.969916127827",
"lat": "45.069889625267",
"page_url": null,
"name": "ATB",
"firm_group": {
"id": "3237499103085728",
"count": "1"
},
"city_name": "Krasnodar",
"city_id": "3237585002430511",
"address": "Turgeneva, 172/1",
"create_time": "2008-07-22 10:02:04 07",
"modification_time": "2013-08-09 20:04:36 07",
"see_also": [
{
"id": "3237491513434577",
"lon": 38.973110606808,
"lat": 45.029031222211,
"name": "Advance",
"hash": "5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e",
"ads": {
"sponsored_article": {
"title": "Center "ADVANCE",
"text": "Business.English."
},
"warning": null
}
}
]
}
The fix is as follows:
import json, re
def fixJSON(jsonStr):
# Substitue all the backslash from JSON string.
jsonStr = re.sub(r'\', '', jsonStr)
try:
return json.loads(jsonStr)
except ValueError:
while True:
# Search json string specifically for '"'
b = re.search(r'[w|"]s?(")s?[w|"]', jsonStr)
# If we don't find any the we come out of loop
if not b:
break
# Get the location of "
s, e = b.span(1)
c = jsonStr[s:e]
# Replace " with '
c = c.replace('"',"'")
jsonStr = jsonStr[:s] + c + jsonStr[e:]
return json.loads(jsonStr)
This code also works for JSON string mentioned in problem statement
OR you can also do this:
def fixJSON(jsonStr):
# First remove the " from where it is supposed to be.
jsonStr = re.sub(r'\', '', jsonStr)
jsonStr = re.sub(r'{"', '{`', jsonStr)
jsonStr = re.sub(r'"}', '`}', jsonStr)
jsonStr = re.sub(r'":"', '`:`', jsonStr)
jsonStr = re.sub(r'":', '`:', jsonStr)
jsonStr = re.sub(r'","', '`,`', jsonStr)
jsonStr = re.sub(r'",', '`,', jsonStr)
jsonStr = re.sub(r',"', ',`', jsonStr)
jsonStr = re.sub(r'["', '[`', jsonStr)
jsonStr = re.sub(r'"]', '`]', jsonStr)
# Remove all the unwanted " and replace with ' '
jsonStr = re.sub(r'"',' ', jsonStr)
# Put back all the " where it supposed to be.
jsonStr = re.sub(r'`','"', jsonStr)
return json.loads(jsonStr)
Within sources of https://fix-json.com I found a solution, but it’s very dirty and looks like a hack. Just adapt it to python
jsString.match(/:.*"(.*)"/gi).forEach(function(element){
var filtered = element.replace(/(^:s*"|"(,)?$)/gi, '').trim();
jsString = jsString.replace(filtered, filtered.replace(/(\*)"/gi, "\""));
});
it’s not perfect and ugly but it helps to me
def get_json_info(info_row: str, type) -> dict:
try:
info = json.loads(info_row)
except JSONDecodeError:
data = {
}
try:
for s in info_row.split('","'):
if not s:
continue
key, val = s.split(":", maxsplit=1)
key = key.strip().lstrip("{").strip('"')
val: str = re.sub('"', '\"', val.lstrip('"').strip('"}'))
data[key] = val
except ValueError:
print("ERROR:", info_row)
info = data
return info
I make a jsonfixer to solve a problem like this.
It’s Python Package (2.7) (a half-done command line tool)
just see https://github.com/half-pie/half-json
from half_json.core import JSONFixer
f = JSONFixer(max_try=100)
new_s = s.replace('n', '')
result = f.fix(new_s)
d = json.loads(result.line)
# {u'name': u'ATB', u'modification_time': u'2013-08-09 20:04:36 07', u'city_id': u'3237585002430511', u'see_also': [{u'hash': u'5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e', u'ads': {u'warning': None, u'sponsored_article': {u'ADVANCE': u', ', u'text': u'Business.English.', u'title': u'Center '}}, u'lon': 38.973110606808, u'lat': 45.029031222211, u'id': u'3237491513434577', u'name': u'Advance'}], u'response_code': u'200', u'lon': u'38.969916127827', u'firm_group': {u'count': u'1', u'id': u'3237499103085728'}, u'create_time': u'2008-07-22 10:02:04 07', u'city_name': u'Krasnodar', u'address': u'Turgeneva, 172/1', u'lat': u'45.069889625267', u'id': u'3237490513229753', u'api_version': u'1.3', u'page_url': None}
and test case in https://github.com/half-pie/half-json/blob/master/tests/test_cases.py#L76-L80
line = '{"title": "Center "ADVANCE"", "text": "Business.English."}'
ok, newline, _ = JSONFixer().fix(line)
self.assertTrue(ok)
self.assertEqual('{"title": "Center ","ADVANCE":", ","text": "Business.English."}', newline)
Fix #1
If you fetched it from some website, please make sure you are using the same string. In my case, I was doing .replace('\"','"') . Because of this, the data was not the json anymore. If you also did something. like that, please fix that.
Fix #2
Try adding some character in all the places insted of the key name. It will be fine.
From the 2gis API I got the following JSON string.
{
"api_version": "1.3",
"response_code": "200",
"id": "3237490513229753",
"lon": "38.969916127827",
"lat": "45.069889625267",
"page_url": null,
"name": "ATB",
"firm_group": {
"id": "3237499103085728",
"count": "1"
},
"city_name": "Krasnodar",
"city_id": "3237585002430511",
"address": "Turgeneva, 172/1",
"create_time": "2008-07-22 10:02:04 07",
"modification_time": "2013-08-09 20:04:36 07",
"see_also": [
{
"id": "3237491513434577",
"lon": 38.973110606808,
"lat": 45.029031222211,
"name": "Advance",
"hash": "5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e",
"ads": {
"sponsored_article": {
"title": "Center "ADVANCE"",
"text": "Business.English."
},
"warning": null
}
}
]
}
But Python doesn’t recognize it:
json.loads(firm_str)
Expecting , delimiter: line 1 column 3646 (char 3645)
It looks like a problem with quotes in:
“title”: “Center “ADVANCE””
How can I fix it automatically in Python?
If this is exactly what the API is returning then there is a problem with their API. This is invalid JSON. Especially around this area:
"ads": {
"sponsored_article": {
"title": "Образовательный центр "ADVANCE"", <-- here
"text": "Бизнес.Риторика.Английский язык.Подготовка к школе.Подготовка к ЕГЭ."
},
"warning": null
}
The double quotes around ADVANCE are not escaped. You can tell by using something like http://jsonlint.com/ to validate it.
This is a problem with the " not being escaped, the data is bad at the source if this is what you are getting. They need to fix it.
Parse error on line 4:
...азовательный центр "ADVANCE"",
-----------------------^
Expecting '}', ':', ',', ']'
This fixes the problem:
"title": "Образовательный центр "ADVANCE"",
You need to escape double quotes in JSON strings, as follows:
"title": "Образовательный центр "ADVANCE"",
To fix it programmatically, the simplest way would be to modify your JSON parser so you have some context for the error, then attempt to repair it.
The answer by @Michael gave me an idea… not a very pretty idea, but it seems to work, at least on your example: Try to parse the JSON string, and if it fails, look for the character where it failed in the exception string1 and replace that character.
while True:
try:
result = json.loads(s) # try to parse...
break # parsing worked -> exit loop
except Exception as e:
# "Expecting , delimiter: line 34 column 54 (char 1158)"
# position of unexpected character after '"'
unexp = int(re.findall(r'(char (d+))', str(e))[0])
# position of unescaped '"' before that
unesc = s.rfind(r'"', 0, unexp)
s = s[:unesc] + r'"' + s[unesc+1:]
# position of correspondig closing '"' (+2 for inserted '')
closg = s.find(r'"', unesc + 2)
s = s[:closg] + r'"' + s[closg+1:]
print result
You may want to add some additional checks to prevent this from ending in an infinite loop (e.g., at max as many repetitions as there are characters in the string). Also, this will still not work if an incorrect " is actually followed by a comma, as pointed out by @gnibbler.
Update: This seems to work pretty well now (though still not perfect), even if the unescaped " is followed by a comma, or closing bracket, as in this case it will likely get a complaint about a syntax error after that (expected property name, etc.) and trace back to the last ". It also automatically escapes the corresponding closing " (assuming there is one).
1) The exception’s str is "Expecting , delimiter: line XXX column YYY (char ZZZ)", where ZZZ is the position in the string where the error occurred. Note, though, that this message may depend on the version of Python, the json module, the OS, or the locale, and thus this solution may have to be adapted accordingly.
The only real and definitive solution is to ask 2gis to fix their API.
In the meantime it is possible to fix the badly encoded JSON escaping double quotes inside strings. If every key-value pair is followed by a newline (as it seems to be from the posted data) the following function will do the job:
def fixjson(badjson):
s = badjson
idx = 0
while True:
try:
start = s.index( '": "', idx) + 4
end1 = s.index( '",n',idx)
end2 = s.index( '"n', idx)
if end1 < end2:
end = end1
else:
end = end2
content = s[start:end]
content = content.replace('"', '\"')
s = s[:start] + content + s[end:]
idx = start + len(content) + 6
except:
return s
Please, note that some assumtions made:
The function attemps to escape double quotes characters inside value string belonging to key-value pairs.
It is assumed that the text to be escaped begins after the sequence
": "
and ends before the sequence
",n
or
"n
Passing the posted JSON to the function results in this returned value
{
"api_version": "1.3",
"response_code": "200",
"id": "3237490513229753",
"lon": "38.969916127827",
"lat": "45.069889625267",
"page_url": null,
"name": "ATB",
"firm_group": {
"id": "3237499103085728",
"count": "1"
},
"city_name": "Krasnodar",
"city_id": "3237585002430511",
"address": "Turgeneva, 172/1",
"create_time": "2008-07-22 10:02:04 07",
"modification_time": "2013-08-09 20:04:36 07",
"see_also": [
{
"id": "3237491513434577",
"lon": 38.973110606808,
"lat": 45.029031222211,
"name": "Advance",
"hash": "5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e",
"ads": {
"sponsored_article": {
"title": "Center "ADVANCE"",
"text": "Business.English."
},
"warning": null
}
}
]
}
Keep in mind you can easily customize the function if your needs are not fully satisfied.
The above Idea is good but I had problem with that. My json Sting consisted only one additional double quote in it.
So, I made a fix to the above given code.
The jsonStr was
{
"api_version": "1.3",
"response_code": "200",
"id": "3237490513229753",
"lon": "38.969916127827",
"lat": "45.069889625267",
"page_url": null,
"name": "ATB",
"firm_group": {
"id": "3237499103085728",
"count": "1"
},
"city_name": "Krasnodar",
"city_id": "3237585002430511",
"address": "Turgeneva, 172/1",
"create_time": "2008-07-22 10:02:04 07",
"modification_time": "2013-08-09 20:04:36 07",
"see_also": [
{
"id": "3237491513434577",
"lon": 38.973110606808,
"lat": 45.029031222211,
"name": "Advance",
"hash": "5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e",
"ads": {
"sponsored_article": {
"title": "Center "ADVANCE",
"text": "Business.English."
},
"warning": null
}
}
]
}
The fix is as follows:
import json, re
def fixJSON(jsonStr):
# Substitue all the backslash from JSON string.
jsonStr = re.sub(r'\', '', jsonStr)
try:
return json.loads(jsonStr)
except ValueError:
while True:
# Search json string specifically for '"'
b = re.search(r'[w|"]s?(")s?[w|"]', jsonStr)
# If we don't find any the we come out of loop
if not b:
break
# Get the location of "
s, e = b.span(1)
c = jsonStr[s:e]
# Replace " with '
c = c.replace('"',"'")
jsonStr = jsonStr[:s] + c + jsonStr[e:]
return json.loads(jsonStr)
This code also works for JSON string mentioned in problem statement
OR you can also do this:
def fixJSON(jsonStr):
# First remove the " from where it is supposed to be.
jsonStr = re.sub(r'\', '', jsonStr)
jsonStr = re.sub(r'{"', '{`', jsonStr)
jsonStr = re.sub(r'"}', '`}', jsonStr)
jsonStr = re.sub(r'":"', '`:`', jsonStr)
jsonStr = re.sub(r'":', '`:', jsonStr)
jsonStr = re.sub(r'","', '`,`', jsonStr)
jsonStr = re.sub(r'",', '`,', jsonStr)
jsonStr = re.sub(r',"', ',`', jsonStr)
jsonStr = re.sub(r'["', '[`', jsonStr)
jsonStr = re.sub(r'"]', '`]', jsonStr)
# Remove all the unwanted " and replace with ' '
jsonStr = re.sub(r'"',' ', jsonStr)
# Put back all the " where it supposed to be.
jsonStr = re.sub(r'`','"', jsonStr)
return json.loads(jsonStr)
Within sources of https://fix-json.com I found a solution, but it’s very dirty and looks like a hack. Just adapt it to python
jsString.match(/:.*"(.*)"/gi).forEach(function(element){
var filtered = element.replace(/(^:s*"|"(,)?$)/gi, '').trim();
jsString = jsString.replace(filtered, filtered.replace(/(\*)"/gi, "\""));
});
it’s not perfect and ugly but it helps to me
def get_json_info(info_row: str, type) -> dict:
try:
info = json.loads(info_row)
except JSONDecodeError:
data = {
}
try:
for s in info_row.split('","'):
if not s:
continue
key, val = s.split(":", maxsplit=1)
key = key.strip().lstrip("{").strip('"')
val: str = re.sub('"', '\"', val.lstrip('"').strip('"}'))
data[key] = val
except ValueError:
print("ERROR:", info_row)
info = data
return info
I make a jsonfixer to solve a problem like this.
It’s Python Package (2.7) (a half-done command line tool)
just see https://github.com/half-pie/half-json
from half_json.core import JSONFixer
f = JSONFixer(max_try=100)
new_s = s.replace('n', '')
result = f.fix(new_s)
d = json.loads(result.line)
# {u'name': u'ATB', u'modification_time': u'2013-08-09 20:04:36 07', u'city_id': u'3237585002430511', u'see_also': [{u'hash': u'5698hn745A8IJ1H86177uvgn94521J3464he26763737242Cf6e654G62J0I7878e', u'ads': {u'warning': None, u'sponsored_article': {u'ADVANCE': u', ', u'text': u'Business.English.', u'title': u'Center '}}, u'lon': 38.973110606808, u'lat': 45.029031222211, u'id': u'3237491513434577', u'name': u'Advance'}], u'response_code': u'200', u'lon': u'38.969916127827', u'firm_group': {u'count': u'1', u'id': u'3237499103085728'}, u'create_time': u'2008-07-22 10:02:04 07', u'city_name': u'Krasnodar', u'address': u'Turgeneva, 172/1', u'lat': u'45.069889625267', u'id': u'3237490513229753', u'api_version': u'1.3', u'page_url': None}
and test case in https://github.com/half-pie/half-json/blob/master/tests/test_cases.py#L76-L80
line = '{"title": "Center "ADVANCE"", "text": "Business.English."}'
ok, newline, _ = JSONFixer().fix(line)
self.assertTrue(ok)
self.assertEqual('{"title": "Center ","ADVANCE":", ","text": "Business.English."}', newline)
Fix #1
If you fetched it from some website, please make sure you are using the same string. In my case, I was doing .replace('\"','"') . Because of this, the data was not the json anymore. If you also did something. like that, please fix that.
Fix #2
Try adding some character in all the places insted of the key name. It will be fine.