ValueError when using pandas.read_json
Question:
I made a 250MB json file that should look like this:
[ {"A":"uniquevalue0", "B":[1,2,3]},
{"A":"uniquevalue1", "B":[1]},
{"A":"uniquevalue2", "B":[1,2,3,4]} ]
where the “B” value can be variable len >= 1. This says I have valid JSON.
I call
df = pandas.read_json('ut1.json', orient = 'records', dtype={"A":str, "B":list})
Here is the documentation. When reading into a pandas dataframe, I get the following traceback:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../pandas/io/json.py", line 198, in read_json
date_unit).parse()
File "/.../pandas/io/json.py", line 266, in parse
self._parse_no_numpy()
File "/.../pandas/io/json.py", line 496, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Unexpected character found when decoding 'true'
Can’t think of what is going wrong. The python file that is throwing the error is not that helpful.
Answers:
I had the same error message, and I solved it by using an absolute path.
import os
basePath = os.path.dirname(os.path.abspath(__file__))
df = pandas.read_json(basePath + '/ut1.json', orient = 'records', dtype={"A":str, "B":list})
That worked for me!
I was getting the “Value Error: Expected object or value” today while calling pandas.read_json(‘my_file.json’). I have ran this code with the same file earlier, so was very worried to see it is not working today. Later, I found for some reason, the json file was not there in the same dir. Then, I downloaded the file from git by right clicking the file link. That was a bad idea :(. I guess the json file was not encoded properly, so I kept getting the same error even when the json file was there in the same dir. At the end, I deleted the json file, cloned the original git repo to get the json file and put it in the same dir again. Then, this pandas.read_json did work. So, first of all please make sure the json file exists in the proper dir and then make sure, the file is not corrupted.
After tried @learn2day’s answer, I still cannot get a good result from there, but I do try the following code and everything works for me. (PS: I’m opening a JSON file where Chinese characters were UTF-8 characters appeared – Chinese characters)
pandas.read_json(open("ut1.json", "r", encoding="utf8"))
The encoding="utf8" is the key part of this code.
In my case, the path was wrong.
Make sure you check your current working directory, by placing this just before the pandas.read_json:
import os
print(os.getcwd())
Posting this because the answers above did not match my problem with this error: It just occurred for me when reading a very long string document I thought would be valid json format but contained nan as exported from python to string while it should be "null" for valid json. If the document was not created using a json package it might have faulty values indicated by this error message.
I had the same issue and then realized that the issue was while copying and pasting text from the browser to my file. It introduced carriage returns so that each line, for a given key was broken up into multiple rows. THAT was the issue. hope this helps someone!
I had the same error when running the code on linux. I realised the file name given to read_csv had .JSON instead of .json. Changing it to lower case worked for me.
For me, the issue was that the file has a UTF-8 BOM character at the beginning. Using the following encoding solved the problem:
df = pd.read_json(r'C:tempfoo.jsonl', lines=True, encoding='utf-8-sig')
try this
df = pd.read_json('file.jsonl', lines=True) #if its a jsonl file
-
First make sure your file is there, now ofcourse it is but what worked for me is deleting the file and recopying to the location and be cautious while renaming it and then read the file as :
df_test = pd.read_json(‘test_file.json’,lines=True)
Note that I have had encountered all different errors talked in this particular thread and finally this solution work which I have explained i.e. recopying the file without renaming it and then reading the file ( also making sure my code and file shares the same dir if I am not providing absolute path while reading the file).
I made a 250MB json file that should look like this:
[ {"A":"uniquevalue0", "B":[1,2,3]},
{"A":"uniquevalue1", "B":[1]},
{"A":"uniquevalue2", "B":[1,2,3,4]} ]
where the “B” value can be variable len >= 1. This says I have valid JSON.
I call
df = pandas.read_json('ut1.json', orient = 'records', dtype={"A":str, "B":list})
Here is the documentation. When reading into a pandas dataframe, I get the following traceback:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../pandas/io/json.py", line 198, in read_json
date_unit).parse()
File "/.../pandas/io/json.py", line 266, in parse
self._parse_no_numpy()
File "/.../pandas/io/json.py", line 496, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Unexpected character found when decoding 'true'
Can’t think of what is going wrong. The python file that is throwing the error is not that helpful.
I had the same error message, and I solved it by using an absolute path.
import os
basePath = os.path.dirname(os.path.abspath(__file__))
df = pandas.read_json(basePath + '/ut1.json', orient = 'records', dtype={"A":str, "B":list})
That worked for me!
I was getting the “Value Error: Expected object or value” today while calling pandas.read_json(‘my_file.json’). I have ran this code with the same file earlier, so was very worried to see it is not working today. Later, I found for some reason, the json file was not there in the same dir. Then, I downloaded the file from git by right clicking the file link. That was a bad idea :(. I guess the json file was not encoded properly, so I kept getting the same error even when the json file was there in the same dir. At the end, I deleted the json file, cloned the original git repo to get the json file and put it in the same dir again. Then, this pandas.read_json did work. So, first of all please make sure the json file exists in the proper dir and then make sure, the file is not corrupted.
After tried @learn2day’s answer, I still cannot get a good result from there, but I do try the following code and everything works for me. (PS: I’m opening a JSON file where Chinese characters were UTF-8 characters appeared – Chinese characters)
pandas.read_json(open("ut1.json", "r", encoding="utf8"))
The encoding="utf8" is the key part of this code.
In my case, the path was wrong.
Make sure you check your current working directory, by placing this just before the pandas.read_json:
import os
print(os.getcwd())
Posting this because the answers above did not match my problem with this error: It just occurred for me when reading a very long string document I thought would be valid json format but contained nan as exported from python to string while it should be "null" for valid json. If the document was not created using a json package it might have faulty values indicated by this error message.
I had the same issue and then realized that the issue was while copying and pasting text from the browser to my file. It introduced carriage returns so that each line, for a given key was broken up into multiple rows. THAT was the issue. hope this helps someone!
I had the same error when running the code on linux. I realised the file name given to read_csv had .JSON instead of .json. Changing it to lower case worked for me.
For me, the issue was that the file has a UTF-8 BOM character at the beginning. Using the following encoding solved the problem:
df = pd.read_json(r'C:tempfoo.jsonl', lines=True, encoding='utf-8-sig')
try this
df = pd.read_json('file.jsonl', lines=True) #if its a jsonl file
-
First make sure your file is there, now ofcourse it is but what worked for me is deleting the file and recopying to the location and be cautious while renaming it and then read the file as :
df_test = pd.read_json(‘test_file.json’,lines=True)
Note that I have had encountered all different errors talked in this particular thread and finally this solution work which I have explained i.e. recopying the file without renaming it and then reading the file ( also making sure my code and file shares the same dir if I am not providing absolute path while reading the file).