ValueError: DataFrame constructor not properly called! when coverting dictionaries within list to pandas dataframe

Question:

I want to convert a list of dictionaries to a pandas dataframe, however, I got ValueError: DataFrame constructor not properly called!

Below is an example and how I got the data:

import requests
import pandas as pd

# Send an HTTP GET request to the URL
response = requests.get(url)

# Decode the JSON data into a dictionary
scrapped_data = response.text

scraped_data = [{"id":123456,"date":"12-12-2022","value":37},
{"id":123456,"date":"13-12-2022","value":38}]

I want to convert it to a dataframe format like the following:

id date value
123456 12-12-2022 37
123456 13-12-2022 38

I tried the following methods:

df = pd.DataFrame(scrapped_data)

df = pd.DataFrame_from_dict(scrapped_data)

df = pd.DataFrame(scrapped_data, orient='columns')

all got the same value errors.

I also tried:

df = pd.json_normalize(scrapped_data)

but got NotImplementedError

The type for scrapped_data is string format

Thanks for your help, let me know if you have any questions

Asked By: user20350073

||
Source

Answers:

As you said, scrapped_data is a string then you need to convert it into a dictionary (with the method loads from the json library for example).

If scrapped_data = ‘[{"id":"123456","date":"12-12-2022","value":"37"}, {"id":"123456","date":"13-12-2022","value":"38"}]’,
then you can just do df = pd.DataFrame(scrapped_data).

Answered By: Jonathan Dauwe

One reason for receiving this error from pandas is providing str as data. I think your data come as str, If it is the case then Try this:

import json
import pandas as pd

orignal_data='[{"id":"123456","date":"12-12-2022","value":"37"}, {"id":"123456","date":"13-12-2022","value":"38"}]'
scraped_data = json.loads(orignal_data)
df = pd.DataFrame(data=scraped_data)
df
Answered By: Niyaz