Pandas DataFrame from a JSON object

Question

I’m trying to create a DataFrame from my JSON output which looks like as shown below.

{  
   "tags":[  
      {  
     "stats":{  
        "rawCount":9
     },
     "name":"Temperature1",
     "results":[  
        {  
           "attributes":{  
              "Location":[  
                 "3rd Floor"
              ],
              "Sensor-Serial-Number":[  
                 "PT100"
              ]
           },
           "values":[  
              [  
                 1460958592800,
                 24.2,
                 3
              ],
              [  
                 1460958602800,
                 24.1,
                 1
              ],
              [  
                 1460958612800,
                 23.9,
                 1
              ],
              [  
                 1460958622800,
                 24.2,
                 1
              ],
              [  
                 1460958632800,
                 24.5,
                 1
              ],
              [  
                 1460958642800,
                 24.9,
                 1
              ],
              [  
                 1460958652800,
                 24.6,
                 1
              ],
              [  
                 1460958662800,
                 24.7,
                 1
              ],
              [  
                 1460958672800,
                 24.7,
                 1
              ]
           ],
           "groups":[  
              {  
                 "type":"number",
                 "name":"type"
              }
           ]
        }
     ]
      }
   ]
}

I only require values, which I need to convert into a DataFrame as shown in the below image.

Timeseries data

Asked By: user6255719

||

Source

Answer 1

try this to pull out only a list of values from your json

import json
import ast
import pandas as pd
mystr = """
{'tags': [{'name': 'Temperature1',
  'results': [{'attributes': {'Location': ['3rd Floor'],
  'Sensor-Serial-Number': ['PT100']},
  'groups': [{'name': 'type', 'type': 'number'}],
  'values': [[1460958592800, 24.2, 3],
  [1460958602800, 24.1, 1],
  [1460958612800, 23.9, 1],
  [1460958622800, 24.2, 1],
  [1460958632800, 24.5, 1],
  [1460958642800, 24.9, 1],
  [1460958652800, 24.6, 1],
  [1460958662800, 24.7, 1],
  [1460958672800, 24.7, 1]]}],
 'stats': {'rawCount': 9}}]}
"""
val = ast.literal_eval(mystr)
val1 = json.loads(json.dumps(val))
val2 = val1['tags'][0]['results'][0]['values']
print pd.DataFrame(val2, columns=["time", "temperature", "quality"])

the result turns out to be

            time  temperature  quality
0  1460958592800         24.2        3
1  1460958602800         24.1        1
2  1460958612800         23.9        1
3  1460958622800         24.2        1
4  1460958632800         24.5        1
5  1460958642800         24.9        1
6  1460958652800         24.6        1
7  1460958662800         24.7        1
8  1460958672800         24.7        1

which is your table for dataset

Answered By: sameera sy

Answer 2

There’s a specialized pandas function pd.json_normalize() that converts json data into a flat table. Since the data to be converted into a dataframe is nested under multiple keys, we can pass the path to it as a list as the record_path= kwarg. The path to values is tags -> results -> values, so we pass it as a list.

# first load the json file
import json
with open(file_path, 'r') as f:
    data = json.load(f)

# convert `data` into a dataframe
df = pd.json_normalize(data, record_path=['tags', 'results', 'values']).set_axis(['time', 'temperature', 'quality'], axis=1)

Answered By: cottontail

Pandas DataFrame from a JSON object

Question:

Answers: