How to convert JSON data inside a pandas column into new columns


I have this short version of ADSB json data and would like to convert it into dataFrame columns as Icao, Alt, Lat, Long, Spd, Cou…..

After Alperen told me to do this

df = pd.read_json('2016-06-20-2359Z.json', lines=True), 

I can load it into a DataFrame. However, df.acList is

[{'Id': 10537990, 'Rcvr': 1, 'HasSig': False, ...
Name: acList, dtype: object

How can I get the Icao, Alt, Lat, Long, Spd, Cou data?

         "Cou":"United States",
         "Cou":"United States",
Asked By: user8149657



If you already have your data in acList column in a pandas DataFrame, simply do:

import pandas as pd[0])

Alt AltT    Bad CMsgs   CNum    Call    CallSus Cou EngMount    EngType ... Sqk TSecs   TT  Tisb    TrkH    Trt Type    VsiT    WTC Year
0   NaN 0   False   1   7503    NaN False   United States   1   3   ...     3   a   False   False   1   CRJ2    0   2   2001
1   8025.0  0   False   35445   NaN TEST1234    False   United States   0   0   ... 0262    75229   a   False   False   1   NaN 0   0   NaN

Since pandas 1.0 the imports should be:

import pandas as pd
Answered By: Sergey Bushmanov

@Sergey’s answer solved the issue for me but I was running into issues because the json in my data frame column was kept as a string and not as an object. I had to add the additional step of mapping the column:

import json
import pandas as pd
Answered By: ThinkBonobo

I can’t comment yet on ThinkBonobo’s answer but in case the JSON in the column isn’t exactly a dictionary you can keep doing .apply until it is. So in my case

import json
import pandas as pd

    .apply(lambda x: x[0]) # the inner JSON is list with the dictionary as the only item
Answered By: blockw

Since pandas 1.0, json_normalize is available in the top-level namespace.
Therefore use:

import pandas as pd
Answered By: Jonne Kleijer

In my case I had some missing values (None) then I created a more specific code that also drops the original column after creating the new ones:

for prefix in ['column1', 'column2']:
    df_temp = df[prefix].apply(lambda x: {} if pd.isna(x) else x)
    df_temp =
    df_temp = df_temp.add_prefix(prefix + '_')
    df.drop([prefix], axis=1, inplace=True)
    df = pd.concat([df, df_temp], axis = 1, sort=False)
Answered By: Felipe Augusto
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.