More efficient way of converting a list of a single dictionary in pandas rows to column titles and columnar data

Question:

I have a pandas dataframe column that consists of a list of a single dictionary in each row (created from a machine learning algorithm run on other data in the dataframe). My goal is to convert the dictionary keys into the title of new dataframe columns and the row values will be the values of the dictionaries. I’ve searched through a number of different articles on stackoverflow, but have not found a definitive answer.

The list of a single dictionaries looks like this in each row of an existing column:

[{'label': 'POSITIVE', 'score': 0.9969509840011597}] 

I know I can break the task into multiple steps, such as a function that will convert the list of dictionaries into a single dictionary, and then convert those keys into the new column titles – like so:

# function to break the list of single dictionary into a dictionary
def make_dict(list):
    d1={}
    for i in my_list:
        d1.update(i)
    return d1

analyze['sentiment'] = analyze['sentiment'].swifter.apply(make_dict)

The output in the ‘sentiment’ column in the dataframe is what you would expect:

{'label': 'POSITIVE', 'score': 0.9969509840011597} 

From here I would just assign the keys to column names and values to the columnar data. However, is there a more efficient way with existing pandas method to extract the keys into columns and values into rows without the intermediate step of using the function?

Asked By: Ja4H3ad

||

Answers:

If each list has only one dictionary, try:

analyze[['label','score']] = analyze['sentiment'].agg(pd.Series)[0].agg(pd.Series)

The first agg(pd.Series) will give rows only with dictionaries, and since you have only one dictionary, use agg again on the first exploded column.
Output:

       label    score
0   POSITIVE    0.996951
1   POSITIVE    0.996951
2   POSITIVE    0.996951
3   POSITIVE    0.996951
4   POSITIVE    0.996951
5   POSITIVE    0.996951
Answered By: Nuri Taş
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.