Pandas correlation matrix with value_counts column of strings

Question:

I want to create a correlation matrix from string columns value counts. So here I have Accident severity and Time.
I am trying to show the correlation between the Time of day and the severity of an accident

Part of the Pandas dataframe (df) :

+-----------------------+-------------------+------------------+
| Accident_Index        | Time              | Accident_Severity|
+-----------------------+-------------------+------------------+
| 200501BS00001         | Morning           | Serious          |
| 200501BS00002         | Night             | Slight           |
| 200501BS00003         | Evening           | Slight           |
| 200501BS00004         | Afternoon         | Fatal            |
+-----------------------+-------------------+------------------+

My expected Output is something like this:

+---------+-----------+-------+---------+-----------+
|         |   Morning | Night | Evening | Afternoon |
+---------+-----------+-------+---------+-----------+
| Serious |       0.9 |   0.3 |     0.3 |       0.3 |
| Slight  |       0.8 |     1 |     0.2 |       0.5 |
| Fatal   |       0.4 |   0.3 |       1 |       0.3 |
+---------+-----------+-------+---------+-----------+

I have tried this sort of thing:

s_corr = df.Accident_Severity.str.get_dummies(' ').corrwith(df.Time.value_counts() / df.Time.value_counts().max())
print(s_corr)

Output:

  • Fatal NaN
  • Serious NaN
  • Slight NaN

and this:

corrs = df.pivot('Time','Accident_Severity').T.corr().stack()
        corrs.index.names = 'Time', 'Accident_Severity'
        corrs.reset_index()
print(corrs)

Output:

  • ValueError: Index contains duplicate entries, cannot reshape

and this:

corrs = df.reset_index().pivot_table('Time','Accident_Severity').T.corr().stack()
print(corrs)

Output:

  • pandas.core.base.DataError: No numeric types to aggregate

and this:

acc = df['Accident_Severity'].value_counts()
ti = df['Time'].value_counts()
print(acc.corr(ti))

Output:

  • nan
Asked By: PyStraw45

||

Answers:

I don’t really understand the expected output here. But given some data:

import random

severity_choices = ['Slight', 'Serious', 'Fatal']
time_choices = ['Morning', 'Afternoon', 'Evening', 'Night']


df = pd.DataFrame({
   'Severity': [random.choice(severity_choices) for i in range(0, 1000)], 
   'Time': [random.choice(time_choices) for i in range(0, 1000)]
})

We can calculate the proportion of each Severity using pd.crosstab and normalize set to index.

>> pd.crosstab(df['Severity'], df['Time'], normalize='index')

Time        Afternoon   Evening     Morning     Night
Severity                
Fatal       0.246106    0.249221    0.224299    0.280374
Serious     0.253125    0.234375    0.253125    0.259375
Slight      0.233983    0.233983    0.267409    0.264624
Answered By: user3471881