Pandas correlation matrix with value_counts column of strings
Question:
I want to create a correlation matrix from string columns value counts. So here I have Accident severity and Time.
I am trying to show the correlation between the Time of day and the severity of an accident
Part of the Pandas dataframe (df) :
+-----------------------+-------------------+------------------+
| Accident_Index | Time | Accident_Severity|
+-----------------------+-------------------+------------------+
| 200501BS00001 | Morning | Serious |
| 200501BS00002 | Night | Slight |
| 200501BS00003 | Evening | Slight |
| 200501BS00004 | Afternoon | Fatal |
+-----------------------+-------------------+------------------+
My expected Output is something like this:
+---------+-----------+-------+---------+-----------+
| | Morning | Night | Evening | Afternoon |
+---------+-----------+-------+---------+-----------+
| Serious | 0.9 | 0.3 | 0.3 | 0.3 |
| Slight | 0.8 | 1 | 0.2 | 0.5 |
| Fatal | 0.4 | 0.3 | 1 | 0.3 |
+---------+-----------+-------+---------+-----------+
I have tried this sort of thing:
s_corr = df.Accident_Severity.str.get_dummies(' ').corrwith(df.Time.value_counts() / df.Time.value_counts().max())
print(s_corr)
Output:
- Fatal NaN
- Serious NaN
- Slight NaN
and this:
corrs = df.pivot('Time','Accident_Severity').T.corr().stack()
corrs.index.names = 'Time', 'Accident_Severity'
corrs.reset_index()
print(corrs)
Output:
- ValueError: Index contains duplicate entries, cannot reshape
and this:
corrs = df.reset_index().pivot_table('Time','Accident_Severity').T.corr().stack()
print(corrs)
Output:
- pandas.core.base.DataError: No numeric types to aggregate
and this:
acc = df['Accident_Severity'].value_counts()
ti = df['Time'].value_counts()
print(acc.corr(ti))
Output:
- nan
Answers:
I don’t really understand the expected output here. But given some data:
import random
severity_choices = ['Slight', 'Serious', 'Fatal']
time_choices = ['Morning', 'Afternoon', 'Evening', 'Night']
df = pd.DataFrame({
'Severity': [random.choice(severity_choices) for i in range(0, 1000)],
'Time': [random.choice(time_choices) for i in range(0, 1000)]
})
We can calculate the proportion of each Severity
using pd.crosstab
and normalize
set to index
.
>> pd.crosstab(df['Severity'], df['Time'], normalize='index')
Time Afternoon Evening Morning Night
Severity
Fatal 0.246106 0.249221 0.224299 0.280374
Serious 0.253125 0.234375 0.253125 0.259375
Slight 0.233983 0.233983 0.267409 0.264624
I want to create a correlation matrix from string columns value counts. So here I have Accident severity and Time.
I am trying to show the correlation between the Time of day and the severity of an accident
Part of the Pandas dataframe (df) :
+-----------------------+-------------------+------------------+
| Accident_Index | Time | Accident_Severity|
+-----------------------+-------------------+------------------+
| 200501BS00001 | Morning | Serious |
| 200501BS00002 | Night | Slight |
| 200501BS00003 | Evening | Slight |
| 200501BS00004 | Afternoon | Fatal |
+-----------------------+-------------------+------------------+
My expected Output is something like this:
+---------+-----------+-------+---------+-----------+
| | Morning | Night | Evening | Afternoon |
+---------+-----------+-------+---------+-----------+
| Serious | 0.9 | 0.3 | 0.3 | 0.3 |
| Slight | 0.8 | 1 | 0.2 | 0.5 |
| Fatal | 0.4 | 0.3 | 1 | 0.3 |
+---------+-----------+-------+---------+-----------+
I have tried this sort of thing:
s_corr = df.Accident_Severity.str.get_dummies(' ').corrwith(df.Time.value_counts() / df.Time.value_counts().max())
print(s_corr)
Output:
- Fatal NaN
- Serious NaN
- Slight NaN
and this:
corrs = df.pivot('Time','Accident_Severity').T.corr().stack()
corrs.index.names = 'Time', 'Accident_Severity'
corrs.reset_index()
print(corrs)
Output:
- ValueError: Index contains duplicate entries, cannot reshape
and this:
corrs = df.reset_index().pivot_table('Time','Accident_Severity').T.corr().stack()
print(corrs)
Output:
- pandas.core.base.DataError: No numeric types to aggregate
and this:
acc = df['Accident_Severity'].value_counts()
ti = df['Time'].value_counts()
print(acc.corr(ti))
Output:
- nan
I don’t really understand the expected output here. But given some data:
import random
severity_choices = ['Slight', 'Serious', 'Fatal']
time_choices = ['Morning', 'Afternoon', 'Evening', 'Night']
df = pd.DataFrame({
'Severity': [random.choice(severity_choices) for i in range(0, 1000)],
'Time': [random.choice(time_choices) for i in range(0, 1000)]
})
We can calculate the proportion of each Severity
using pd.crosstab
and normalize
set to index
.
>> pd.crosstab(df['Severity'], df['Time'], normalize='index')
Time Afternoon Evening Morning Night
Severity
Fatal 0.246106 0.249221 0.224299 0.280374
Serious 0.253125 0.234375 0.253125 0.259375
Slight 0.233983 0.233983 0.267409 0.264624