Pandas 'describe' is not returning summary of all columns
Question:
I am running ‘describe()’ on a dataframe and getting summaries of only int columns (pandas 14.0).
The documentation says that for object columns frequency of most common value, and additional statistics would be returned. What could be wrong? (no error message is returned by the way)
Edit:
I think it’s how the function is set to behave on mixed column types in a dataframe. Although the documentation fails to mention it.
Example code:
df_test = pd.DataFrame({'$a':[1,2], '$b': [10,20]})
df_test.dtypes
df_test.describe()
df_test['$a'] = df_test['$a'].astype(str)
df_test.describe()
df_test['$a'].describe()
df_test['$b'].describe()
My ugly work around in the meanwhile:
def my_df_describe(df):
objects = []
numerics = []
for c in df:
if (df[c].dtype == object):
objects.append(c)
else:
numerics.append(c)
return df[numerics].describe(), df[objects].describe()
Answers:
‘describe()’ on a DataFrame only works for numeric types. If you think you have a numeric variable and it doesn’t show up in ‘decribe()’, change the type with:
df[['col1', 'col2']] = df[['col1', 'col2']].astype(float)
You could also create new columns for handling the numeric part of a mix type column, or convert strings to numbers using a dictionary and the map() function.
‘describe()’ on a non-numerical Series will give you some statistics (like count, unique and the most frequently occurring value).
As of pandas v15.0, use the parameter, DataFrame.describe(include = 'all') to get a summary of all the columns when the dataframe has mixed column types. The default behavior is to only provide a summary for the numerical columns.
Example:
In[1]:
df = pd.DataFrame({'$a':['a', 'b', 'c', 'd', 'a'], '$b': np.arange(5)})
df.describe(include = 'all')
Out[1]:
$a $b
count 5 5.000000
unique 4 NaN
top a NaN
freq 2 NaN
mean NaN 2.000000
std NaN 1.581139
min NaN 0.000000
25% NaN 1.000000
50% NaN 2.000000
75% NaN 3.000000
max NaN 4.000000
The numerical columns will have NaNs for summary statistics pertaining to objects (strings) and vice versa.
Summarizing only numerical or object columns
- To call
describe() on just the numerical columns use describe(include = [np.number])
-
To call describe() on just the objects (strings) using describe(include = ['O']).
In[2]:
df.describe(include = [np.number])
Out[3]:
$b
count 5.000000
mean 2.000000
std 1.581139
min 0.000000
25% 1.000000
50% 2.000000
75% 3.000000
max 4.000000
In[3]:
df.describe(include = ['O'])
Out[3]:
$a
count 5
unique 4
top a
freq 2
In addition to DataFrame.describe(include = 'all') one can also use Series.value_counts() for each categorical column:
In[1]:
df = pd.DataFrame({'$a':['a', 'b', 'c', 'd', 'a'], '$b': np.arange(5)})
df['$a'].value_counts()
Out[1]:
$a
a 2
d 1
b 1
c 1
You can execute df_test.info() to get the list of datatypes your data frame contains.If your data frame contains only numerical columns than df_test.describe() will work perfectly fine.As by default, it provides the summary of numerical values. If you want the summary of your Object(String) features you can use df_test.describe(include=['O']).
Or in short, you can just use df_test.describe(include='all') to get summary of all the feature columns when your data frame has columns of various data types.
pd.options.display.max_columns = DATA.shape[1] will work.
Here DATA is a 2d matrix, and above code will display stats vertically.
In addition to the data type issues discussed in the other answers, you might also have too many columns to display. If there are too many columns, the middle columns will be replaced with a total of three dots (...).
Other answers have pointed out that the include='all' parameter of describe can help with the data type issue. Another question asked, “How do I expand the output display to see more columns?” The solution is to modify the display.max_columns setting, which can even be done temporarily. For example, to display up to 40 columns of output from a single describe statement:
with pd.option_context('display.max_columns', 40):
print(df.describe(include='all'))
I am running ‘describe()’ on a dataframe and getting summaries of only int columns (pandas 14.0).
The documentation says that for object columns frequency of most common value, and additional statistics would be returned. What could be wrong? (no error message is returned by the way)
Edit:
I think it’s how the function is set to behave on mixed column types in a dataframe. Although the documentation fails to mention it.
Example code:
df_test = pd.DataFrame({'$a':[1,2], '$b': [10,20]})
df_test.dtypes
df_test.describe()
df_test['$a'] = df_test['$a'].astype(str)
df_test.describe()
df_test['$a'].describe()
df_test['$b'].describe()
My ugly work around in the meanwhile:
def my_df_describe(df):
objects = []
numerics = []
for c in df:
if (df[c].dtype == object):
objects.append(c)
else:
numerics.append(c)
return df[numerics].describe(), df[objects].describe()
‘describe()’ on a DataFrame only works for numeric types. If you think you have a numeric variable and it doesn’t show up in ‘decribe()’, change the type with:
df[['col1', 'col2']] = df[['col1', 'col2']].astype(float)
You could also create new columns for handling the numeric part of a mix type column, or convert strings to numbers using a dictionary and the map() function.
‘describe()’ on a non-numerical Series will give you some statistics (like count, unique and the most frequently occurring value).
As of pandas v15.0, use the parameter, DataFrame.describe(include = 'all') to get a summary of all the columns when the dataframe has mixed column types. The default behavior is to only provide a summary for the numerical columns.
Example:
In[1]:
df = pd.DataFrame({'$a':['a', 'b', 'c', 'd', 'a'], '$b': np.arange(5)})
df.describe(include = 'all')
Out[1]:
$a $b
count 5 5.000000
unique 4 NaN
top a NaN
freq 2 NaN
mean NaN 2.000000
std NaN 1.581139
min NaN 0.000000
25% NaN 1.000000
50% NaN 2.000000
75% NaN 3.000000
max NaN 4.000000
The numerical columns will have NaNs for summary statistics pertaining to objects (strings) and vice versa.
Summarizing only numerical or object columns
- To call
describe()on just the numerical columns usedescribe(include = [np.number]) -
To call
describe()on just the objects (strings) usingdescribe(include = ['O']).In[2]: df.describe(include = [np.number]) Out[3]: $b count 5.000000 mean 2.000000 std 1.581139 min 0.000000 25% 1.000000 50% 2.000000 75% 3.000000 max 4.000000 In[3]: df.describe(include = ['O']) Out[3]: $a count 5 unique 4 top a freq 2
In addition to DataFrame.describe(include = 'all') one can also use Series.value_counts() for each categorical column:
In[1]:
df = pd.DataFrame({'$a':['a', 'b', 'c', 'd', 'a'], '$b': np.arange(5)})
df['$a'].value_counts()
Out[1]:
$a
a 2
d 1
b 1
c 1
You can execute df_test.info() to get the list of datatypes your data frame contains.If your data frame contains only numerical columns than df_test.describe() will work perfectly fine.As by default, it provides the summary of numerical values. If you want the summary of your Object(String) features you can use df_test.describe(include=['O']).
Or in short, you can just use df_test.describe(include='all') to get summary of all the feature columns when your data frame has columns of various data types.
pd.options.display.max_columns = DATA.shape[1] will work.
Here DATA is a 2d matrix, and above code will display stats vertically.
In addition to the data type issues discussed in the other answers, you might also have too many columns to display. If there are too many columns, the middle columns will be replaced with a total of three dots (...).
Other answers have pointed out that the include='all' parameter of describe can help with the data type issue. Another question asked, “How do I expand the output display to see more columns?” The solution is to modify the display.max_columns setting, which can even be done temporarily. For example, to display up to 40 columns of output from a single describe statement:
with pd.option_context('display.max_columns', 40):
print(df.describe(include='all'))