Enable full pandas summary on a huge data frame
Question:
I have a large data frame. Usually, when I have a data frame like this I get the summary for that data frame, where I get the info of how many non-NaN values in each column and column names. However for this one I get an even shorter summary:
<class 'pandas.core.frame.DataFrame'>
Index: 138289 entries, 1993-07-23 to 2012-11-26
Columns: 101 entries, AAT to ZZT
dtypes: object(101)
I’d like to get a standard summary, with info about each column.
I’m using ipython notebook and pandas 0.9.1 if that has anything to do with it
Answers:
Try setting the maximum rows displayed in a DataFrame using set_printoptions:
pd.set_printoptions(max_columns=101)
This should allow you to see all of the columns in the (summarized) DataFrame.
The max_rows and max_columns control how many rows and columns of DataFrame objects are shown by default.
when the dataframe is large it does not display in ipython notebook .
I just force it to:
from IPython.display import HTML
HTML(df.head().to_html())
make sure to use head 🙂
DataFrame.info unfortunately is hacked to not display the full summary unless there are fewer than 100 columns (look at the source code). We’ll get it fixed for 0.10:
I have a large data frame. Usually, when I have a data frame like this I get the summary for that data frame, where I get the info of how many non-NaN values in each column and column names. However for this one I get an even shorter summary:
<class 'pandas.core.frame.DataFrame'>
Index: 138289 entries, 1993-07-23 to 2012-11-26
Columns: 101 entries, AAT to ZZT
dtypes: object(101)
I’d like to get a standard summary, with info about each column.
I’m using ipython notebook and pandas 0.9.1 if that has anything to do with it
Try setting the maximum rows displayed in a DataFrame using set_printoptions:
pd.set_printoptions(max_columns=101)
This should allow you to see all of the columns in the (summarized) DataFrame.
The
max_rowsandmax_columnscontrol how many rows and columns of DataFrame objects are shown by default.
when the dataframe is large it does not display in ipython notebook .
I just force it to:
from IPython.display import HTML
HTML(df.head().to_html())
make sure to use head 🙂
DataFrame.info unfortunately is hacked to not display the full summary unless there are fewer than 100 columns (look at the source code). We’ll get it fixed for 0.10: