Is there a concise way to show all rows in pandas for just the current command?
Question:
Sometimes I want to show all of the rows in a pandas DataFrame, but only for a single command or code-block.
Of course I can set the “max_rows” display option to a large number, but then I have to repeat the command afterwards in order to revert to my preferred setting. (I like 12 rows max, personally).
pd.options.display.max_rows=1000
myDF
pd.options.display.max_rows=12
That’s annoying.
I read in the documentation that I can use the pd.option_context() function to accomplish this if I combine my command with a “with” statement:
with pd.option_context("display.max_rows", 1000): myDF
I couldn’t get that to work, (no output is returned).
But I think such a solution would still be too much typing for routine incidental usage!
I wish there was some quick pythonic way to override the display options!
Does one exist? Have I overlooked something?
I like how one can alter the # of rows that the .head() function outputs by passing it an argument for the # of rows, but it still must be lower than the “display.max_rows” setting…
I know I could keep the “display.max_rows” setting really high all the time, and then tack a .head(12) function on most of the time, but I think most people would agree on how annoying that would be.
I am indeed aware that one can view all (or most of?) the values in a pandas Series by passing it to a core function such as list(). But that is tricky to do with a DF. Furthermore, it’s hard to read when it’s not in a tabular format.
Similar to the solution for my first question, I imagine there’s probably a way to write my own function (to be placed in a startup script), but I’m not sure the best way to write it.
Answers:
You could write a function that explicitly calls display
E.g., consider this function:
from IPython.display import display
def show_more(df, lines):
foo = 1
display(df)
foo = 2
When I call the function (just tried it):
>> show_more(df, 1000)
... # <- Shows here the DF
then it displays the dataframe, even though the line foo = 2 is executed after. It follows that you can set the options instead of the line foo = 1 and unset it in the line foo = 2. In fact, you can just use the context manager from your question, probably.
This won’t display anything because it does not return anything:
with pd.option_context("display.max_rows", 1000): myDF
Calling display inside the with block should work:
with pd.option_context("display.max_rows", 1000):
display(myDF)
This seems to work as expected in pandas 0.22.0 (importing only pandas, without IPython):
import pandas as pd
with pd.option_context("display.max_rows", 1000): myDF
Presumably that’s because the default behaviour is to return the repr of myDF. IDEs may well override this.
If that’s too much typing, then a straightforward print to the terminal also works when wrapped in a function:
from __future__ import print_statement # for python2
def show_rows(df, nrows=1000):
with pd.option_context("display.max_rows", nrows): print(df)
Edit: calling show_rows(df) will by default print the first 1000 rows of your dataframe df to standard output.
You can use following helper function to print full data frame and set max_rows to normal after printing.
def print_full(df):
import pandas as pd
pd.set_option('display.max_rows', len(df))
print(df)
pd.reset_option('display.max_rows')
One-liner to force display all rows (in jupyter):
import IPython.display
IPython.display.HTML(df.to_html())
df.show(number of records)
Example:
df=spark.read.format('csv').load('hdfs://localhost:9000/user/test/region_location', header='false', nullValue='null', schema=transchema)
df.show(50)
Best way to display required number of rows in a resulting dataframe:
For rows:
pd.set_option('display.max_rows',100)
For columns:
Use display.max_columns option
pd.set_option('display.max_columns',100)
Just set the option to display max rows:
max_number_rows_to_display = 100
pd.set_option('display.max_rows', max_number_rows_to_display)
Sometimes I want to show all of the rows in a pandas DataFrame, but only for a single command or code-block.
Of course I can set the “max_rows” display option to a large number, but then I have to repeat the command afterwards in order to revert to my preferred setting. (I like 12 rows max, personally).
pd.options.display.max_rows=1000
myDF
pd.options.display.max_rows=12
That’s annoying.
I read in the documentation that I can use the pd.option_context() function to accomplish this if I combine my command with a “with” statement:
with pd.option_context("display.max_rows", 1000): myDF
I couldn’t get that to work, (no output is returned).
But I think such a solution would still be too much typing for routine incidental usage!
I wish there was some quick pythonic way to override the display options!
Does one exist? Have I overlooked something?
I like how one can alter the # of rows that the .head() function outputs by passing it an argument for the # of rows, but it still must be lower than the “display.max_rows” setting…
I know I could keep the “display.max_rows” setting really high all the time, and then tack a .head(12) function on most of the time, but I think most people would agree on how annoying that would be.
I am indeed aware that one can view all (or most of?) the values in a pandas Series by passing it to a core function such as list(). But that is tricky to do with a DF. Furthermore, it’s hard to read when it’s not in a tabular format.
Similar to the solution for my first question, I imagine there’s probably a way to write my own function (to be placed in a startup script), but I’m not sure the best way to write it.
You could write a function that explicitly calls display
E.g., consider this function:
from IPython.display import display
def show_more(df, lines):
foo = 1
display(df)
foo = 2
When I call the function (just tried it):
>> show_more(df, 1000)
... # <- Shows here the DF
then it displays the dataframe, even though the line foo = 2 is executed after. It follows that you can set the options instead of the line foo = 1 and unset it in the line foo = 2. In fact, you can just use the context manager from your question, probably.
This won’t display anything because it does not return anything:
with pd.option_context("display.max_rows", 1000): myDF
Calling display inside the with block should work:
with pd.option_context("display.max_rows", 1000):
display(myDF)
This seems to work as expected in pandas 0.22.0 (importing only pandas, without IPython):
import pandas as pd
with pd.option_context("display.max_rows", 1000): myDF
Presumably that’s because the default behaviour is to return the repr of myDF. IDEs may well override this.
If that’s too much typing, then a straightforward print to the terminal also works when wrapped in a function:
from __future__ import print_statement # for python2
def show_rows(df, nrows=1000):
with pd.option_context("display.max_rows", nrows): print(df)
Edit: calling show_rows(df) will by default print the first 1000 rows of your dataframe df to standard output.
You can use following helper function to print full data frame and set max_rows to normal after printing.
def print_full(df):
import pandas as pd
pd.set_option('display.max_rows', len(df))
print(df)
pd.reset_option('display.max_rows')
One-liner to force display all rows (in jupyter):
import IPython.display
IPython.display.HTML(df.to_html())
df.show(number of records)
Example:
df=spark.read.format('csv').load('hdfs://localhost:9000/user/test/region_location', header='false', nullValue='null', schema=transchema)
df.show(50)
Best way to display required number of rows in a resulting dataframe:
For rows:
pd.set_option('display.max_rows',100)
For columns:
Use display.max_columns option
pd.set_option('display.max_columns',100)
Just set the option to display max rows:
max_number_rows_to_display = 100
pd.set_option('display.max_rows', max_number_rows_to_display)