pandas dropna dropping the whole dataframe, need only to drop empty rows

Question

I’m using this piece of code:

import pandas as pd
df = pd.read_excel('input.xls', sheet_name='Nouveau concept')
print(f"Dataframe:n{df}")
new_df = df.dropna()
print(f"Dataframe now:n{new_df}")

To read an Excel file (it has to be xls and not xlsx) and drop all empty rows, i.e., rows that contain no data at all.

When I run the above, I get this:

Anibals-New-MacBook-Air:UCNI anibal$ python3 test.py
Dataframe:
     Source Terminology Version  Requestor Internal ID    Parent ID                    Parent FSN  ... Unnamed: 77 Unnamed: 78 Unnamed: 79  Unnamed: 80
0                september 2022                    NaN  283403005.0  Cut of ear region (disorder)  ...         NaN         NaN         NaN          NaN
1                september 2022                    NaN  283403005.0  Cut of ear region (disorder)  ...         NaN         NaN         NaN          NaN
2                september 2022                    NaN  283412007.0   Cut of upper arm (disorder)  ...         NaN         NaN         NaN          NaN
3                september 2022                    NaN  283412007.0   Cut of upper arm (disorder)  ...         NaN         NaN         NaN          NaN
4                september 2022                    NaN  283413002.0       Cut of elbow (disorder)  ...         NaN         NaN         NaN          NaN
...                         ...                    ...          ...                           ...  ...         ...         ...         ...          ...
5056                        NaN                    NaN          NaN                           NaN  ...         NaN         NaN         NaN          NaN
5057                        NaN                    NaN          NaN                           NaN  ...         NaN         NaN         NaN          NaN
5058                        NaN                    NaN          NaN                           NaN  ...         NaN         NaN         NaN          NaN
5059                        NaN                    NaN          NaN                           NaN  ...         NaN         NaN         NaN          NaN
5060                        NaN                    NaN          NaN                           NaN  ...         NaN         NaN         NaN          NaN

[5061 rows x 81 columns]
Dataframe now:
Empty DataFrame
Columns: [Source Terminology Version, Requestor Internal ID, Parent ID, Parent FSN, FSN (*), Semantic Tag (*), PT (*), Synonym (1), Synonym (2), Definition, Reason for Change, Notes, References, Unnamed: 13, Unnamed: 14, Unnamed: 15, Unnamed: 16, Unnamed: 17, Unnamed: 18, Unnamed: 19, Unnamed: 20, Unnamed: 21, Unnamed: 22, Unnamed: 23, Unnamed: 24, Unnamed: 25, Unnamed: 26, Unnamed: 27, Unnamed: 28, Unnamed: 29, Unnamed: 30, Unnamed: 31, Unnamed: 32, Unnamed: 33, Unnamed: 34, Unnamed: 35, Unnamed: 36, Unnamed: 37, Unnamed: 38, Unnamed: 39, Unnamed: 40, Unnamed: 41, Unnamed: 42, Unnamed: 43, Unnamed: 44, Unnamed: 45, Unnamed: 46, Unnamed: 47, Unnamed: 48, Unnamed: 49, Unnamed: 50, Unnamed: 51, Unnamed: 52, Unnamed: 53, Unnamed: 54, Unnamed: 55, Unnamed: 56, Unnamed: 57, Unnamed: 58, Unnamed: 59, Unnamed: 60, Unnamed: 61, Unnamed: 62, Unnamed: 63, Unnamed: 64, Unnamed: 65, Unnamed: 66, Unnamed: 67, Unnamed: 68, Unnamed: 69, Unnamed: 70, Unnamed: 71, Unnamed: 72, Unnamed: 73, Unnamed: 74, Unnamed: 75, Unnamed: 76, Unnamed: 77, Unnamed: 78, Unnamed: 79, Unnamed: 80]
Index: []

So, the second dataframe is completely empty. Why?

I just need to read the rows that contain any data, i.e., if a row is just empty, skip it.

The input file input.xls can be found here:

https://docs.google.com/spreadsheets/d/1pXfhPHklnd0v45yLbff5E5dp2ypVIbxG/edit?usp=share_link&ouid=117900420544251849196&rtpof=true&sd=true

Any ideas?

I can’t clean up the file by the way. This input file is generated by another system and my piece is supposed to automate handling this file, so I can’t just load it in Excel and clean it up.

I tried a whole bunch of combinations of dropna to no avail. I also tried several other solutions found in stackoverflow and again, to no avail.

Asked By: Anibal Jodorcovsky

||

Source

Answer 1

First thing, import only the required columns (i.e. exclude blank ones by using use_cols)

df = pd.read_excel('input.xls', sheet_name='Nouveau concept',usecols="A:M")

Then, to drop the empty rows, you have to consider a subset of columns. Currently, there are a few columns that are completely empty, so that is the reason why all rows are dropped. To combat this, use the following:

new_df =df.dropna(subset=['Source Terminology Version'], how = 'all')
# In this example, I used only one column, but you can pass in a list.

Answered By: Karel Räppo

pandas dropna dropping the whole dataframe, need only to drop empty rows

Question:

Answers: