Replace some values in a dataframe with NaN's if the index of the row does not exist in a multi-index object

Question

I have a really large dataframe similar to this:

    Customer_Id   Day    Hour   Latitude   Longitude     
0.        a        dd     hh      x1         y1
1.        a        dd     hh'     x2         y2
2.        a        dd     hh'     x3         y3
3.        b        dd     hh'     x4         y4

And then I have an object (that I can convert into a DataFrame if necessary) with a sample per hour per day per customer of the latitude and longitude. However, Customer_Id, Day and Hour are all indices here whereas before they were not. It looks something like this:

                                 Latitude   Longitude
Customer_Id    Day     Hour
    a          dd       hh         x1         y1
               dd       hh'        x3         y3
    b          dd       hh'        x4         y4

Previously, I had two dataframes with only one index each (let’s call them df1 for the first, which is the first dataframe here, and df2 for the second, which is the single index dataframe that I had instead of the second object) so I used:

df1['Latitude']= np.where((~df1.index.isin(df2.index)), np.nan, df1['Latitude'])

Previously this code worked but now under this new scenario it returns this error:

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

I have tried to change the code accordingly but it’s not working. Could someone take a look at this?

Asked By: Nocas

||

Source

Answer 1

You need to reset your index with DataFrame.reset_index:

df1['Latitude']= np.where((~df1.index.isin(df2.reset_index().index)), np.nan, df1['Latitude'])

print(df1)
  Customer_Id Day Hour Latitude Longitude
0           a  dd   hh       x1        y1
1           a  dd  hh'       x2        y2
2           a  dd  hh'       x3        y3
3           b  dd  hh'      NaN        y4

What does reset_index do?
It converts the indices back to columns:

print(df2.reset_index())
  Customer_Id Day Hour Latitude Longitude
0           a  dd   hh       x1        y1
1           a  dd  hh'       x3        y3
2           b  dd  hh'       x4        y4

Answered By: Erfan

Replace some values in a dataframe with NaN's if the index of the row does not exist in a multi-index object

Question:

Answers: