Determining when a column value changes in pandas dataframe

Question

I am looking to write a quick script that will run through a csv file with two columns and provide me the rows in which the values in column B switch from one value to another:

eg:

dataframe:

# |  A  |  B  
--+-----+-----
1 |  2  |  3
2 |  3  |  3
3 |  4  |  4
4 |  5  |  4
5 |  5  |  4

would tell me that the change happened between row 2 and row 3. I know how to get these values using for loops but I was hoping there was a more pythonic way of approaching this problem.

Asked By: badrobit

||

Source

Answer 1

You can create a new column for the difference

> df['C'] = df['B'].diff()
> print df
   #  A  B   C
0  1  2  3 NaN
1  2  3  3   0
2  3  4  4   1
3  4  5  4   0
4  5  5  4   0

> df_filtered = df[df['C'] != 0]
> print df_filtered
   #  A  B  C
2  3  4  4  1

This will your required rows

Answered By: Kathirmani Sukumar

Answer 2

You can do the following which also works for non numerical values:

>>> import pandas as pd
>>> df = pd.DataFrame({"Status": ["A","A","B","B","C","C","C"]})
>>> df["isStatusChanged"] = df["Status"].shift(1, fill_value=df["Status"].head(1)) != df["Status"]
>>> df
  Status  isStatusChanged
0      A            False
1      A            False
2      B             True
3      B            False
4      C             True
5      C            False
6      C            False
>>>

Note the fill_value could be different depending on your application.

Answered By: Hagalín Ásgrímur Guðmundsson

Answer 3

you can use this it is much faster, hope it helps!!

my_column_changes = df["MyStringColumn"].shift() != df["MyStringColumn"]

Answered By: Khalid Tamine

Determining when a column value changes in pandas dataframe

Question:

Answers: