Pandas dataframe – reshaping columns using start and end pairs


I have been trying to figure out a way to transform this dataframe. It contains only two columns; one is timeseries data and the other is event markers. Here is an example of the initial dataframe:

df1 = pd.DataFrame({'Time': ['1:42 AM','2:30 AM','3:29 AM','4:19 AM','4:37 AM','4:59 AM','5:25 AM','5:33 AM','6:48 AM'], 
                    'Event': ['End','Start','End','Start','End','Start','Start','End','Start']})

This is how I would like it to look after transformation:

df2 = pd.DataFrame({'Start': ['', '2:30 AM', '4:19 AM', '4:59 AM', '5:25 AM', '6:48 AM'],
                    'End': ['1:42 AM', '3:29 AM', '4:37 AM', '', '5:33 AM', '']})

Essentially, I want to make the event markers the new columns, with the start and end times paired up down the table as they happen.

This example includes both exceptions that will sometimes happen:

  1. First row of data is an ‘End’ marker (data gets truncated by date).
  2. Sometimes there will not be an ‘End’ marker for a specific ‘Start’ marker (job failed or wasn’t complete at time the report was run).

I looked at pivot and pivot_table but I wasn’t able to add an index that got the output I wanted. Pretty sure this should be possible, I just am not an expert with dataframes yet.

Asked By: MBlankfield




df1["tmp"] = df1["Event"].eq("Start").cumsum()
df1 = df1.pivot(index="tmp", columns="Event", values="Time").fillna(""), = None, None

print(df1[["Start", "End"]])


     Start      End
0           1:42 AM
1  2:30 AM  3:29 AM
2  4:19 AM  4:37 AM
3  4:59 AM         
4  5:25 AM  5:33 AM
5  6:48 AM         
Answered By: Andrej Kesely

The difficulty of this operation is that you have to create your custom index. Besides using pivot_table and some index operations to get your desired format:

piv = (
    .pivot_table(index="idx", columns="Event", values="Time", aggfunc=lambda x: x, fill_value="")
    .rename_axis(None, axis="columns")
    .reindex(columns=["Start", "End"])

     Start      End
0           1:42 AM
1  2:30 AM  3:29 AM
2  4:19 AM  4:37 AM
3  4:59 AM         
4  5:25 AM  5:33 AM
5  6:48 AM  
Answered By: Erfan
Categories: questions Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.