How to index out the first and last row of each day in a time series dataframe?

Question

So I have a csv file with minute stock data and the code looks like this:

d = pd.read_csv('/Volumes/Seagate Portable/usindex_2020_all_tickers_awvbxk9/SPX_2020_2020.txt')
d.columns = ['Dates', 'Open', 'High', 'Low', 'Close']
d.set_index('Dates', inplace=True)
d.drop(['High', 'Low'], axis=1, inplace=True)
d = d.between_time('9:30', '16:00')

So each index has a year, day, month and time. What I am looking to do is by using the day date, to index out the first and last quote of the day, between 9:30 and 4:00.

The output looks like this:

    Open    Close
Dates       
2020-01-02 09:31:00 3247.19 3245.22
2020-01-02 09:32:00 3245.07 3244.66
2020-01-02 09:33:00 3244.89 3247.61
2020-01-02 09:34:00 3247.38 3246.92
2020-01-02 09:35:00 3246.89 3249.09
... ... ...
2020-12-24 13:17:00 3703.06 3703.06
2020-12-24 13:18:00 3703.06 3703.06
2020-12-24 13:19:00 3703.06 3703.06
2020-12-24 13:20:00 3703.06 3703.06
2020-12-24 14:22:00 3703.06 3703.06

As the code shows, that the first and last price is not always 9:30 and 4:00, so Im trying to find a way to index out just the first and last price no matter the time.

Asked By: benito.cano

||

Source

Answer 1

Use groupby:

d = d.between_time('9:30', '16:00')
d.groupby(pd.Grouper(freq='D')).agg({'Open':'first', 'Close':'last'})

Answered By: Quang Hoang

Answer 2

If the answer above works its much shorter but I didn’t get it to work.

import datetime as dt

d = pd.DataFrame({'Dates': ['2020-01-02 09:15:00', '2020-01-02 09:31:00', '2020-01-02 09:32:00', '2020-01-02 09:33:00', '2020-01-02 09:34:00', '2020-01-03 09:35:00', '2020-01-03 16:00:00'], 'Open': [3247.19, 3247.19, 3245.07, 3244.89, 3247.38, 3246.89, 3247.19], 'Close': [3245.22, 3245.22, 3244.66, 3247.61, 3246.92, 3249.09, 3245.22]})
# df['Dates'].astype('datetime64[ns]')
d['Dates']=d['Dates']

d['Dates'] = pd.to_datetime(d['Dates'])
d['just_date'] = d['Dates'].dt.date
d['just_time'] = d['Dates'].dt.time
d2=df[(d['just_time'] >= dt.time(9,30,0)) & (d['just_time'] <= dt.time(16,0,0))]

dmins=df.groupby('just_date').min()
dmaxs=df.groupby('just_date').max()

d2=dfmins.append(dfmaxs)
d2.sort_index(inplace=True)
d2

Answered By: AndrewGraham

Answer 3

Slight modification for better result

import datetime as dt

d = pd.DataFrame({'Dates': ['2020-01-02 09:15:00', '2020-01-02 09:31:00', '2020-01-02 09:32:00', '2020-01-02 09:33:00', '2020-01-02 09:34:00', '2020-01-03 09:35:00', '2020-01-03 16:00:00'], 'Open': [3247.19, 3247.19, 3245.07, 3244.89, 3247.38, 3246.89, 3247.19], 'Close': [3245.22, 3245.22, 3244.66, 3247.61, 3246.92, 3249.09, 3245.22]})
# df['Dates'].astype('datetime64[ns]')
d['Dates']=d['Dates']

d['Dates'] = pd.to_datetime(d['Dates'])
d['just_date'] = d['Dates'].dt.date
d['just_time'] = d['Dates'].dt.time
d2=df[(d['just_time'] >= dt.time(9,30,0)) & (d['just_time'] <= dt.time(16,0,0))]

dmins=df.groupby('just_date').min()
dmaxs=df.groupby('just_date').max()

d2=dfmins.append(dfmaxs)
d2.sort_values(by='date',inplace=True)
d2

Answered By: Bibin Varghses

How to index out the first and last row of each day in a time series dataframe?

Question:

Answers: