Rolling window for 1D arrays in Numpy?
Question:
Is there a way to efficiently implement a rolling window for 1D arrays in Numpy?
For example, I have this pure Python code snippet to calculate the rolling standard deviations for a 1D list, where observations is the 1D list of values, and n is the window length for the standard deviation:
stdev = []
for i, data in enumerate(observations[n-1:]):
strip = observations[i:i+n]
mean = sum(strip) / n
stdev.append(sqrt(250*sum([(s-mean)**2 for s in strip])/(n-1)))
Is there a way to do this completely within Numpy, i.e., without any Python loops? The standard deviation is trivial with numpy.std, but the rolling window part completely stumps me.
I found this blog post regarding a rolling window in Numpy, but it doesn’t seem to be for 1D arrays.
Answers:
Just use the blog code, but apply your function to the result.
i.e.
numpy.std(rolling_window(observations, n), 1)
where you have (from the blog):
def rolling_window(a, window):
shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
strides = a.strides + (a.strides[-1],)
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
def moving_avg(x,n):
mv = np.convolve(x,np.ones(n)/n,mode='valid')
return np.concatenate(([np.NaN for k in range(n-1)],mv))
With only one line of code…
import pandas as pd
pd.Series(observations).rolling(n).std()
I tried using so12311‘s answer listed above on a 2D array with shape [samples, features] in order to get an output array with shape [samples, timesteps, features] for use with a convolution or lstm neural network, but it wasn’t working quite right. After digging into how the strides were working, I realized that it was moving the window along the last axis, so I made some adjustments so that the window is moved along the first axis instead:
def rolling_window(a, window_size):
shape = (a.shape[0] - window_size + 1, window_size) + a.shape[1:]
strides = (a.strides[0],) + a.strides
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
NOTE: there is no difference in the output if you are only using a 1D input array. In my search this was the first result to get close to what I wanted to do, so I am adding this to help any others searching for a similar answer.
Starting in Numpy 1.20, you can directly get a rolling window with sliding_window_view:
from numpy.lib.stride_tricks import sliding_window_view
sliding_window_view(np.array([1, 2, 3, 4, 5, 6]), window_shape = 3)
# array([[1, 2, 3],
# [2, 3, 4],
# [3, 4, 5],
# [4, 5, 6]])
Based on latter answers, here I add code for rolling 1-D numpy arrays choosing window size and window steps frequency.
a = np.arange(50)
def rolling_window(array, window_size,freq):
shape = (array.shape[0] - window_size + 1, window_size)
strides = (array.strides[0],) + array.strides
rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
return rolled[np.arange(0,shape[0],freq)]
rolling_window(a,10,5)
Output:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[15, 16, 17, 18, 19, 20, 21, 22, 23, 24],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[25, 26, 27, 28, 29, 30, 31, 32, 33, 34],
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[35, 36, 37, 38, 39, 40, 41, 42, 43, 44],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]])
I needed a rolling window to apply to any intermediate axis of an n-dimensional array, so I extended the code from the already accepted answer and @Miguel Gonzalez. The corresponding code to apply a rolling window to an n-d array along any axis:
def rolling_window(array, window, freq, axis=0):
shape = array.shape[:axis] + (array.shape[axis] - window_size + 1, window_size) + array.shape[axis+1:]
strides = array.strides[:axis] + (array.strides[axis],) + array.strides[axis:]
rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
return np.take(rolled, np.arange(0,shape[axis],freq), axis=axis)
An example to create a test to assert validity of the function:
arr = np.random.randint(1, 1000, size=(2,108,21,5))
arr_windowed = rolling_window_ndimensional(arr, 12, 12, axis=1)
print(arr.shape)
print(arr_windowed.shape)
np.allclose(arr, arr_windowed.reshape(2,-1, 21,5))
Is there a way to efficiently implement a rolling window for 1D arrays in Numpy?
For example, I have this pure Python code snippet to calculate the rolling standard deviations for a 1D list, where observations is the 1D list of values, and n is the window length for the standard deviation:
stdev = []
for i, data in enumerate(observations[n-1:]):
strip = observations[i:i+n]
mean = sum(strip) / n
stdev.append(sqrt(250*sum([(s-mean)**2 for s in strip])/(n-1)))
Is there a way to do this completely within Numpy, i.e., without any Python loops? The standard deviation is trivial with numpy.std, but the rolling window part completely stumps me.
I found this blog post regarding a rolling window in Numpy, but it doesn’t seem to be for 1D arrays.
Just use the blog code, but apply your function to the result.
i.e.
numpy.std(rolling_window(observations, n), 1)
where you have (from the blog):
def rolling_window(a, window):
shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
strides = a.strides + (a.strides[-1],)
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
def moving_avg(x,n):
mv = np.convolve(x,np.ones(n)/n,mode='valid')
return np.concatenate(([np.NaN for k in range(n-1)],mv))
With only one line of code…
import pandas as pd
pd.Series(observations).rolling(n).std()
I tried using so12311‘s answer listed above on a 2D array with shape [samples, features] in order to get an output array with shape [samples, timesteps, features] for use with a convolution or lstm neural network, but it wasn’t working quite right. After digging into how the strides were working, I realized that it was moving the window along the last axis, so I made some adjustments so that the window is moved along the first axis instead:
def rolling_window(a, window_size):
shape = (a.shape[0] - window_size + 1, window_size) + a.shape[1:]
strides = (a.strides[0],) + a.strides
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
NOTE: there is no difference in the output if you are only using a 1D input array. In my search this was the first result to get close to what I wanted to do, so I am adding this to help any others searching for a similar answer.
Starting in Numpy 1.20, you can directly get a rolling window with sliding_window_view:
from numpy.lib.stride_tricks import sliding_window_view
sliding_window_view(np.array([1, 2, 3, 4, 5, 6]), window_shape = 3)
# array([[1, 2, 3],
# [2, 3, 4],
# [3, 4, 5],
# [4, 5, 6]])
Based on latter answers, here I add code for rolling 1-D numpy arrays choosing window size and window steps frequency.
a = np.arange(50)
def rolling_window(array, window_size,freq):
shape = (array.shape[0] - window_size + 1, window_size)
strides = (array.strides[0],) + array.strides
rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
return rolled[np.arange(0,shape[0],freq)]
rolling_window(a,10,5)
Output:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[15, 16, 17, 18, 19, 20, 21, 22, 23, 24],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[25, 26, 27, 28, 29, 30, 31, 32, 33, 34],
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[35, 36, 37, 38, 39, 40, 41, 42, 43, 44],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]])
I needed a rolling window to apply to any intermediate axis of an n-dimensional array, so I extended the code from the already accepted answer and @Miguel Gonzalez. The corresponding code to apply a rolling window to an n-d array along any axis:
def rolling_window(array, window, freq, axis=0):
shape = array.shape[:axis] + (array.shape[axis] - window_size + 1, window_size) + array.shape[axis+1:]
strides = array.strides[:axis] + (array.strides[axis],) + array.strides[axis:]
rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
return np.take(rolled, np.arange(0,shape[axis],freq), axis=axis)
An example to create a test to assert validity of the function:
arr = np.random.randint(1, 1000, size=(2,108,21,5))
arr_windowed = rolling_window_ndimensional(arr, 12, 12, axis=1)
print(arr.shape)
print(arr_windowed.shape)
np.allclose(arr, arr_windowed.reshape(2,-1, 21,5))