numpy.reshape breaks when dealing with a view of a big size
Question:
I have an initial NumPy array of size (512,512,100) of type np.float64 and then I use view_as_windows function to get an array of size (499,499,100,64,64). This function returns a view that consumes much less memory than an actual NumPy array. I want to reshape the view to (499*499*100,64,64). Using the regular np.reshape function, it takes too much time trying to reshape the array, and then it breaks due to trying to convert it to an actual NumPy array. I tried the following, and none of them works:
#gs.shape gives (499,499,100,64,64)
gs.shape = (gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)
gs = np.reshape(gs,(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,))
gs = gs.reshape(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)
What is the correct way to change the view without actually causing memory overload?
Answers:
Why do you want to do this reshape?
Your starting array (with a small dtype):
In [100]: arr = np.ones((512,512,100), 'int8')
In [101]: arr.nbytes, arr.strides
Out[101]: (26214400, (51200, 100, 1))
Using the view_as_windows as ported to the numpy core:
In [102]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1))
In [103]: new.shape, new.strides
Out[103]: ((449, 449, 100, 64, 64),
(51200, 100, 1, 51200, 100))
In [105]: new.nbytes/1e9 # the number of elements (memory if copy)
Out[105]: 82.5757696
Note that the arr final dimension, 100, is now 3rd; in the view is still has a single element stride.
Effectively the first dim, 512, has been ‘split’ between axis 0 and 3; and the 2nd 512, axis (1,4).
To get the shape, (499499100,64,64), strides would have to be (? , 51200, 100). I’m not an expert on as_strided, but have enough experience to say there’s no way of reworking the first 3 strides (51200, 100, 1) into one.
Let’s remove the size 100 dimension to simplify things a bit.
In [106]: arr = np.ones((512,512), 'int8')
In [107]: arr.strides
Out[107]: (512, 1)
The windowed:
In [108]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1))
In [109]: new.shape, new.strides
Out[109]: ((449, 449, 64, 64), (512, 1, 512, 1))
and its 3d reshape (a copy):
In [110]: new1 = new.reshape(-1,64,64)
In [112]: new1.shape, new1.strides
Out[112]: ((201601, 64, 64), (4096, 64, 1))
I have an initial NumPy array of size (512,512,100) of type np.float64 and then I use view_as_windows function to get an array of size (499,499,100,64,64). This function returns a view that consumes much less memory than an actual NumPy array. I want to reshape the view to (499*499*100,64,64). Using the regular np.reshape function, it takes too much time trying to reshape the array, and then it breaks due to trying to convert it to an actual NumPy array. I tried the following, and none of them works:
#gs.shape gives (499,499,100,64,64)
gs.shape = (gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)
gs = np.reshape(gs,(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,))
gs = gs.reshape(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)
What is the correct way to change the view without actually causing memory overload?
Why do you want to do this reshape?
Your starting array (with a small dtype):
In [100]: arr = np.ones((512,512,100), 'int8')
In [101]: arr.nbytes, arr.strides
Out[101]: (26214400, (51200, 100, 1))
Using the view_as_windows as ported to the numpy core:
In [102]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1))
In [103]: new.shape, new.strides
Out[103]: ((449, 449, 100, 64, 64),
(51200, 100, 1, 51200, 100))
In [105]: new.nbytes/1e9 # the number of elements (memory if copy)
Out[105]: 82.5757696
Note that the arr final dimension, 100, is now 3rd; in the view is still has a single element stride.
Effectively the first dim, 512, has been ‘split’ between axis 0 and 3; and the 2nd 512, axis (1,4).
To get the shape, (499499100,64,64), strides would have to be (? , 51200, 100). I’m not an expert on as_strided, but have enough experience to say there’s no way of reworking the first 3 strides (51200, 100, 1) into one.
Let’s remove the size 100 dimension to simplify things a bit.
In [106]: arr = np.ones((512,512), 'int8')
In [107]: arr.strides
Out[107]: (512, 1)
The windowed:
In [108]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1))
In [109]: new.shape, new.strides
Out[109]: ((449, 449, 64, 64), (512, 1, 512, 1))
and its 3d reshape (a copy):
In [110]: new1 = new.reshape(-1,64,64)
In [112]: new1.shape, new1.strides
Out[112]: ((201601, 64, 64), (4096, 64, 1))