What is an intuitive explanation of np.unravel_index?
Question:
Pretty much what the title says. I’ve read the documentation and I’ve played with the function for a while now but I can’t discern what the physical manifestation of this transformation is.
Answers:
Computer memory is addressed linearly. Each memory cell corresponds to a number. A block of memory can be addressed in terms of a base, which is the memory address of its first element, and the item index. For example, assuming the base address is 10,000:
item index 0 1 2 3
memory address 10,000 10,001 10,002 10,003
To store multi-dimensional blocks, their geometry must somehow be made to fit into linear memory. In C and NumPy, this is done row-by-row. A 2D example would be:
| 0 1 2 3
--+------------------------
0 | 0 1 2 3
1 | 4 5 6 7
2 | 8 9 10 11
So, for example, in this 3-by-4 block the 2D index (1, 2) would correspond to the linear index 6 which is 1 x 4 + 2.
unravel_index does the inverse. Given a linear index, it computes the corresponding ND index. Since this depends on the block dimensions, these also have to be passed. So, in our example, we can get the original 2D index (1, 2) back from the linear index 6:
>>> np.unravel_index(6, (3, 4))
(1, 2)
Note: The above glosses over a few details. 1) Translating the item index to memory address also has to account for item size. For example, an integer typically has 4 or 8 bytes. So, in the latter case, the memory address for item i would be base + 8 x i. 2). NumPy is a bit more flexible than suggested. It can organize ND data column-by-column if desired. It can even handle data that are not contiguous in memory but for example leave gaps, etc.
Bonus reading: internal memory layout of an ndarray
We will start with an example in the documentation.
>>> np.unravel_index([22, 41, 37], (7,6))
(array([3, 6, 6]), array([4, 5, 1]))
First, (7,6) specifies the dimension of target array that we want to turn the indices back into. Second, [22, 41, 37] are some indices on this array if the array is flattened. If a 7 by 6 array is flattened, its indices will look like
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, *22*, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35, 36, *37*, 38, 39, 40, *41*]
If we unflatten these indices back to their original positions in a dim (7, 6) array, it would be
[[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, *22*, 23], <- (3, 4)
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35],
[36, *37*, 38, 39, 40, *41*]]
(6, 1) (6,5)
The return values of the unravel_index function tell you what should have been the indices of [22, 41, 37] if the array is not flattened. These indices should have been [(3, 4), (6, 5), (6,1)] if the array is not flattened. In other words, the function transfers the indices in a flatten array back to its unflatten version.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.unravel_index.html
This isn’t different in content than the other two answers, but it might be more intuitive. If you have a 2-D matrix, or array, you can reference it in different ways. You could type the (row, col), to get the value at (row, col), or you can give each cell a single-number index. unravel_index just translates between these two ways of referencing values in a matrix.
This is extendable to dimensions larger than 2. You should also be aware of np.ravel_multi_index(), which performs the reverse transformation. Note that it requires the (row, col) and the shape of the array.
I also see I have two 10s in the index matrix–whoops.
I can explain it with very simple example. This is for np.ravel_multi_index as well as np.unravel_index
>>> X = np.array([[4, 2],
[9, 3],
[8, 5],
[3, 3],
[5, 6]])
>>> X.shape
(5, 2)
Find where all the value 3 presents in X:
>>> idx = np.where(X==3)
>>> idx
(array([1, 3, 3], dtype=int64), array([1, 0, 1], dtype=int64))
i.e. x = [1,3,3] , y = [1,0,1]
It returns the x, y of indices (because X is 2-dimensional).
If you apply ravel_multi_index for idx obtained:
>>> idx_flat = np.ravel_multi_index(idx, X.shape)
>>> idx_flat
array([3, 6, 7], dtype=int64)
idx_flat is a linear index of X where value 3 presents.
From the above example, we can understand:
- ravel_multi_index converts multi-dimensional indices (nd array) into single-dimensional indices (linear array)
- It works only on indices i.e. both input and output are indices
The result indices will be direct indices of X.ravel(). You can verify in the below x_linear:
>>> x_linear = X.ravel()
>>> x_linear
array([4, 2, 9, 3, 8, 5, 3, 3, 5, 6])
Whereas, unravel_index is very simple, just reverse of above (np.ravel_multi_index)
>>> idx = np.unravel_index(idx_flat , X.shape)
>>> idx
(array([1, 3, 3], dtype=int64), array([1, 0, 1], dtype=int64))
Which is same as idx = np.where(X==3)
- unravel_index converts single-dimensional indices (linear array) into multi-dimensional indices (nd array)
- It works only on indices i.e. both input and output are indices
This is only applicable for the 2D case, but the two coordinates np.unravel_index functions returns in this case are equivalent to doing floor division and applying the modulo function respectively.
for j in range(1,1000):
for i in range(j):
assert(np.unravel_index(i,(987654321,j))==(i//j,i%j))
The first element of the shape array (ie 987654321) is meaningless except to put an upper bound on how large an unraveled linear index can be passed through the function.
Given a raveled_index into a .ravel()ed array, np.unravel_index figures out the equivalent unraveled index into the base array:
import numpy as np
my_array = np.random.random((100, 42))
raveled_array = my_array.ravel()
raveled_index = 1337
unraveled_index = np.unravel_index(raveled_index, my_array.shape)
assert raveled_array[raveled_index] == my_array[unraveled_index]
Two nice to knows:
-
A raveled_array is also called a flat_array; hence a raveled_index or flat_index is nothing but an index into the "flattened" array. Further, since the flat array has "lost" its original shape information you need to add this when calling np.unravel_index.
-
The unraveled_index is typically called a multi_index. This is because you need multiple values (an N-tuple) to index an element in any array with my_array.dim == N. The inverse of np.unravel_index is thus called np.ravel_multi_index.
Pretty much what the title says. I’ve read the documentation and I’ve played with the function for a while now but I can’t discern what the physical manifestation of this transformation is.
Computer memory is addressed linearly. Each memory cell corresponds to a number. A block of memory can be addressed in terms of a base, which is the memory address of its first element, and the item index. For example, assuming the base address is 10,000:
item index 0 1 2 3
memory address 10,000 10,001 10,002 10,003
To store multi-dimensional blocks, their geometry must somehow be made to fit into linear memory. In C and NumPy, this is done row-by-row. A 2D example would be:
| 0 1 2 3
--+------------------------
0 | 0 1 2 3
1 | 4 5 6 7
2 | 8 9 10 11
So, for example, in this 3-by-4 block the 2D index (1, 2) would correspond to the linear index 6 which is 1 x 4 + 2.
unravel_index does the inverse. Given a linear index, it computes the corresponding ND index. Since this depends on the block dimensions, these also have to be passed. So, in our example, we can get the original 2D index (1, 2) back from the linear index 6:
>>> np.unravel_index(6, (3, 4))
(1, 2)
Note: The above glosses over a few details. 1) Translating the item index to memory address also has to account for item size. For example, an integer typically has 4 or 8 bytes. So, in the latter case, the memory address for item i would be base + 8 x i. 2). NumPy is a bit more flexible than suggested. It can organize ND data column-by-column if desired. It can even handle data that are not contiguous in memory but for example leave gaps, etc.
Bonus reading: internal memory layout of an ndarray
We will start with an example in the documentation.
>>> np.unravel_index([22, 41, 37], (7,6))
(array([3, 6, 6]), array([4, 5, 1]))
First, (7,6) specifies the dimension of target array that we want to turn the indices back into. Second, [22, 41, 37] are some indices on this array if the array is flattened. If a 7 by 6 array is flattened, its indices will look like
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, *22*, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35, 36, *37*, 38, 39, 40, *41*]
If we unflatten these indices back to their original positions in a dim (7, 6) array, it would be
[[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, *22*, 23], <- (3, 4)
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35],
[36, *37*, 38, 39, 40, *41*]]
(6, 1) (6,5)
The return values of the unravel_index function tell you what should have been the indices of [22, 41, 37] if the array is not flattened. These indices should have been [(3, 4), (6, 5), (6,1)] if the array is not flattened. In other words, the function transfers the indices in a flatten array back to its unflatten version.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.unravel_index.html
This isn’t different in content than the other two answers, but it might be more intuitive. If you have a 2-D matrix, or array, you can reference it in different ways. You could type the (row, col), to get the value at (row, col), or you can give each cell a single-number index. unravel_index just translates between these two ways of referencing values in a matrix.
This is extendable to dimensions larger than 2. You should also be aware of np.ravel_multi_index(), which performs the reverse transformation. Note that it requires the (row, col) and the shape of the array.
I also see I have two 10s in the index matrix–whoops.
I can explain it with very simple example. This is for np.ravel_multi_index as well as np.unravel_index
>>> X = np.array([[4, 2],
[9, 3],
[8, 5],
[3, 3],
[5, 6]])
>>> X.shape
(5, 2)
Find where all the value 3 presents in X:
>>> idx = np.where(X==3)
>>> idx
(array([1, 3, 3], dtype=int64), array([1, 0, 1], dtype=int64))
i.e. x = [1,3,3] , y = [1,0,1]
It returns the x, y of indices (because X is 2-dimensional).
If you apply ravel_multi_index for idx obtained:
>>> idx_flat = np.ravel_multi_index(idx, X.shape)
>>> idx_flat
array([3, 6, 7], dtype=int64)
idx_flat is a linear index of X where value 3 presents.
From the above example, we can understand:
- ravel_multi_index converts multi-dimensional indices (nd array) into single-dimensional indices (linear array)
- It works only on indices i.e. both input and output are indices
The result indices will be direct indices of X.ravel(). You can verify in the below x_linear:
>>> x_linear = X.ravel()
>>> x_linear
array([4, 2, 9, 3, 8, 5, 3, 3, 5, 6])
Whereas, unravel_index is very simple, just reverse of above (np.ravel_multi_index)
>>> idx = np.unravel_index(idx_flat , X.shape)
>>> idx
(array([1, 3, 3], dtype=int64), array([1, 0, 1], dtype=int64))
Which is same as idx = np.where(X==3)
- unravel_index converts single-dimensional indices (linear array) into multi-dimensional indices (nd array)
- It works only on indices i.e. both input and output are indices
This is only applicable for the 2D case, but the two coordinates np.unravel_index functions returns in this case are equivalent to doing floor division and applying the modulo function respectively.
for j in range(1,1000):
for i in range(j):
assert(np.unravel_index(i,(987654321,j))==(i//j,i%j))
The first element of the shape array (ie 987654321) is meaningless except to put an upper bound on how large an unraveled linear index can be passed through the function.
Given a raveled_index into a .ravel()ed array, np.unravel_index figures out the equivalent unraveled index into the base array:
import numpy as np
my_array = np.random.random((100, 42))
raveled_array = my_array.ravel()
raveled_index = 1337
unraveled_index = np.unravel_index(raveled_index, my_array.shape)
assert raveled_array[raveled_index] == my_array[unraveled_index]
Two nice to knows:
-
A
raveled_arrayis also called aflat_array; hence araveled_indexorflat_indexis nothing but an index into the "flattened" array. Further, since the flat array has "lost" its original shape information you need to add this when callingnp.unravel_index. -
The
unraveled_indexis typically called amulti_index. This is because you need multiple values (an N-tuple) to index an element in any array withmy_array.dim == N. The inverse ofnp.unravel_indexis thus callednp.ravel_multi_index.