Check if a large matrix is diagonal matrix in python
Question:
I have compute a very large matrix M with lots of degenerate eigenvectors(different eigenvectors with same eigenvalues). I use QR decomposition to make sure these eigenvectors are orthonormal, so the Q is the orthonormal eigenvectors of M, and Q^{-1}MQ = D, where D is diagonal matrix. Now I want to check if D is truly diagonal matrix, but when I print D, the matrix is too large to show all of them, so how can I know if it is truly diagonal matrix?
Answers:
I afraid if this is the most efficient way of doing this, But the idea is to mask the diagonal elements and check if all the other elements are zero. I guess this is sufficient check to label a matrix as diagonal matrix.
So we create a dummy array with same size as input matrix, initialized with ones. and then replace the diagonal elements with zeros. Now we perform element wise multiplication of input matrix and dummy matrix. So here we replace the diagonal elements of input matrix with zero and leave the other elements as it is.
Now finally we check if there are any non zero elements.
def is_diagonal(matrix):
#create a dummy matrix
dummy_matrix = np.ones(matrix.shape, dtype=np.uint8)
# Fill the diagonal of dummy matrix with 0.
np.fill_diagonal(dummy_matrix, 0)
return np.count_nonzero(np.multiply(dummy_matrix, matrix)) == 0
diagonal_matrix = np.array([[3, 0, 0],
[0, 7, 0],
[0, 0, 4]])
print is_diagonal(diagonal_matrix)
>>> True
random_matrix = np.array([[3, 8, 0],
[1, 7, 8],
[5, 0, 4]])
print is_diagonal(random_matrix)
>>> False
Remove the diagonal and count the non zero elements:
np.count_nonzero(x - np.diag(np.diagonal(x)))
Not sure how fast this is compared to the others, but:
def isDiag(M):
i, j = np.nonzero(M)
return np.all(i == j)
EDIT Let’s time things:
M = np.random.randint(0, 10, 1000) * np.eye(1000)
def a(M): #donkopotamus solution
return np.count_nonzero(M - np.diag(np.diagonal(M)))
%timeit a(M)
100 loops, best of 3: 11.5 ms per loop
%timeit is_diagonal(M)
100 loops, best of 3: 10.4 ms per loop
%timeit isDiag(M)
100 loops, best of 3: 12.5 ms per loop
Hmm, that’s slower, probably from constructing i and j
Let’s try to improve the @donkopotamus solution by removing the subtraction step:
def b(M):
return np.all(M == np.diag(np.diagonal(M)))
%timeit b(M)
100 loops, best of 3: 4.48 ms per loop
That’s a bit better.
EDIT2 I came up with an even faster method:
def isDiag2(M):
i, j = M.shape
assert i == j
test = M.reshape(-1)[:-1].reshape(i-1, j+1)
return ~np.any(test[:, 1:])
This isn’t doing any calculations, just reshaping. Turns out reshaping to +1 rows on a diagonal matrix puts all the data in the first column. You can then check a contiguous block for any nonzeros which is much fatser for numpy Let’s check times:
def Make42(m):
b = np.zeros(m.shape)
np.fill_diagonal(b, m.diagonal())
return np.all(m == b)
%timeit b(M)
%timeit Make42(M)
%timeit isDiag2(M)
100 loops, best of 3: 4.88 ms per loop
100 loops, best of 3: 5.73 ms per loop
1000 loops, best of 3: 1.84 ms per loop
Seems my original is faster than @Make42 for smaller sets
M = np.diag(np.random.randint(0,10,10000))
%timeit b(M)
%timeit Make42(M)
%timeit isDiag2(M)
The slowest run took 35.58 times longer than the fastest. This could mean that an intermediate result is being cached.
1 loop, best of 3: 335 ms per loop
<MemoryError trace removed>
10 loops, best of 3: 76.5 ms per loop
And @Make42 gives memory error on the larger set. But then I don’t seem to have as much RAM as they do.
I believe this is the most succinct way:
np.allclose(np.diag(np.diag(a)), a)
We can actually do quite a bit better than what Daniel F suggested:
import numpy as np
import time
a = np.diag(np.random.random(19999))
t1 = time.time()
np.all(a == np.diag(np.diagonal(a)))
print(time.time()-t1)
t1 = time.time()
b = np.zeros(a.shape)
np.fill_diagonal(b, a.diagonal())
np.all(a == b)
print(time.time()-t1)
results in
2.5737204551696777
0.6501829624176025
One tricks is that np.diagonal(a) actually uses a.diagonal(), so we use that one directly. But what takes the cake the the fast build of b, combined with the in-place operation on b.
quick and dirty way to get the truth. works in a reasonable amount of time
for i in range(0, len(matrix[0])):
for j in range(0, len(matrix[0])):
if ((i != j) and
(matrix[i][j] != 0)) :
return False
return True
import numpy as np
is_diagonal = (np.trace(mat) == np.sum(mat))
Appproach #1 : Using NumPy strides/np.lib.stride_tricks.as_strided
We can leverage NumPy strides to give us the off-diag elements as a view. So, no memory overhead there and virtually free runtime! This idea has been explored before in this post.
Thus, we have –
# https://stackoverflow.com/a/43761941/ @Divakar
def nodiag_view(a):
m = a.shape[0]
p,q = a.strides
return np.lib.stride_tricks.as_strided(a[:,1:], (m-1,m), (p+q,q))
Sample run to showcase its usage –
In [175]: a
Out[175]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
In [176]: nodiag_view(a)
Out[176]:
array([[ 1, 2, 3, 4],
[ 6, 7, 8, 9],
[11, 12, 13, 14]])
Let’s verify the free runtime and no memory overhead claims, by using it on a large array –
In [182]: a = np.zeros((10000,10000), dtype=int)
...: np.fill_diagonal(a,np.arange(len(a)))
In [183]: %timeit nodiag_view(a)
6.42 µs ± 48.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [184]: np.shares_memory(a, nodiag_view(a))
Out[184]: True
Now, how do we use it here? Simply check if all nodiag_view elements are 0s, signalling a diagonal matrix!
Hence, to solve our case here, for an input array a, it would be –
isdiag = (nodiag_view(a)==0).all()
Appproach #2 : Hacky way
For completeness, one hacky way would be to temporarily save diag elements, assign 0s there, check for all elements to be 0s. If so, signalling a diagonal matrix. Finally assign back the diag elements.
The implementation would be –
def hacky_way(a):
diag_elem = np.diag(a).copy()
np.fill_diagonal(a,0)
out = (a==0).all()
np.fill_diagonal(a,diag_elem)
return out
Benchmarking
Let’s time on a large array and see how these compare on performance –
In [3]: a = np.zeros((10000,10000), dtype=int)
...: np.fill_diagonal(a,np.arange(len(a)))
In [4]: %timeit (nodiag_view(a)==0).all()
52.3 ms ± 393 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [5]: %timeit hacky_way(a)
51.8 ms ± 250 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Other approaches from @Daniel F’s post that captured other approaches –
# @donkopotamus solution improved by @Daniel F
def b(M):
return np.all(M == np.diag(np.diagonal(M)))
# @Daniel F's soln without assert check
def isDiag2(M):
i, j = M.shape
test = M.reshape(-1)[:-1].reshape(i-1, j+1)
return ~np.any(test[:, 1:])
# @Make42's soln
def Make42(m):
b = np.zeros(m.shape)
np.fill_diagonal(b, m.diagonal())
return np.all(m == b)
Timings with same setup as earlier –
In [6]: %timeit b(a)
...: %timeit Make42(a)
...: %timeit isDiag2(a)
218 ms ± 1.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
302 ms ± 1.25 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
67.1 ms ± 1.35 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
The solutions nodiag_view and hacky_way by @Divakar and the solution isDiag2 by @Daniel F are efficient. The other solutions are pretty slow.
The accepted solution that is by @donkopotamus, which is easy to implement, is the second slowest and 7.55 times slower than the fastest answer.
from timeit import timeit
setup = '''
import numpy as np
d = np.zeros((10000,10000), dtype=int)
np.fill_diagonal(d,np.arange(len(d)))
def nodiag_view(a):
m = a.shape[0]
p,q = a.strides
return (np.lib.stride_tricks.as_strided(a[:,1:], (m-1,m), (p+q,q)) == 0).all()
def hacky_way(a):
diag_elem = np.diag(a).copy()
np.fill_diagonal(a,0)
out = (a==0).all()
np.fill_diagonal(a,diag_elem)
return out
def isDiag(M):
i, j = np.nonzero(M)
return np.all(i == j)
def isDiag2(M):
i, j = M.shape
assert i == j
test = M.reshape(-1)[:-1].reshape(i-1, j+1)
return ~np.any(test[:, 1:])
def Make42(m):
b = np.zeros(m.shape)
np.fill_diagonal(b, m.diagonal())
return np.all(m == b)
def by_donkopotamus(a):
return np.count_nonzero(a - np.diag(np.diag(a))) == 0
def by_liwt31(a):
np.allclose(np.diag(np.diag(a)), a)
'''
test0 = '''nodiag_view(d)'''
test1 = '''hacky_way(d)'''
test2 = '''isDiag(d)'''
test3 = '''isDiag2(d)'''
test4 = '''Make42(d)'''
test5 = '''by_donkopotamus(d)'''
test6 = '''by_liwt31(d)'''
print('test0:', timeit(test0, setup, number=100))
print('test1:', timeit(test1, setup, number=100))
print('test2:', timeit(test2, setup, number=100))
print('test3:', timeit(test3, setup, number=100))
print('test4:', timeit(test4, setup, number=100))
print('test5:', timeit(test5, setup, number=100))
print('test6:', timeit(test6, setup, number=100))
Results:
test0: 4.194842008990236
test1: 4.11843847198179
test2: 28.11888137299684
test3: 5.095675196003867
test4: 22.56097131301067
test5: 31.05823188900831
test6: 106.19386338599725
I have compute a very large matrix M with lots of degenerate eigenvectors(different eigenvectors with same eigenvalues). I use QR decomposition to make sure these eigenvectors are orthonormal, so the Q is the orthonormal eigenvectors of M, and Q^{-1}MQ = D, where D is diagonal matrix. Now I want to check if D is truly diagonal matrix, but when I print D, the matrix is too large to show all of them, so how can I know if it is truly diagonal matrix?
I afraid if this is the most efficient way of doing this, But the idea is to mask the diagonal elements and check if all the other elements are zero. I guess this is sufficient check to label a matrix as diagonal matrix.
So we create a dummy array with same size as input matrix, initialized with ones. and then replace the diagonal elements with zeros. Now we perform element wise multiplication of input matrix and dummy matrix. So here we replace the diagonal elements of input matrix with zero and leave the other elements as it is.
Now finally we check if there are any non zero elements.
def is_diagonal(matrix):
#create a dummy matrix
dummy_matrix = np.ones(matrix.shape, dtype=np.uint8)
# Fill the diagonal of dummy matrix with 0.
np.fill_diagonal(dummy_matrix, 0)
return np.count_nonzero(np.multiply(dummy_matrix, matrix)) == 0
diagonal_matrix = np.array([[3, 0, 0],
[0, 7, 0],
[0, 0, 4]])
print is_diagonal(diagonal_matrix)
>>> True
random_matrix = np.array([[3, 8, 0],
[1, 7, 8],
[5, 0, 4]])
print is_diagonal(random_matrix)
>>> False
Remove the diagonal and count the non zero elements:
np.count_nonzero(x - np.diag(np.diagonal(x)))
Not sure how fast this is compared to the others, but:
def isDiag(M):
i, j = np.nonzero(M)
return np.all(i == j)
EDIT Let’s time things:
M = np.random.randint(0, 10, 1000) * np.eye(1000)
def a(M): #donkopotamus solution
return np.count_nonzero(M - np.diag(np.diagonal(M)))
%timeit a(M)
100 loops, best of 3: 11.5 ms per loop
%timeit is_diagonal(M)
100 loops, best of 3: 10.4 ms per loop
%timeit isDiag(M)
100 loops, best of 3: 12.5 ms per loop
Hmm, that’s slower, probably from constructing i and j
Let’s try to improve the @donkopotamus solution by removing the subtraction step:
def b(M):
return np.all(M == np.diag(np.diagonal(M)))
%timeit b(M)
100 loops, best of 3: 4.48 ms per loop
That’s a bit better.
EDIT2 I came up with an even faster method:
def isDiag2(M):
i, j = M.shape
assert i == j
test = M.reshape(-1)[:-1].reshape(i-1, j+1)
return ~np.any(test[:, 1:])
This isn’t doing any calculations, just reshaping. Turns out reshaping to +1 rows on a diagonal matrix puts all the data in the first column. You can then check a contiguous block for any nonzeros which is much fatser for numpy Let’s check times:
def Make42(m):
b = np.zeros(m.shape)
np.fill_diagonal(b, m.diagonal())
return np.all(m == b)
%timeit b(M)
%timeit Make42(M)
%timeit isDiag2(M)
100 loops, best of 3: 4.88 ms per loop
100 loops, best of 3: 5.73 ms per loop
1000 loops, best of 3: 1.84 ms per loop
Seems my original is faster than @Make42 for smaller sets
M = np.diag(np.random.randint(0,10,10000))
%timeit b(M)
%timeit Make42(M)
%timeit isDiag2(M)
The slowest run took 35.58 times longer than the fastest. This could mean that an intermediate result is being cached.
1 loop, best of 3: 335 ms per loop
<MemoryError trace removed>
10 loops, best of 3: 76.5 ms per loop
And @Make42 gives memory error on the larger set. But then I don’t seem to have as much RAM as they do.
I believe this is the most succinct way:
np.allclose(np.diag(np.diag(a)), a)
We can actually do quite a bit better than what Daniel F suggested:
import numpy as np
import time
a = np.diag(np.random.random(19999))
t1 = time.time()
np.all(a == np.diag(np.diagonal(a)))
print(time.time()-t1)
t1 = time.time()
b = np.zeros(a.shape)
np.fill_diagonal(b, a.diagonal())
np.all(a == b)
print(time.time()-t1)
results in
2.5737204551696777
0.6501829624176025
One tricks is that np.diagonal(a) actually uses a.diagonal(), so we use that one directly. But what takes the cake the the fast build of b, combined with the in-place operation on b.
quick and dirty way to get the truth. works in a reasonable amount of time
for i in range(0, len(matrix[0])):
for j in range(0, len(matrix[0])):
if ((i != j) and
(matrix[i][j] != 0)) :
return False
return True
import numpy as np
is_diagonal = (np.trace(mat) == np.sum(mat))
Appproach #1 : Using NumPy strides/np.lib.stride_tricks.as_strided
We can leverage NumPy strides to give us the off-diag elements as a view. So, no memory overhead there and virtually free runtime! This idea has been explored before in this post.
Thus, we have –
# https://stackoverflow.com/a/43761941/ @Divakar
def nodiag_view(a):
m = a.shape[0]
p,q = a.strides
return np.lib.stride_tricks.as_strided(a[:,1:], (m-1,m), (p+q,q))
Sample run to showcase its usage –
In [175]: a
Out[175]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
In [176]: nodiag_view(a)
Out[176]:
array([[ 1, 2, 3, 4],
[ 6, 7, 8, 9],
[11, 12, 13, 14]])
Let’s verify the free runtime and no memory overhead claims, by using it on a large array –
In [182]: a = np.zeros((10000,10000), dtype=int)
...: np.fill_diagonal(a,np.arange(len(a)))
In [183]: %timeit nodiag_view(a)
6.42 µs ± 48.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [184]: np.shares_memory(a, nodiag_view(a))
Out[184]: True
Now, how do we use it here? Simply check if all nodiag_view elements are 0s, signalling a diagonal matrix!
Hence, to solve our case here, for an input array a, it would be –
isdiag = (nodiag_view(a)==0).all()
Appproach #2 : Hacky way
For completeness, one hacky way would be to temporarily save diag elements, assign 0s there, check for all elements to be 0s. If so, signalling a diagonal matrix. Finally assign back the diag elements.
The implementation would be –
def hacky_way(a):
diag_elem = np.diag(a).copy()
np.fill_diagonal(a,0)
out = (a==0).all()
np.fill_diagonal(a,diag_elem)
return out
Benchmarking
Let’s time on a large array and see how these compare on performance –
In [3]: a = np.zeros((10000,10000), dtype=int)
...: np.fill_diagonal(a,np.arange(len(a)))
In [4]: %timeit (nodiag_view(a)==0).all()
52.3 ms ± 393 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [5]: %timeit hacky_way(a)
51.8 ms ± 250 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Other approaches from @Daniel F’s post that captured other approaches –
# @donkopotamus solution improved by @Daniel F
def b(M):
return np.all(M == np.diag(np.diagonal(M)))
# @Daniel F's soln without assert check
def isDiag2(M):
i, j = M.shape
test = M.reshape(-1)[:-1].reshape(i-1, j+1)
return ~np.any(test[:, 1:])
# @Make42's soln
def Make42(m):
b = np.zeros(m.shape)
np.fill_diagonal(b, m.diagonal())
return np.all(m == b)
Timings with same setup as earlier –
In [6]: %timeit b(a)
...: %timeit Make42(a)
...: %timeit isDiag2(a)
218 ms ± 1.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
302 ms ± 1.25 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
67.1 ms ± 1.35 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
The solutions nodiag_view and hacky_way by @Divakar and the solution isDiag2 by @Daniel F are efficient. The other solutions are pretty slow.
The accepted solution that is by @donkopotamus, which is easy to implement, is the second slowest and 7.55 times slower than the fastest answer.
from timeit import timeit
setup = '''
import numpy as np
d = np.zeros((10000,10000), dtype=int)
np.fill_diagonal(d,np.arange(len(d)))
def nodiag_view(a):
m = a.shape[0]
p,q = a.strides
return (np.lib.stride_tricks.as_strided(a[:,1:], (m-1,m), (p+q,q)) == 0).all()
def hacky_way(a):
diag_elem = np.diag(a).copy()
np.fill_diagonal(a,0)
out = (a==0).all()
np.fill_diagonal(a,diag_elem)
return out
def isDiag(M):
i, j = np.nonzero(M)
return np.all(i == j)
def isDiag2(M):
i, j = M.shape
assert i == j
test = M.reshape(-1)[:-1].reshape(i-1, j+1)
return ~np.any(test[:, 1:])
def Make42(m):
b = np.zeros(m.shape)
np.fill_diagonal(b, m.diagonal())
return np.all(m == b)
def by_donkopotamus(a):
return np.count_nonzero(a - np.diag(np.diag(a))) == 0
def by_liwt31(a):
np.allclose(np.diag(np.diag(a)), a)
'''
test0 = '''nodiag_view(d)'''
test1 = '''hacky_way(d)'''
test2 = '''isDiag(d)'''
test3 = '''isDiag2(d)'''
test4 = '''Make42(d)'''
test5 = '''by_donkopotamus(d)'''
test6 = '''by_liwt31(d)'''
print('test0:', timeit(test0, setup, number=100))
print('test1:', timeit(test1, setup, number=100))
print('test2:', timeit(test2, setup, number=100))
print('test3:', timeit(test3, setup, number=100))
print('test4:', timeit(test4, setup, number=100))
print('test5:', timeit(test5, setup, number=100))
print('test6:', timeit(test6, setup, number=100))
Results:
test0: 4.194842008990236
test1: 4.11843847198179
test2: 28.11888137299684
test3: 5.095675196003867
test4: 22.56097131301067
test5: 31.05823188900831
test6: 106.19386338599725