Fast check for NaN in NumPy

I think np.isnan(np.min(X)) should do what you want.

Even there exist an accepted answer, I’ll like to demonstrate the following (with Python 2.7.2 and Numpy 1.6.0 on Vista):

In []: x= rand(1e5)
In []: %timeit isnan(x.min())
10000 loops, best of 3: 200 us per loop
In []: %timeit isnan(x.sum())
10000 loops, best of 3: 169 us per loop
In []: %timeit isnan(dot(x, x))
10000 loops, best of 3: 134 us per loop

In []: x[5e4]= NaN
In []: %timeit isnan(x.min())
100 loops, best of 3: 4.47 ms per loop
In []: %timeit isnan(x.sum())
100 loops, best of 3: 6.44 ms per loop
In []: %timeit isnan(dot(x, x))
10000 loops, best of 3: 138 us per loop

Thus, the really efficient way might be heavily dependent on the operating system. Anyway dot(.) based seems to be the most stable one.

Related to this is the question of how to find the first occurrence of NaN. This is the fastest way to handle that that I know of:

index = next((i for (i,n) in enumerate(iterable) if n!=n), None)

If you’re comfortable with numba it allows to create a fast short-circuit (stops as soon as a NaN is found) function:

import numba as nb
import math

@nb.njit
def anynan(array):
    array = array.ravel()
    for i in range(array.size):
        if math.isnan(array[i]):
            return True
    return False

If there is no NaN the function might actually be slower than np.min, I think that’s because np.min uses multiprocessing for large arrays:

import numpy as np
array = np.random.random(2000000)

%timeit anynan(array)          # 100 loops, best of 3: 2.21 ms per loop
%timeit np.isnan(array.sum())  # 100 loops, best of 3: 4.45 ms per loop
%timeit np.isnan(array.min())  # 1000 loops, best of 3: 1.64 ms per loop

But in case there is a NaN in the array, especially if it’s position is at low indices, then it’s much faster:

array = np.random.random(2000000)
array[100] = np.nan

%timeit anynan(array)          # 1000000 loops, best of 3: 1.93 µs per loop
%timeit np.isnan(array.sum())  # 100 loops, best of 3: 4.57 ms per loop
%timeit np.isnan(array.min())  # 1000 loops, best of 3: 1.65 ms per loop

Similar results may be achieved with Cython or a C extension, these are a bit more complicated (or easily avaiable as bottleneck.anynan) but ultimatly do the same as my anynan function.

There are two general approaches here:

• Check each array item for nan and take any.
• Apply some cumulative operation that preserves nans (like sum) and check its result.

While the first approach is certainly the cleanest, the heavy optimization of some of the cumulative operations (particularly the ones that are executed in BLAS, like dot) can make those quite fast. Note that dot, like some other BLAS operations, are multithreaded under certain conditions. This explains the difference in speed between different machines.

import numpy as np
import perfplot


def min(a):
    return np.isnan(np.min(a))


def sum(a):
    return np.isnan(np.sum(a))


def dot(a):
    return np.isnan(np.dot(a, a))


def any(a):
    return np.any(np.isnan(a))


def einsum(a):
    return np.isnan(np.einsum("i->", a))


b = perfplot.bench(
    setup=np.random.rand,
    kernels=[min, sum, dot, any, einsum],
    n_range=[2 ** k for k in range(25)],
    xlabel="len(a)",
)
b.save("out.png")
b.show()

1. use .any()

   if numpy.isnan(myarray).any()

2. numpy.isfinite maybe better than isnan for checking

   if not np.isfinite(prop).all()

Adding to @nico-schlömer and @mseifert ‘s answers, I computed the performance of a numba-test has_nan with early stops, compared to some of the functions that will parse the full array.

On my machine, for an array without nans, the break-even happens for ~10^4 elements.

import perfplot
import numpy as np
import numba
import math

def min(a):
    return np.isnan(np.min(a))

def dot(a):
    return np.isnan(np.dot(a, a))

def einsum(a):
    return np.isnan(np.einsum("i->", a))

@numba.njit
def has_nan(a):
    for i in range(a.size - 1):
        if math.isnan(a[i]):
            return True
    return False


def array_with_missing_values(n, p):
    """ Return array of size n,  p : nans ( % of array length )
    Ex : n=1e6, p=1 : 1e4 nan assigned at random positions """
    a = np.random.rand(n)
    p = np.random.randint(0, len(a), int(p*len(a)/100))
    a[p] = np.nan
    return a


#%%
perfplot.show(
    setup=lambda n: array_with_missing_values(n, 0),
    kernels=[min, dot, has_nan],
    n_range=[2 ** k for k in range(20)],
    logx=True,
    logy=True,
    xlabel="len(a)",
)

What happens if the array has nans ? I investigated the impact of the nan-coverage of the array.

For arrays of length 1,000,000, has_nan becomes a better option is there are ~10^-3 % nans (so ~10 nans) in the array.

#%%
N = 1000000  # 100000
perfplot.show(
    setup=lambda p: array_with_missing_values(N, p),
    kernels=[min, dot, has_nan],
    n_range=np.array([2 ** k for k in range(20)]) / 2**20 * 0.01, 
    logy=True,
    xlabel=f"% of nan in array (N = {N})",
)

If in your application most arrays have nan and you’re looking for ones without, then has_nan is the best approach.
Else; dot seems to be the best option.