Python: sort function breaks in the presence of nan

IEEE754 is the standard that defines floating point operations in this instance. This standard defines the compare operation of operands, at least one of which is a NaN, to be an error. Hence, this is not a bug. You need to deal with the NaNs before operating on your array.

I’m not sure about the bug, but the workaround may be the following:

sorted(
    (2, 1, float('nan')),
    lambda x,y: x is float('nan') and -1 
                or (y is float('nan') and 1
                or cmp(x,y)))

which results in:

('nan', 1, 2)

Or remove nans before sorting or anything else.

The problem is that there’s no correct order if the list contains a NAN, since a sequence a1, a2, a3, ..., an is sorted if a1 <= a2 <= a3 <= ... <= an. If any of these a values is a NAN then the sorted property breaks, since for all a, a <= NAN and NAN <= a are both false.

Assuming you want to keep the NaNs and order them as the lowest “values”, here is a workaround working both with non-unique nan, unique numpy nan, numerical and non numerical objects:

def is_nan(x):
    return (x is np.nan or x != x)

list_ = [2, float('nan'), 'z', 1, 'a', np.nan, 4, float('nan')]
sorted(list_, key = lambda x : float('-inf') if is_nan(x) else x)
# [nan, nan, nan, 1, 2, 4, 'a', 'z']

Regardless of standards, there are many cases where a user-defined ordering of float and NA values is useful. For instance, I was sorting stock returns and wanted highest to lowest with NA last (since those were irrelevant). There are 4 possible combinations

1. Ascending float values, NA values last
2. Ascending float values, NA values first
3. Descending float values, NA values last
4. Descending float values, NA values first

Here is a function that covers all scenarios by conditionally replacing NA values with +/- inf

import math 

def sort_with_na(x, reverse=False, na_last=True):
    """Intelligently sort iterable with NA values

    For reliable behavior with NA values, we should change the NAs to +/- inf
    to guarantee their order rather than relying on the built-in
    ``sorted(reverse=True)`` which will have no effect. To use the ``reverse``
    parameter or other kwargs, use functools.partial in your lambda i.e.

        sorted(iterable, key=partial(sort_with_na, reverse=True, na_last=False))

    :param x: Element to be sorted
    :param bool na_last: Whether NA values should come last or first
    :param bool reverse: Return ascending if ``False`` else descending
    :return bool:
    """
    if not math.isnan(x):
        return -x if reverse else x
    else:
        return float('inf') if na_last else float('-inf')

Testing out each of the 4 combinations

from functools import partial

a = [2, float('nan'), 1]
sorted(a, key=sort_with_na)                                         # Default
sorted(a, key=partial(sort_with_na, reverse=False, na_last=True))   # Ascend, NA last
sorted(a, key=partial(sort_with_na, reverse=False, na_last=False))  # Ascend, NA first
sorted(a, key=partial(sort_with_na, reverse=True, na_last=True))    # Descend, NA last
sorted(a, key=partial(sort_with_na, reverse=True, na_last=False))   # Descend, NA first

A resilient sort involves a compare of 2 items and returning: less, equal, greater.

If cmp(a,b) is "greater", then cmp(b,a) must be "less".

If cmp(a,b) is "zero", then cmp(b,a) must be "zero".

What is missing in answers to date is the case of comparing 2 floats which are both NANs and preserving the above properties. 2 NANs should compare as equal or perhaps based on some consistent interpretation of their payloads.

Alternate compare algorithm to put all NAN > +inf

if isnan(a)
  if isnan(b)
    return 0 (or maybe compare payloads/bit patterns)
  return 1
if isnan(b) return 1
if a > b return 1
if a < b return -1
return 0

To recap the problems:

NaN

This always returns False for every comparison, so it stays where it is in the list:

>>> sorted([float('nan'), 0])
[nan, 0]
>>> sorted([0, float('nan')])
[0, nan]

-0.0

This is == to 0.0, but has a different repr, a different json representation, and slightly different numerical properties. It has the same problem that the positive and negative zeros will stay in the same order they were in the original list:

>>> sorted([0.0, -0.0])
[0.0, -0.0]
>>> sorted([-0.0, 0.0])
[-0.0, 0.0]

Other solution?

@khachik’s solution has inconsistent sorting behavior for NaN and -inf

>>> key=lambda x: float('-inf') if math.isnan(x) else x
>>> sorted([float('nan'), float('-inf')], key=key)
[nan, -inf]
>>> sorted([float('-inf'), float('nan')], key=key)
[-inf, nan]

Solution: more complex key function.

So there are problems with signs and nans. We can just include those in a key function:

def stable_float_sort_key(x: float):
    return math.copysign(1, x), math.isnan(x), x

That works on all the examples above:

>>> sorted([float('nan'), 0.0], key=stable_float_sort_key)
[0.0, nan]
>>> sorted([0.0, float('nan')], key=stable_float_sort_key)
[0.0, nan]
>>> sorted([float('nan'), float('-inf')], key=stable_float_sort_key)
[-inf, nan]
>>> sorted([float('-inf'), float('nan')], key=stable_float_sort_key)
[-inf, nan]
>>> sorted([0.0, -0.0], key=stable_float_sort_key)
[-0.0, 0.0]
>>> sorted([-0.0, 0.0], key=stable_float_sort_key)
[-0.0, 0.0]

Indeed, you can write a hypothesis test showing it’s consistent across all floating point numbers:

import json
from hypothesis import given, settings
from hypothesis import strategies as st

@given(nums=st.lists(st.floats()), random=st.randoms())
@settings(max_examples=10000)
def test_stable_json_sorting(nums, random):
    shuffled = list(nums)
    random.shuffle(shuffled)
    l1 = sorted(nums, key=stable_float_sort_key)
    l2 = sorted(shuffled, key=stable_float_sort_key)
    assert json.dumps(l1) == json.dumps(l2)

It does have some oddities however, since some NaN’s are negative! For example:

>>> sorted([float('nan'), -0.0, 0.0, float('-nan')], key=stable_float_sort_key)
[-0.0, nan, 0.0, nan]

If that bothers you, you can fix this by switching the ordering:

def stable_float_sort_key(x: float):
    return math.isnan(x), math.copysign(1, x), x

That sorts negative numbers first, followed by positive numbers, followed by NaNs.

Does any of this make sense?

Of course other answerers are correct that in some sense none of this makes sense. Comparison of NaNs is a conceptual error of some kind. However, even in cases where the question doesn’t "make sense", you may want to have invariants like serializing sets of floating point numbers generated by the same code to exactly the same JSON representation, despite hash randomization (my use case). That’s more of a formal property of python code, rather than something where there’s a "right answer" per an IEEE standard.