Overflow Error in Python's numpy.exp function
Question:
I want to use numpy.exp like this:
cc = np.array([
[0.120,0.34,-1234.1]
])
print 1/(1+np.exp(-cc))
But this gives me error:
/usr/local/lib/python2.7/site-packages/ipykernel/__main__.py:5: RuntimeWarning: overflow encountered in exp
I can’t understand why? How can I fix this? It seems the problem is with third number (-1234.1)
Answers:
The largest value representable by a numpy float is 1.7976931348623157e+308, whose logarithm is about 709.782, so there is no way to represent np.exp(1234.1).
In [1]: import numpy as np
In [2]: np.finfo('d').max
Out[2]: 1.7976931348623157e+308
In [3]: np.log(_)
Out[3]: 709.78271289338397
In [4]: np.exp(709)
Out[4]: 8.2184074615549724e+307
In [5]: np.exp(710)
/usr/local/bin/ipython:1: RuntimeWarning: overflow encountered in exp
#!/usr/local/bin/python3.5
Out[5]: inf
As fuglede says, the issue here is that np.float64 can’t handle a number as large as exp(1234.1). Try using np.float128 instead:
>>> cc = np.array([[0.120,0.34,-1234.1]], dtype=np.float128)
>>> cc
array([[ 0.12, 0.34, -1234.1]], dtype=float128)
>>> 1 / (1 + np.exp(-cc))
array([[ 0.52996405, 0.58419052, 1.0893812e-536]], dtype=float128)
Note however, that there are certain quirks with using extended precision. It may not work on Windows; you don’t actually get the full 128 bits of precision; and you might lose the precision whenever the number passes through pure python. You can read more about the details here.
For most practical purposes, you can probably approximate 1 / (1 + <a large number>) to zero. That is to say, just ignore the warning and move on. Numpy takes care of the approximation for you (when using np.float64):
>>> 1 / (1 + np.exp(-cc))
/usr/local/bin/ipython3:1: RuntimeWarning: overflow encountered in exp
#!/usr/local/bin/python3.4
array([[ 0.52996405, 0.58419052, 0. ]])
If you want to suppress the warning, you could use scipy.special.expit, as suggested by WarrenWeckesser in a comment to the question:
>>> from scipy.special import expit
>>> expit(cc)
array([[ 0.52996405, 0.58419052, 0. ]])
A possible solution is to use the decimal module, which lets you work with arbitrary precision floats. Here is an example where a numpy array of floats with 100 digits precision is used:
import numpy as np
import decimal
# Precision to use
decimal.getcontext().prec = 100
# Original array
cc = np.array(
[0.120,0.34,-1234.1]
)
# Fails
print(1/(1 + np.exp(-cc)))
# New array with the specified precision
ccd = np.asarray([decimal.Decimal(el) for el in cc], dtype=object)
# Works!
print(1/(1 + np.exp(-ccd)))
exp(-1234.1) is too small for 32bit or 64bit floating-point numbers. Since it cannot be represented, numpy produces the correct warning.
Using IEEE 754 32bit floating-point numbers, the smallest positive number it can represent is 2^(-149), which is roughly 1e-45.
If you use IEEE 754 64 bit floating-point numbers, the smallest positive number is 2^(-1074) which is roughy 1e-327.
In either case, it cannot represent a number as small as exp(-1234.1) which is about 1e-535.
You should be using the expit function from scipy to compute the sigmoid function. This would give you better precision.
For practical purposes, exp(-1234.1) is a very small number. If rounding to zero makes sense in your use case, numpy produces benign results by rounding it to zero.
If you don’t care about precision, you can use numpy.clip.
In float64:
cc = np.clip(cc, -709.78, 709.78)
In float32:
cc = np.clip(cc, -88.72, 88.72)
As mentioned earlier by Praveen, you can use expit from scipy
So the problem can be solved by using: 1 / (1+ exp(-x)) = exp(x) / (1+exp(x))
>>> import numpy as np
>>> cc = np.array([[0.120,0.34,-1234.1]])
>>> np.exp(cc) / (1 + np.exp(cc))
array([[0.52996405, 0.58419052, 0. ]])
I want to use numpy.exp like this:
cc = np.array([
[0.120,0.34,-1234.1]
])
print 1/(1+np.exp(-cc))
But this gives me error:
/usr/local/lib/python2.7/site-packages/ipykernel/__main__.py:5: RuntimeWarning: overflow encountered in exp
I can’t understand why? How can I fix this? It seems the problem is with third number (-1234.1)
The largest value representable by a numpy float is 1.7976931348623157e+308, whose logarithm is about 709.782, so there is no way to represent np.exp(1234.1).
In [1]: import numpy as np
In [2]: np.finfo('d').max
Out[2]: 1.7976931348623157e+308
In [3]: np.log(_)
Out[3]: 709.78271289338397
In [4]: np.exp(709)
Out[4]: 8.2184074615549724e+307
In [5]: np.exp(710)
/usr/local/bin/ipython:1: RuntimeWarning: overflow encountered in exp
#!/usr/local/bin/python3.5
Out[5]: inf
As fuglede says, the issue here is that np.float64 can’t handle a number as large as exp(1234.1). Try using np.float128 instead:
>>> cc = np.array([[0.120,0.34,-1234.1]], dtype=np.float128)
>>> cc
array([[ 0.12, 0.34, -1234.1]], dtype=float128)
>>> 1 / (1 + np.exp(-cc))
array([[ 0.52996405, 0.58419052, 1.0893812e-536]], dtype=float128)
Note however, that there are certain quirks with using extended precision. It may not work on Windows; you don’t actually get the full 128 bits of precision; and you might lose the precision whenever the number passes through pure python. You can read more about the details here.
For most practical purposes, you can probably approximate 1 / (1 + <a large number>) to zero. That is to say, just ignore the warning and move on. Numpy takes care of the approximation for you (when using np.float64):
>>> 1 / (1 + np.exp(-cc))
/usr/local/bin/ipython3:1: RuntimeWarning: overflow encountered in exp
#!/usr/local/bin/python3.4
array([[ 0.52996405, 0.58419052, 0. ]])
If you want to suppress the warning, you could use scipy.special.expit, as suggested by WarrenWeckesser in a comment to the question:
>>> from scipy.special import expit
>>> expit(cc)
array([[ 0.52996405, 0.58419052, 0. ]])
A possible solution is to use the decimal module, which lets you work with arbitrary precision floats. Here is an example where a numpy array of floats with 100 digits precision is used:
import numpy as np
import decimal
# Precision to use
decimal.getcontext().prec = 100
# Original array
cc = np.array(
[0.120,0.34,-1234.1]
)
# Fails
print(1/(1 + np.exp(-cc)))
# New array with the specified precision
ccd = np.asarray([decimal.Decimal(el) for el in cc], dtype=object)
# Works!
print(1/(1 + np.exp(-ccd)))
exp(-1234.1) is too small for 32bit or 64bit floating-point numbers. Since it cannot be represented, numpy produces the correct warning.
Using IEEE 754 32bit floating-point numbers, the smallest positive number it can represent is 2^(-149), which is roughly 1e-45.
If you use IEEE 754 64 bit floating-point numbers, the smallest positive number is 2^(-1074) which is roughy 1e-327.
In either case, it cannot represent a number as small as exp(-1234.1) which is about 1e-535.
You should be using the expit function from scipy to compute the sigmoid function. This would give you better precision.
For practical purposes, exp(-1234.1) is a very small number. If rounding to zero makes sense in your use case, numpy produces benign results by rounding it to zero.
If you don’t care about precision, you can use numpy.clip.
In float64:
cc = np.clip(cc, -709.78, 709.78)
In float32:
cc = np.clip(cc, -88.72, 88.72)
As mentioned earlier by Praveen, you can use expit from scipy
So the problem can be solved by using: 1 / (1+ exp(-x)) = exp(x) / (1+exp(x))
>>> import numpy as np
>>> cc = np.array([[0.120,0.34,-1234.1]])
>>> np.exp(cc) / (1 + np.exp(cc))
array([[0.52996405, 0.58419052, 0. ]])