Deal with overflow in exp using numpy
Question:
Using numpy, I have this definition of a function:
def powellBadlyScaled(X):
f1 = 10**4 * X[0] * X[1] - 1
f2 = numpy.exp(-numpy.float(X[0])) + numpy.exp(-numpy.float(X[1])) - 1.0001
return f1 + f2
This function is evaluated a huge number of times on an optimization routine. It often raises exception:
RuntimeWarning: overflow encountered in exp
I understand that operand cannot be stored in allocated space for a float. But how can I overcome the problem?
Answers:
You can use the bigfloat package. It supports arbitrary precision floating point operations.
http://packages.python.org/bigfloat/
import bigfloat
bigfloat.exp(5000,bigfloat.precision(100))
# -> BigFloat.exact('2.9676283840236670689662968052896e+2171', precision=100)
Are you using a function optimization framework? They usually implement value boundaries (using penalty terms). Try that. Are the relevant values really that extreme? In optimization it’s not uncommon to minimize log(f). (approximate log likelihood etc etc). Are you sure you want to optimize on that exp value and not log(exp(f)) == f. ?
Have a look at my answer to this question: logit and inverse logit functions for extreme values
Btw, if all you do is minimize powellBadlyScaled(x,y) then the minimum is at x -> + inf and y -> + inf, so no need for numerics.
Maybe you can improve your algorithm by checking for which areas you get warnings (it will probably bellow certain values for X[ 0 ],X[ 1 ]), and replacing the result with a really big number. You need to see how your function behaves, I thing you should check e.g. exp(-x)+exp(-y)+x*y
You can use numpy.seterr to control how numpy behaves in this circumstance: http://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html
You can also use the warnings module to control how warnings are or are not presented: http://docs.python.org/library/warnings.html
Depending on your specific needs, it may be useful to crop the input argument to exp(). If you actually want to get an inf out if it overflows or you want to get absurdly huge numbers, then other answers will be more appropriate.
def powellBadlyScaled(X):
f1 = 10**4 * X[0] * X[1] - 1
f2 = numpy.exp(-numpy.float(X[0])) + numpy.exp(-numpy.float(X[1])) - 1.0001
return f1 + f2
def powellBadlyScaled2(X):
f1 = 10**4 * X[0] * X[1] - 1
arg1 = -numpy.float(X[0])
arg2 = -numpy.float(X[1])
too_big = log(sys.float_info.max / 1000.0) # The 1000.0 puts a margin in to avoid overflow later
too_small = log(sys.float_info.min * 1000.0)
arg1 = max([min([arg1, too_big]), too_small])
arg2 = max([min([arg2, too_big]), too_small])
# print(' too_small = {}, too_big = {}'.format(too_small, too_big)) # Uncomment if you're curious
f2 = numpy.exp(arg1) + numpy.exp(arg2) - 1.0001
return f1 + f2
print('nTest against overflow: ------------')
x = [-1e5, 0]
print('powellBadlyScaled({}) = {}'.format(x, powellBadlyScaled(x)))
print('powellBadlyScaled2({}) = {}'.format(x, powellBadlyScaled2(x)))
print('nTest against underflow: ------------')
x = [0, 1e20]
print('powellBadlyScaled({}) = {}'.format(x, powellBadlyScaled(x)))
print('powellBadlyScaled2({}) = {}'.format(x, powellBadlyScaled2(x)))
Result:
Test against overflow: ------------
*** overflow encountered in exp
powellBadlyScaled([-100000.0, 0]) = inf
powellBadlyScaled2([-100000.0, 0]) = 1.79769313486e+305
Test against underflow: ------------
*** underflow encountered in exp
powellBadlyScaled([0, 1e+20]) = -1.0001
powellBadlyScaled2([0, 1e+20]) = -1.0001
Notice that powellBadlyScaled2 didn’t over/underflow when the original powellBadlyScaled did, but the modified version gives 1.79769313486e+305 instead of inf in one of the tests. I imagine there are plenty of applications where 1.79769313486e+305 is practically inf and this would be fine, or even preferred because 1.79769313486e+305 is a real number and inf is not.
Try scipy’s –
scipy.special.expit(x).
I had the same problem. If precision can be ignored to some degree, use np.round(my_float_array, decimals=<a smaller number>) to overcome this runtime warning.
Using numpy, I have this definition of a function:
def powellBadlyScaled(X):
f1 = 10**4 * X[0] * X[1] - 1
f2 = numpy.exp(-numpy.float(X[0])) + numpy.exp(-numpy.float(X[1])) - 1.0001
return f1 + f2
This function is evaluated a huge number of times on an optimization routine. It often raises exception:
RuntimeWarning: overflow encountered in exp
I understand that operand cannot be stored in allocated space for a float. But how can I overcome the problem?
You can use the bigfloat package. It supports arbitrary precision floating point operations.
http://packages.python.org/bigfloat/
import bigfloat
bigfloat.exp(5000,bigfloat.precision(100))
# -> BigFloat.exact('2.9676283840236670689662968052896e+2171', precision=100)
Are you using a function optimization framework? They usually implement value boundaries (using penalty terms). Try that. Are the relevant values really that extreme? In optimization it’s not uncommon to minimize log(f). (approximate log likelihood etc etc). Are you sure you want to optimize on that exp value and not log(exp(f)) == f. ?
Have a look at my answer to this question: logit and inverse logit functions for extreme values
Btw, if all you do is minimize powellBadlyScaled(x,y) then the minimum is at x -> + inf and y -> + inf, so no need for numerics.
Maybe you can improve your algorithm by checking for which areas you get warnings (it will probably bellow certain values for X[ 0 ],X[ 1 ]), and replacing the result with a really big number. You need to see how your function behaves, I thing you should check e.g. exp(-x)+exp(-y)+x*y
You can use numpy.seterr to control how numpy behaves in this circumstance: http://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html
You can also use the warnings module to control how warnings are or are not presented: http://docs.python.org/library/warnings.html
Depending on your specific needs, it may be useful to crop the input argument to exp(). If you actually want to get an inf out if it overflows or you want to get absurdly huge numbers, then other answers will be more appropriate.
def powellBadlyScaled(X):
f1 = 10**4 * X[0] * X[1] - 1
f2 = numpy.exp(-numpy.float(X[0])) + numpy.exp(-numpy.float(X[1])) - 1.0001
return f1 + f2
def powellBadlyScaled2(X):
f1 = 10**4 * X[0] * X[1] - 1
arg1 = -numpy.float(X[0])
arg2 = -numpy.float(X[1])
too_big = log(sys.float_info.max / 1000.0) # The 1000.0 puts a margin in to avoid overflow later
too_small = log(sys.float_info.min * 1000.0)
arg1 = max([min([arg1, too_big]), too_small])
arg2 = max([min([arg2, too_big]), too_small])
# print(' too_small = {}, too_big = {}'.format(too_small, too_big)) # Uncomment if you're curious
f2 = numpy.exp(arg1) + numpy.exp(arg2) - 1.0001
return f1 + f2
print('nTest against overflow: ------------')
x = [-1e5, 0]
print('powellBadlyScaled({}) = {}'.format(x, powellBadlyScaled(x)))
print('powellBadlyScaled2({}) = {}'.format(x, powellBadlyScaled2(x)))
print('nTest against underflow: ------------')
x = [0, 1e20]
print('powellBadlyScaled({}) = {}'.format(x, powellBadlyScaled(x)))
print('powellBadlyScaled2({}) = {}'.format(x, powellBadlyScaled2(x)))
Result:
Test against overflow: ------------
*** overflow encountered in exp
powellBadlyScaled([-100000.0, 0]) = inf
powellBadlyScaled2([-100000.0, 0]) = 1.79769313486e+305
Test against underflow: ------------
*** underflow encountered in exp
powellBadlyScaled([0, 1e+20]) = -1.0001
powellBadlyScaled2([0, 1e+20]) = -1.0001
Notice that powellBadlyScaled2 didn’t over/underflow when the original powellBadlyScaled did, but the modified version gives 1.79769313486e+305 instead of inf in one of the tests. I imagine there are plenty of applications where 1.79769313486e+305 is practically inf and this would be fine, or even preferred because 1.79769313486e+305 is a real number and inf is not.
Try scipy’s –
scipy.special.expit(x).
I had the same problem. If precision can be ignored to some degree, use np.round(my_float_array, decimals=<a smaller number>) to overcome this runtime warning.