How to fix 'The array returned by a function changed size between calls' while using lmfit for minimization?

Question

How can I fix code with error ‘The array returned by a function changed size between calls’ while using lmfit for minimization?

Please find my code below:

import numpy as np
import pandas as pd
import lmfit as lf

#model needs to be fitted
x0 = 75
def func(params, x, Tsky):
    A = params['amp']
    w = params['width']
    t = params['thickness']
    v0 = params['mid_freq']
    b0 = params['b0']
    b1 = params['b1']
    b2 = params['b2']
    b3 = params['b3']
    b4 = params['b4']
    B = (4 * (x - v0)**2. / w**2.) * np.log(-1./t * np.log((1 + np.exp(-t))/2))
    T21 = -A * (1 - np.exp(-t * np.exp(B)))/(1 - np.exp(-t))
    model = T21 + b0 * ((x/x0)**(-2.5 + b1 + b2 * np.log(x/x0))) * np.exp(-b3*(x/x0)**-2.) + b4 *  (x/x0)**-2.
    return (Tsky-model)

#read the data
df = pd.read_csv('figure1_plotdata.csv')
data_list = df.T.values.tolist()
xdata = np.array(data_list[0])
Tsky = np.array(data_list[2])

#initial value of the parameters
params = lf.Parameters()
params.add('amp', value=0.2)
params.add('width', value=10)
params.add('thickness', value=5)
params.add('mid_freq', value=70)
params.add('b0', value=500)
params.add('b1', value=-0.5)
params.add('b2', value=-0.5)
params.add('b3', value=-0.5)
params.add('b4', value=500)

#minimize the function
out = lf.minimize(func, params, args=(xdata, Tsky), method='leastsq', kws=None, iter_cb=None, scale_covar=True, nan_policy='omit', calc_covar=True)

print(lf.fit_report(out))

Here is the error message:

File "/home/ankita/Dropbox/Python/Bowman_work/min.py", line 81, in <module>
    out = lf.minimize(func, params, args=(xdata, Tsky), method='leastsq', kws=None, iter_cb=None, scale_covar=True, nan_policy='omit', calc_covar=True)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/lmfit-0.9.13-py3.7.egg/lmfit/minimizer.py", line 2300, in minimize
    return fitter.minimize(method=method)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/lmfit-0.9.13-py3.7.egg/lmfit/minimizer.py", line 1949, in minimize
    return function(**kwargs)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/lmfit-0.9.13-py3.7.egg/lmfit/minimizer.py", line 1492, in leastsq
    lsout = scipy_leastsq(self.__residual, variables, **lskws)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 394, in leastsq
    gtol, maxfev, epsfcn, factor, diag)

**ValueError: The array returned by a function changed size between calls**

Asked By: Ankita Bera

||

Source

Answer 1

If you had used

out = lf.minimize(func, params,...,nan_policy='raise')

you would have seen an exception being raised telling you that there are NaNs. When you use nan_policy='omit', any such NaNs that are generated by your model are removed from the residual array, and so the size of the array changes between calls. The fitting cannot handle NaNs or changes to the size of the arrays — you have to eliminate them.

In particular, np.log(x) is NaN when x<0. You have np.log() with a complicated argument that depends on the value of the parameter t. If that argument goes below 0 for some value of t, the model has NaNs and makes no sense.
You will have to ensure that this argument cannot be below 0. It might be that
use

params.add('thickness', value=5, min=0)

is sufficient. But you should examine your model in more detail and determine if that makes sense.

Your model looks quite complicated to me. I cannot guess where such a model derives from. Taking multiple np.exp() and np.log() is sort of asking for numerical instabilities. So, I don’t know that simply forcing t to be positive will give a good fit, but it might point you in the right direction.

Answered By: M Newville

Answer 2

Just in case you arrived at this question and your not using lf.minimize: in my case, the problem was that I was forgetting to add a variable as array when using np.vectorize.

My code was:

my_fn = np.vectorize(lambda x1, x2, x3: _my_fn_(a, m, n, c_m,
                                                 a_qv, b_qv, r_wb, x4,
                                                 x1, x2, x3))

return my_fn(x1, x2, x3)

But x4 was an array, so I needed another parameter:

my_fn = np.vectorize(lambda x1, x2, x3, x4: _my_fn_(a, m, n, c_m,
                                                 a_qv, b_qv, r_wb, x4,
                                                 x1, x2, x3))

return my_fn(x1, x2, x3, x4)

Answered By: Esteban

How to fix 'The array returned by a function changed size between calls' while using lmfit for minimization?

Question:

Answers: