Need of coercing array_like input data?

[jcjaskula-aws]

Hello,

Input data (tuples, lists and pd.Series) is now transformed into numpy.array and returned back in their new type in ModelResult.data. Recasting pandas.Series in numpy.array results in a loss of convenient functionalities.

y=pd.Series(range(8), pd.MultiIndex.from_product([[1,1,2,3], ["a", "b"]], names=["digits", "letters"]))
m=lmfit.models.LinearModel().fit(y, x=[1,1,1,1,2,2,3,3])
m.eval(x=m.data.index.unique(level="digits"))

This was not the case before since this commit when it was very convenient to use pandas functionalities directly on the fitted data.

What's the reason behind this? Is keeping this constraint mandatory?

[newville]

Post a minimal, complete example that demonstrates what you are seeing, and be explicit about what you expected to see and what you actually see. Am I supposed to know what you think is wrong? I do not.

Yes, some things changed with 1.2.0 for how data is coerced into numpy arrays.

I think it is not correct that we change the data you passed in: that is, your y should not be changed and still be your Series.

Internally, data and weights for a model are converted to numpy arrays of type float64. Independent data is of type list, tuple, pandas.Series can easily be interpreted as "really meant to be a float64 array" and is converted. Independent data of other data types, especially data that is ndarrays of different precision, or dicts, are not converted.

[jcjaskula-aws]

I agree with your statement that data passed in should not be changed. My post here is to report my experience that they are modified and I would like to change this behavior.

Let me simplify and be more explicit.

import pandas as pd
import lmfit
y=pd.Series(range(8), pd.MultiIndex.from_product([[1,1,2,3], ["a", "b"]], names=["digits", "letters"]))
y
digits  letters
1       a          0
        b          1
        a          2
        b          3
2       a          4
        b          5
3       a          6
        b          7
dtype: int64

is a pandas.Series with multiple indices. If I run y.index.unique(level="digits"), I get a list of unique first indices: Int64Index([1, 2, 3], dtype='int64', name='digits').

if I understand correctly, when I run

m=lmfit.models.LinearModel().fit(y, x=[1,1,1,1,2,2,3,3])

y and m.data should the the same data.

However, I get the following error with m.data.index.unique(level="digits"):

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 3
      1 y=pd.Series(range(8), pd.MultiIndex.from_product([[1,1,2,3], ["a", "b"]], names=["digits", "letters"]))
      2 m=lmfit.models.LinearModel().fit(y, x=[1,1,1,1,2,2,3,3])
----> 3 m.eval(x=m.data.index.unique(level="digits"))

AttributeError: 'numpy.ndarray' object has no attribute 'index'

This is just a complicated way to say that type(m.data) is different from type(y).
I would advocate to insure that ModelResult.data is not modified, even though lmfit needs an internal numpy.array copy.

MVE:

import pandas as pd
import lmfit
y=pd.Series(range(8))
m=lmfit.models.LinearModel().fit(y, x=list(range(8)))
assert type(y) == type(m.data)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[7], line 5
      3 y=pd.Series(range(8))
      4 m=lmfit.models.LinearModel().fit(y, x=list(range(8)))
----> 5 assert type(y) == type(m.data)

AssertionError: 

[newville]

@jcjaskula-aws Well, ModelResult.data is array-like. It is not Series-like. That is what the documentation says.

The changes we made are not API changes. The call structures for the methods were not changed in a backward-incompatible way. The internals of what Model does with the inputs did change. That can happen on every change and is not an API change.

It seems perfectly reasonable to me to say that trying to support pandas.Series has proven to be more trouble than it is worth, as the pandas users then expect that their data structure is driving the code is or is fully supported. There are other "array-like" structures, such as xarray and dask that we do not test. Though it might seem obvious how to convert these (or Series) into the ndarray we actually use, it is also reasonable to "refuse to guess" and to not automatically convert any data to ndarrays, but to raise a TypeError exception if the input data is not a float64 ndarray. Honestly, I do not think that is necessary, but this discussion topic is definitely leading me in that direction.

[rayosborn]

Since I am not a maintainer of lmfit, I would certainly defer to the judgement of @newville and @reneeotten about what they should and should not support. However, I think what @jcjaskula-aws asks for could have been accomplished by using the np.array function on the y argument instead of coercing it directly.

>>>  y=pd.Series(range(8), pd.MultiIndex.from_product([[1,1,2,3], ["a", "b"]], names=["digits", "letters"]))
>>> np.array(y).astype(np.float64)
array([0., 1., 2., ..., 5., 6., 7.])

I have checked that this also works for xarray, dask, and NXfield objects, the latter defined by the nexusformat package (which is why I am interested). The only requirement is that the array format has to have a __array__ function, which NumPy uses to return a valid ndarray, and clearly this is fairly routinely implemented if these packages are anything to go by. As @jcjaskula-aws suggests, this would need a private coerced copy of the y data within lmfit, but it doesn't involve any guess-work about what these different array formats do.

I can see arguments for not doing this, since it could add to future maintainability problems, and sorry for not suggesting this when the original data coercion debate was underway. In the case of nexusformat, it is just as simple to pass the NXfield.nxvalue property to lmfit to get the same result, so I'm not too bothered either way.

Edit: I should probably have checked the lmfit code before commenting. Perhaps you already use the np.array function to coerce the input, in which case the only debate is whether you should keep the coerced values in a private attribute.

[newville]

@rayosborn Yes we use np.asfarray(x) on the object passed in as data. I think the only real change is that we do it early on the data (and weights) array, and then on the result of calling the model function. FWIW, I think the main driving issue was for data sent in that was Int32.

I think the question is whether we need to hold onto that Series, xarray, HDF5 group, etc that the user passed in or just the float64 array.

[jcjaskula-aws]

Yes, indeed I think it just boils down to how to vend back the modeled data in the model results. My point was also not to focus on Series itself but any type that is cast into ndarray (i.e. tuple, list, Series). And I'm currently wondering what would happen if I provide an array_like object that is not one of these three mentioned before (I'm not familiar with nxvalue but I feel it would end up in this situation). My guess (when I read this part of the code) is that the type would not be coerced.

[newville]

@jcjaskula-aws My view on this might be evolving, partly informed from other discussions. I think the recent changes are probably in the right direction.

We definitely need to be passing a 1d Float64 ndarray to the fitting methods. It is OK if that is Complex128 ndarray or a single Float64 value for the scalar Minimizers. I will call this float64 array in general. Other ndarray dtypes or other array-like objects must be converted to float64. I do mean "must".

We could coercion "early" or "late", but we must do it.

For independent data used by the user to calculate the model, I think we should never do anything to the data. Currently we do some coercion for some cases.

We have code throughout model.py that handles pandas Series specially. I think we should probably remove all of this so that any array-like object is handled.

Unlike the independent data, the primary data and weights need to be array-like.
When we do (data-model), we currently coerce the data early and expect but do not ensure that the model is a Float64 array. Separately, I think we should ensure that Model.eval gives a Float64
ndarray. At that point, I think that we do not need to coerce the data, but may want to also coerce the result (data-model)*weights. That is, we should probably change to "coerce late." It might add a little overhead, but that should be acceptable.

I think that would then better allow other array-like objects without us having to add special code. I think the main complication would be handling Nans, but again, I think that doing this late would be OK.

[newville]

an update on this. I just merged #899 into the master branch which addresses this issue, I think in a better way.

First data returned by an objective function is better coerced into Float64 or Complex128 1D-array, ready to send on to the solver code. That should make objective functions that do use "array-like" objects more guaranteed to work well.

Second, for Model.fit(), there are just too many examples of using plain lists floating around to actually do absolutely nothing with input data (or independent data).

With #899, the default behavior is to "coerce early" data and independent data that is "array-like" (list, tuple, pandas Series, h5py Datasets, anything with an __array__() method).

But: the user can pass coerce_farray=False to Model.fit() which will then not do any coercion at all. With that option, the user is able to send pandas Series, etc into their model function, but they are also then taking responsibility for working with the data types they have.

Independent data can still be any complex object: if it is not "array-like", it is not converted to a float64 array. But that also implies the user intended to do this.

I think that adding coerce_farray=True to Model.fit() strikes a decent balance between making the common cases easy (sure, the novice can send in an array of numbers or a pandas Series), while also making more complex cases possible and not too hard.