How to prevent poor fitting

[dsandber]

The image below shows a fit I got using pwlf. Adjacent linear segments are shown in alternating colors for easy visualization. The blue dots are the original data points.

Is this issue not enough points in that region? Any suggestions on how to improve this?

Thanks!

[cjekel]

This looks like a failure in the optimization. Can you try fit fast with a large population. Maybe pop=50. The runtime might be quite a bit longer. How many line segments do you have? It's possible you have too many to be a well posed problem.

…

On Mon, Sep 11, 2023, 12:33 dsandber ***@***.***> wrote: The image below shows a fit I got using pwlf. Adjacent linear segments are shown in alternating colors for easy visualization. Is this issue not enough points in that region? Any suggestions on how to improve this? [image: image] <https://user-images.githubusercontent.com/5387753/266991252-c130e5e5-91da-46e8-a396-198453c337d5.png> Thanks! — Reply to this email directly, view it on GitHub <#111>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADJ5Z4NHHGRLSBO2YA5VBLLXZ3SJLANCNFSM6AAAAAA4TDIV7Y> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dsandber]

The knot-points are fixed, so there's no optimisation if I'm understanding you right. The knot points have to be fixed because I'm optimising for an absurdly CPU constrained microcontroller. There's 128 segments. Seems like I should first fit a curve somehow, and then use pwlf after that once the data is continuous. Any suggestions for what to use to fit the curve? The underlying curve is very smooth with only slow changes in slope.

Update: Seems I want a smoothing spline. And then I'll feed that to PWLF.

[cjekel]

Are you running into something like this? https://jekel.me/piecewise_linear_fit_py/examples.html#bad-fits-when-you-have-more-unknowns-than-data If you don't need 128 segments, i would reduce the number of segments down to a dozen or so fixed break points. As you said, it's not that non linear of a relationship. If you need to get artificially more data, fit a linear interpolation model to the data. Then sample from that model. Scipy and numpy have these linear interpolation schemes. There is also linear kernel rbf.

…

On Tue, Sep 12, 2023, 11:37 dsandber ***@***.***> wrote: The knot-points are fixed, so there's no optimisation if I'm understanding you right. The knot points have to be fixed cause I'm optimising for a very CPU constrained microcontroller. There's 128 segments. Seems like I should first fit a curve somehow, and then use pwlf after that once the data is continuous. Any suggestions for what to use to fit the curve? It's a very smooth with only slow changes in slope. — Reply to this email directly, view it on GitHub <#111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADJ5Z4PH3T3VCGLSQXDWTRLX2AUNBANCNFSM6AAAAAA4TDIV7Y> . You are receiving this because you commented.Message ID: ***@***.***>

[dsandber]

In the process of making an example for you I realised I had a bug that was causing the issue, however after fixing it there's still a problem.

I have 256 data points and 129 knots. I've attached an example with the data as a json. Running it should open the graphs in a browser. Seems like the gaps are causing an issue even though there's more data points than segments.

The graphs look like:

data.txt
example.txt