Compatibility of pickles across river versions

[martinvanhensbergen]

Hi all,

For a PoC, I used river==0.10.1 and I used the recommended pickling to store the state of our models. Later I upgraded to river==0.15.1 and my code didn't work anymore after the pickles were loaded. Deleting the old pkl files and starting anew solved the problem (but ofcourse resulted in state being gone).

What is the recommended way of dealing with upgrades of the river package and stored pkls? For example, the state of the RunninQuantile class is basically the stored window but there seems to be no way to export just this state for pickling, nor a way to import this state.

Any guidance on how to safely export and import model state between river versions would be very helpful :-)

Thanks in advance!

[AdilZouitine]

Hi Martin,
The pickling of the rolling quantile is incompatible between river 0.10.1 and 0.15.1 because we rewrote a significant part of the stat module in rust #1025, and internally the object had a lot of modification.
It is a side effect, and now the stat module is stable, so that the compatibility will be ok for the future version.

I hope my answer answers your questions 😄

[martinvanhensbergen]

Hi Adil, thank you for your incredibly fast response :-)

This indeed answers my question regarding this particular class, but I am still wondering how it should be done in general?

As I understand correctly, pickling the river classes for storage does not only store the state of the class but basically the entire code as well in that class?

So if I pickle, say, RandomQuantile of 0.15.1 in mystate.pkl, upgrade the river package to 0.16.1, and then in my code read mystate.pkl again, then new instances of that class will run the 0.16 code while the old pickled classes wil run the 0.15.1 code? This would result in unpredictable code being run?

Or is there some internal (de)serializing happening under the hood that makes sure that only the actual parameters of the state are being saved and was the Quantile class in 0.10.1 an exception?

Thanks again for your help in this :-)

[AdilZouitine]

This indeed answers my question regarding this particular class, but I am still wondering how it should be done in general?

As far as I know is pickling the object.

As I understand correctly, pickling the river classes for storage does not only store the state of the class but basically the entire code as well in that class?
al (de)serializing happening under the hood that makes sure that only the actual parameters of the state are being saved, and was the Quantile class in 0.10.1 an exception?

It depends on pure Python classes; the comportment is usual; it stores the entire code and the state, but for the Rust stat, we save only the actual parameters of the state, for example, the rolling quantile: In python binding side we override __setstate__, __getstate__, __getnewargs__ to make this class pickle compatible and in rust side we serialize each attribute of the struct.
So if we don't change the underlying data structure on the Rust side, it will stay the same across the versions.

[martinvanhensbergen]

Okay, thanks very much for explaining! This answers my question indeed :-)

Best wishes,

Martin