Continuing running a simulation with a different version of Oceananigans

[Yixiao-Zhang]

I am trying to continue running a simulation with a different version of Oceananigans.

I admit that it is generally not a good idea to use checkpoint files among different versions of the same model. I am doing this because I am planning to replicate some old simulations with the latest version of Oceananigans. To save time, I want to start from the equilibrium state.

First, I tried putting the 3D velocity and tracer fields in the 3D output file from the old simulation (from Oceananigans v0.63.0) into the new simulation (with Oceananigans v0.83.0), but this simulation produced a significant adjustment. I guess that this might be because it is not a bit-by-bit restart since at least the tendencies are not included the output file. Therefore, I am thinking of using checkpoint files, which contain all information to a bit-by-bit restart. I tried using the checkpoint file directly, and it failed. For my understanding, it is because the definitions of certain types have evolved over time, and checkpoint files save not only data but also types, which results a mismatch.

I am wondering how hard it is to do a bit-by-bit restart in a different version of Oceananigans. If it takes lots of efforts, I will likely not do it.

[glwagner]

I think it's likely that a significant adjustment indicates a real difference between the old setup and the new setup --- not just a tiny difference that might be cured by bit-by-bit restart. Do you have the old script available? If I had to guess, it's because the setup relies on a default that was changed. Or, there is a shift because halo regions need to be accounted for...

As for the final question --- what exactly does a bit-by-bit restart mean? Since the versions have changed as you've indicated, presumably the only way to achieve such a restart is to use the original version 0.63.0. In that case, provided the tendencies are handled correctly (which is basically the only thing the Checkpointer does on top of simply saving the prognostic state), I think you'll achieve perfect restart.

[Yixiao-Zhang]

When I said "a bit-by-bit restart", I was thinking about possible reasons for the adjustment in the new setup. One possibility is that the new simulation does restart from the exactly same state, because some information, like data in the halo regions and tendencies from previous steps, for example, is missing. Besides, the output files are single-precision, which my simulations run on double precision. Therefore, geostrophic balance or other balance in the flow is broken, and this will result in an adjustment before reaching the same equilibrium state. This adjustment can take a long time.

Yesterday, I found that the radius of the planet, which is a parameter in my simulation, was changed from 252.1 kilometers to 252. kilometers when I set up the new simulation. This might be the cause of the adjustment in the new simulation, and I will look into this later.

[glwagner]

The setup change could be it!

Besides, the output files are single-precision, which my simulations run on double precision.

I think that this tiny difference would not cause a noticeable adjustment when restarting a simulation. Restarting from single precision would make a detectable difference --- it's not a "bit-by-bit" restart --- and, in the long run, could lead to a divergence of trajectories if your simulation is turbulent / chaotic.

However, I do not think restarting from single precision would cause an "adjustment", as in a step change from one dynamical state to another across the boundary of the restart. The reason is that I don't think the instantaneous dynamics are sensitive to O(1e-6) fractional changes in the state.

But thinking aloud --- maybe the above reasoning fails if the original / saved solution is contaminated by grid scale noise? If you're using WENO then I could see that leading to an instantaneous change in numerical dissipation...

because some information, like data in the halo regions and tendencies from previous steps, for example, is missing.

Halo region data should not matter because the halo regions are computable from the prognostic state. As you noted, RungeKutta3 is self-starting so previous tendencies do not matter. Note, we have tried to design the code to be robust to different ways that users might initialize / run a simulation. However, it's possible that something is slipping through the cracks here. It would help if you could share your script.

[Yixiao-Zhang]

Thank you very much for you reply!

I am using a Runge-Kutt timestep, so I guess tendencies from previous steps are not required in the checkpoint file (might be wrong).

[glwagner]

Yes that's right