restart with implicit solver

[JustinRayAngus]

WarpX cannot perform a restart when using the implicit solvers.

Here is the error message from trying to restart a 2D simulation:
--- INFO : Runtime real component x_n was found in the checkpoint file, but
it has not been added yet. Adding it now.
--- INFO : Runtime real component z_n was found in the checkpoint file, but
it has not been added yet. Adding it now.
--- INFO : Runtime real component ux_n was found in the checkpoint file, but
it has not been added yet. Adding it now.
--- INFO : Runtime real component uy_n was found in the checkpoint file, but
it has not been added yet. Adding it now.
--- INFO : Runtime real component uz_n was found in the checkpoint file, but
it has not been added yet. Adding it now.
amrex::Abort::444::ParticleContainer::Restart(): nr not the expected value !!!
SIGABRT

The --INFO lines are produced around line 172 of Source/Diagnostics/ParticleIO.cpp. The implicit solver defines these extra variables at runtime. They are written to the checkpoint file, but they are not needed. At the time the restart file is read, these extra variables have not yet been added to the particle container, so these variables are added in the Restart function in ParticleIO.cpp (should they be added here?).

The Abort comes from the Restart function called in amrex/Src/Particle/AMReX_ParticleIO.H around line 700. There it reads in the number of components (which is 5 more now than it is normally for a 2D simulation), and it fails because it is expected this value to be equal to
nrc = NStructReal + NumRealComps() - AMREX_SPACEDIM, which I have verified is 5 time less than what nr is equal to.

This needs to be fixed. Question is how to fix it. Which part of this problem is the one that is non-ideal? @RemiLehe @WeiqunZhang @dpgrote

Ideas?

[atmyers]

Hi @JustinRayAngus - could you see if PR #5489 fixes the issue for you?

Basically there was a block of code setting up the implicit solver that was being call in InitFromScratch but not InitFromCheckPoint. If the extra variables that save the positions and velocities at the start of the time step aren't needed, we could also remove those from the Checkpoint files. However, the call to m_implicit_solver.Define() still needs to be done when restarting from the Checkpoint file.

[JustinRayAngus]

Hi @atmyers. Thanks for looking into this. I haven't tried it yet, but I believe it will work. As you mentioned though, the extra variables are not needed, so I think the best solution would be to remove them from writing to the checkpoint files.