openPMD file timestamps and accessing data individually

[bzdjordje]

Is there a simple explanation as what WarpX is doing with its openPMD output dumps? I understand preallocating storage for future dumps, but it seems to keep updating later dump files as well as ones that already are in the "past" as the simulation is running, and when I try to check on a simulation as it is running I often get an error for the *0000.h5 case. Likewise, in python/jupyter can I use openPMDViewer on a single file as opposed to an entire directory? It has been getting 'stuck' and I naively assume that my file dumps are getting too big for it to read into memory or whatever it is doing. It would be nice to take advantage of the backtransform and spectral->RZ transform tools you already have built into it.

-rw------- 1 bdjordje bdjordje 1375079672 Sep 20 08:59 openpmd_00035.h5
-rw------- 1 bdjordje bdjordje 1379826512 Sep 20 09:00 openpmd_00025.h5
-rw------- 1 bdjordje bdjordje 1367655024 Sep 20 09:00 openpmd_00015.h5
-rw------- 1 bdjordje bdjordje 1343445176 Sep 20 09:01 openpmd_00072.h5
-rw------- 1 bdjordje bdjordje 1353728184 Sep 20 09:01 openpmd_00005.h5
-rw------- 1 bdjordje bdjordje 1343749944 Sep 20 09:01 openpmd_00062.h5
-rw------- 1 bdjordje bdjordje 1345544920 Sep 20 09:02 openpmd_00052.h5
-rw------- 1 bdjordje bdjordje 1371393800 Sep 20 09:02 openpmd_00042.h5
-rw------- 1 bdjordje bdjordje 1373752568 Sep 20 09:03 openpmd_00032.h5
-rw------- 1 bdjordje bdjordje 1377229864 Sep 20 09:03 openpmd_00022.h5

[ax3l]

Hi @bzdjordje,

I understand preallocating storage for future dumps, but it seems to keep updating later dump files as well as ones that already are in the "past" as the simulation is running

Is this a boosted frame simulation?
In order to reconstruct the lab frame physics in a boosted frame simulation, we do so-called back-transformed diagnostics (BTD). In BTD, we launch a plane for fixed lab-frame times t of interest and propagate it through the simulation from back to front until all z planes that correspond to this time in the lab-frame are collected.

Thus, we add for many, many times to the same lab-frame time steps.

in python/jupyter can I use openPMDViewer on a single file as opposed to an entire directory?

openPMD-viewer does not support the feature to open only a selected file of a data series (cc @RemiLehe), but it would make a good feature to track therein.
You can currently copy an individual file of a file-based openPMD series in a separate directory and open only that.

Alternatively, openPMD-api supports to open only one file by passing a fixed filename instead of a pattern that contain the %T placeholder to openpmd_api.Series.

It has been getting 'stuck' and I naively assume that my file dumps are getting too big for it to read into memory or whatever it is doing.

This happens on open or on access to an individual iteration/step?

It would be nice to take advantage of the backtransform and spectral->RZ transform tools you already have built into it.

Yes, that is the idea with openPMD-viewer to make easy.
So to summarize, you are talking about an

• RZ
• boosted frame

simulation?