New Access type READ_LINEAR #1291

franzpoeschel · 2022-06-09T15:54:16Z

Situation:
Using ADIOS2 steps in openPMD was always a bit challenging due to slightly incompatible IO workflows.
Traditionally in openPMD, we parse a Series upon opening it and users expect to use random-access for navigating through the Series. Alternatively, in ADIOS2 one can parse each iteration only after opening its corresponding IO step. With an IO step opened, ADIOS2 will only show the data pertaining to exactly that step and no other data (except for attributes that are designed to be constant). With the novel BP5 engine, this distinction is now mandatory to make since it has two access modes: adios2::Mode::Read and adios2::Mode::ReadRandomAccess.

Also adds the missing Append mode to the Python bindings.

This PR adds a new read mode READ_LINEAR and slightly restricts the usage of READ_ONLY:

READ_ONLY corresponds to our traditional workflow of parsing everything up-front and then not again. It is implemented in the backend via adios2::Mode::ReadRandomAccess.
New in this PR: If the engine supports it at all, the Series will be parsed before opening any IO step. In variable-based iteration encoding, this means that data from a later iteration might leak into the first.
Iterations that were parsed up front will not be parsed again upon opening their corresponding IO step.
Optionally: Stop using IO steps altogether in this access mode in front- and backend. That would be API breaking for streaming (all other API breakage of this PR only affects the experimental new ADIOS2 schema), but with streaming, the other read mode makes more sense anyway.
READ_LINEAR: Don't parse anything upon opening the Series. To get any data, Series::readIterations() must be used.

TODO

Don't use steps at all in READ_ONLY (-> breaking change since this makes READ_LINEAR required for streaming workflows)
→ Probably best for the next release: Give a runtime warning when using READ_ONLY in contexts that should require READ_LINEAR (such as streaming), and disable the functionality altogether in the release after that
Memory optimization: Delete old iterations in READ_LINEAR mode (also ok for a future PR)
Merge Parsing logic: fail gracefully on unexpected input #1237 first
ADIOS 2.7 compatibility
Actually test BP5, especially with old iteration encoding
Documentation
Test that Streaming keeps working with READ_ONLY mode for now
What happens if writing a Series with /data/snapshot attribute and then reading in random-access mode
-> /data/snapshot gets properly ignored, bp4_steps tests this
Ignore /data/snapshot attribute if the backend reports that it prefers upfront parsing

Commits:

Basic implementation
9a7fab7 Prepare scaffolding for Access::ReadLinearly
6f0e74c Adapt frontend to linear read mode
ed40837 Make OPEN_FILE task parameters writable
73bae08 ADIOS2 implementation

Testing and bindings
0b64fa2 Necessary fixes for tests
8dad845 Add and adapt Python bindings
0ff801b Testing

Documentation
b06d257 Documentation: C++ and Python
fbca5e8 rst documentation

Allow parsing global attributes before opening first iteration¹
c841156 Make SeriesIterator into a handle
357e487 Reading attributes without needing to open the first step

Workaround for ADIOS2 aggregation issue
891cba0 Don't write attributes too early

Delete old iterations from data structures ²
dd4228a Delete old iterations in READ_LINEAR mode
87d2004 DEREGISTER task

¹) In READ_LINEAR, the Series object is completely empty after constructing it. This is weird when we want to read global attributes:

Series s("data.bp", Access::READ_LINEAR);
// no attributes available
for(auto iteration: s.readIterations())
{
    // Attributes are now readable
    // It's weird having to open the first iteration for this
}

Solution:

Series s("data.bp", Access::READ_LINEAR);
// no attributes available
s.readIterations();
// global attributes can now be read, no iterations yet accessible
for(auto iteration: s.readIterations()) ...

²) In READ_LINEAR mode, iterations can only be accessed via series.readIterations(). We should not store them into series.iterations (except the currently active one), to avoid (1) users getting confused why old iterations are still there and (2) slowly rising memory usage in long-running simulation workflows.

lgtm-com · 2022-06-09T16:30:51Z

This pull request introduces 2 alerts when merging 77522f9 into 20e271b - view on LGTM.com

new alerts:

2 for Missing return statement

lgtm-com · 2022-06-10T16:12:59Z

This pull request introduces 2 alerts when merging 018608b into 20e271b - view on LGTM.com

new alerts:

2 for Missing return statement

lgtm-com · 2022-06-13T13:29:51Z

This pull request introduces 2 alerts when merging 5b63442 into 20e271b - view on LGTM.com

new alerts:

2 for Missing return statement

Currently only available for BP5 engine, will be generalized into Linear read mode in openPMD#1291. If the backend does not support the snapshot attribute, then iterate in ascending order, skipping duplicate and non-linear iteration indices. Not possible if the Series is parsed ahead of time.

franzpoeschel · 2022-08-16T09:24:36Z

src/IO/ADIOS/ADIOS2IOHandler.cpp

+                    m_mode = adios2::Mode::ReadRandomAccess;
+                    // Overwrite the old IO object since it is stained with the
+                    // objects of the now closed engine
+                    create_IO();


Problem: This deletes all configuration

src/ReadIterations.cpp

-ReadIterations::ReadIterations(Series series) : m_series(std::move(series))
-{}
+ReadIterations::ReadIterations(
+    Series series,


Use access::readOnly for runtime checks in backends

Needed so backends can give feedback on their preferred parse mode.

Commit with proper tests will follow later, this is just to highlight the API changes brought forth by this PR (API changes only for things that were dev and/or experimental)

ax3l

This looks great to me 🚀 ✨

Thank you also for the clear structure for reviewing 🙏

ax3l · 2023-02-13T18:54:22Z

test/SerialIOTest.cpp

@@ -4923,6 +4925,17 @@ TEST_CASE("bp4_steps", "[serial][adios2]")
    bp4_steps("../samples/nullcore.bp", nullcore, "");
    bp4_steps("../samples/bp4steps_default.bp", "{}", "{}");

+    // bp4_steps(


Just flagging: these tests are still commented - intentional?
Remove dead code in favor of a comment?

ax3l · 2023-02-13T18:54:44Z

test/SerialIOTest.cpp

-                ParseMode::WithSnapshot,
-                jsonConfigOld);
+            // This test config does not make sense
+            // append_mode(


Just flagging: this test is still commented - intentional?
Remove and just leave a comment?

franzpoeschel added frontend: C++17 frontend: Python3 internal backend: ADIOS2 labels Jun 9, 2022

franzpoeschel marked this pull request as draft June 10, 2022 15:10

franzpoeschel force-pushed the topic-linear-read branch from 9b590a8 to 018608b Compare June 10, 2022 15:31

franzpoeschel marked this pull request as ready for review June 13, 2022 15:00

franzpoeschel force-pushed the topic-linear-read branch 2 times, most recently from d581733 to 43d1316 Compare June 17, 2022 13:56

franzpoeschel force-pushed the topic-linear-read branch 2 times, most recently from a62161d to ff8f8e8 Compare June 22, 2022 11:37

franzpoeschel force-pushed the topic-linear-read branch 3 times, most recently from 35ae314 to be43a49 Compare July 11, 2022 10:27

franzpoeschel force-pushed the topic-linear-read branch 2 times, most recently from df20208 to 148706d Compare July 19, 2022 09:57

franzpoeschel force-pushed the topic-linear-read branch 4 times, most recently from 54aec65 to 5681b62 Compare July 27, 2022 12:20

franzpoeschel force-pushed the topic-linear-read branch from 31f8eb0 to cc24363 Compare July 29, 2022 09:44

franzpoeschel force-pushed the topic-linear-read branch from cc24363 to 68a209f Compare August 15, 2022 09:15

franzpoeschel commented Aug 16, 2022

View reviewed changes

franzpoeschel mentioned this pull request Aug 16, 2022

ADIOS2 schema 2022_07_26, based on ADIOS2 modifiable attributes #1310

Merged

5 tasks

github-advanced-security bot found potential problems Jan 5, 2023

View reviewed changes

src/ReadIterations.cpp Fixed Show fixed Hide fixed

franzpoeschel force-pushed the topic-linear-read branch from 8f0ac22 to 1c9907c Compare January 5, 2023 17:42

This was referenced Jan 13, 2023

BP4 and BP3: MPI metadata aggregation nonfunctional when the first defined attribute has two leading underscores ornladios/ADIOS2#3433

Closed

Linear read mode: Delete old iterations franzpoeschel/openPMD-api#4

Merged

github-advanced-security bot found potential problems Jan 25, 2023

View reviewed changes

franzpoeschel added 6 commits January 27, 2023 16:05

Prepare scaffolding for Access::ReadLinearly

9a7fab7

Use access::readOnly for runtime checks in backends

Adapt frontend to linear read mode

6f0e74c

Make OPEN_FILE task parameters writable

ed40837

Needed so backends can give feedback on their preferred parse mode.

ADIOS2 implementation

73bae08

Necessary fixes for tests

0b64fa2

Commit with proper tests will follow later, this is just to highlight the API changes brought forth by this PR (API changes only for things that were dev and/or experimental)

Add and adapt Python bindings

8dad845

franzpoeschel force-pushed the topic-linear-read branch from 32f1e65 to dd68119 Compare January 27, 2023 15:26

franzpoeschel added 8 commits January 27, 2023 16:36

Testing

0ff801b

Documentation: C++ and Python

b06d257

rst documentation

fbca5e8

Make SeriesIterator into a handle

c841156

Reading attributes without needing to open the first step

357e487

Don't write attributes too early

891cba0

Delete old iterations in READ_LINEAR mode

dd4228a

DEREGISTER task

87d2004

franzpoeschel force-pushed the topic-linear-read branch from dd68119 to 87d2004 Compare January 27, 2023 15:36

franzpoeschel mentioned this pull request Feb 10, 2023

Update examples #1371

Merged

1 task

ax3l approved these changes Feb 13, 2023

View reviewed changes

ax3l reviewed Feb 13, 2023

View reviewed changes

ax3l merged commit 71b3e94 into openPMD:dev Feb 13, 2023

ax3l added this to the 0.15.0 milestone Feb 13, 2023

franzpoeschel mentioned this pull request Feb 24, 2023

Fix for openpmd-pipe in Linear read mode: Open first iteration before reading any attributes #1379

Merged

franzpoeschel mentioned this pull request Mar 21, 2023

Add a RandomAccess read mode for BP5 #1274

Closed

BenWibking mentioned this pull request Jan 17, 2024

ADIOS2 reader cannot open BP5 files (need to bump version to 2.8.0 or newer) visit-dav/visit#19176

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Access type READ_LINEAR #1291

New Access type READ_LINEAR #1291

franzpoeschel commented Jun 9, 2022 •

edited

Loading

lgtm-com bot commented Jun 9, 2022

lgtm-com bot commented Jun 10, 2022

lgtm-com bot commented Jun 13, 2022

franzpoeschel Aug 16, 2022

ax3l left a comment

ax3l Feb 13, 2023 •

edited

Loading

ax3l Feb 13, 2023 •

edited

Loading

New Access type READ_LINEAR #1291

New Access type READ_LINEAR #1291

Conversation

franzpoeschel commented Jun 9, 2022 • edited Loading

Commits:

lgtm-com bot commented Jun 9, 2022

lgtm-com bot commented Jun 10, 2022

lgtm-com bot commented Jun 13, 2022

franzpoeschel Aug 16, 2022

Choose a reason for hiding this comment

ax3l left a comment

Choose a reason for hiding this comment

ax3l Feb 13, 2023 • edited Loading

Choose a reason for hiding this comment

ax3l Feb 13, 2023 • edited Loading

Choose a reason for hiding this comment

franzpoeschel commented Jun 9, 2022 •

edited

Loading

ax3l Feb 13, 2023 •

edited

Loading

ax3l Feb 13, 2023 •

edited

Loading