Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix chunk ends late issue #275

Merged
merged 3 commits into from
Oct 29, 2024
Merged

Fix chunk ends late issue #275

merged 3 commits into from
Oct 29, 2024

Conversation

cfuselli
Copy link
Member

@cfuselli cfuselli commented Oct 23, 2024

We increase the default value for the min_electron_gap_length_for_splitting. This defines the time windows to split chunks at s2 photon propagation.

This was creating issues when afterpulses of large S2s would go out of the chunk boundaries. For some high energy simulations (BiPos with ~5k events/job) this was having a fail rate of ~50% of jobs submitted. With the new configuration the issue now appears less than 1%.

The errors would be of the type:

ValueError: Attempt to create chunk (00009.pulse_windows: 1565sec 869075581 ns - 1777sec 483266062 ns, 74217 items, 0.0 MB/s) whose data ends late at 1777483266961

and it created problems with different plugins, see here out of 393 submittet jobs the plugin that failed:

Total number of files scanned: 393
Total number of files with the pattern: 201
Summary of chunk types (from last occurrence per file):
  raw_records: 9
  pulse_windows: 36
  pmt_afterpulses: 156

with the new configuration only 3 runs failed.

Note that this change does not produce lineage change.

Thank you very much to @HenningSE for taking the time to look into this and identify the problem and find a solution :)

@coveralls
Copy link

coveralls commented Oct 25, 2024

Pull Request Test Coverage Report for Build 11571190056

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 79.905%

Totals Coverage Status
Change from base Build 11571184860: 0.0%
Covered Lines: 2525
Relevant Lines: 3160

💛 - Coveralls

Copy link
Collaborator

@HenningSE HenningSE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@HenningSE HenningSE merged commit 084662e into main Oct 29, 2024
4 checks passed
@HenningSE HenningSE deleted the fix-chunk-ends-late branch October 29, 2024 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants