O2sim: fix kine published only sending first <aggregate> events

[aalkin]

Proper fix for the issue from #12140

[DanielSamitz]

Hi @aalkin , thank you. This definitely fixes the issue that always the same events were sent.

I tested this and I noticed that in my analysis task that picks up the events sent by the kine_publisher, the run function gets called exactly one time, with all nEvents events in the McHeader and McTracks input, independent of what I set for --aggregate-timeframe in the kine_publisher. Because the while loop here is never left until all events are processed. Is this the expected behavior?

I thought (but maybe I misunderstood) that the run function in my analysis task should be called multiple times, each time with exactly aggregate number of events being processed. That's what I saw with my version in #12140 .
Which one is the expected behavior that this aggregate-timeframe option in the kine_publisher should produce?

[aalkin]

The issue with both kine- and hepmc-publisher is that they are both DPL sources by themselves, there is no "timing" input like in other cases - in analysis and with mctracks-proxy. Therefore their core functions are called exactly once. But the fact that all events are in one message is definitely not expected. At least this is not what I see with hepmc-publisher that works in the same way. I will investigate.

I am planning to make them work similar to the mctracks-proxy eventually, by ensuring they have a "timing" input that we can use to designate timeframes and thus simplify their core functions. Probably I should speed this up.

[DanielSamitz]

Ok, it would be great to have a working version (not perfect, just working for the moment) online before the O2 Tutorial preparation day, because we would like to inlcude something that relies on the kine_publisher in the hands-on.

With #12140 everything seems to work so far.
With this version I crash with the error:

[17011:o2sim-kine-publisher]: [14:21:12][ERROR] Exception caught: shmem: could not create a message of size 482820, alignment: 64, free memory: 26048

when I run on larger files.

[aalkin]

#12140 works because it only returns first aggregate events and exits the for loop, #12141 fails because it tries to get all the events at once, these are both wrong. I'll re-implement this with DPL timer input then. I'll need a couple of days for that, should be ready not later than mid-next-week. You can proceed with #12140 for now, bearing in mind that it only publishes a fraction of events. Proper implementation will not break the workflows.

[DanielSamitz]

OK, thank you very much for taking care of this.

You can proceed with #12140 for now, bearing in mind that it only publishes a fraction of events.

Btw, I tested #12140 and I do get ALL events, not only a fraction (that was exactly the point that I wanted to fix with it. Before, it was also sending the correct number of nEvents, but always a repetition of the same first aggregate events again and again.).

[aalkin]

Interesting, then how come there is no out-of-memory issue?

[DanielSamitz]

I don't know. I don't understand these things well enough, I can only say what I see. And it seems that the events do get shipped in packages of aggregate events at a time (i.e. what I see is that the run function in my analysis task gets called nEvents/aggregate times, each time processing aggregate events, until all events are done.)