fix: DIA-1556: Not getting predictions for all tasks

[hakan458]

acks='all' ensures that all replicas have written the message that is produced

group_id for consumers is to allow multiple consumers to consume from the same topic. In this case we have a 1-to-1 relationship and each group_id should be unique in order to properly track the offset for each topic

manual offset committing in the consumer seems to have solved the issue. 4 in a row runs with 10k/10k outputs.

We still need the time.sleep(0.1) in the submit_batch endpoint, without it we still hit the same offset issue on the consumer side... unclear why exactly.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.70%. Comparing base (0c667eb) to head (7695136).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #240      +/-   ##
==========================================
- Coverage   65.89%   65.70%   -0.19%     
==========================================
  Files          47       47              
  Lines        2372     2391      +19     
==========================================
+ Hits         1563     1571       +8     
- Misses        809      820      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[niklub]

Great to see these findings! approved, but please clarify the meaning of each arguments by adding the comments in the code