Events duplication on Consumers

[adamgrad]

As title says, events duplication happens on consumers.

Here is the way it was discovered:

 Hermes::DistributedTrace.where(event_class: "Events::Transfer::Requested").where("event_body -> 'source_id' = '\"e5171595-7698-409e-bae6-7564e6760fc9\"'").where(service: "bsa_secure").pluck(:trace)
 
["cec17e8e69f39a062502d3acf8f999fab874dc898e788f319318b43a17137e0c",
  "2a563c5791afb4a0cc6abc5a6b7994303ee5aeda58b8576209052389433b7a0d",
  "cec17e8e69f39a062502d3acf8f999fab874dc898e788f319318b43a17137e0c"]

Trace "cec17e8e69f39a062502d3acf8f999fab874dc898e788f319318b43a17137e0c" is duplicated and here is creation time for both of them:

Hermes::DistributedTrace.where(event_class: "Events::Transfer::Requested").where("event_body -> 'source_id' = '\"e5171595-7698-409e-bae6-7564e6760fc9\"'").where(service: "bsa_secure").where(trace: "cec17e8e69f39a062502d3acf8f999fab874dc898e788f319318b43a17137e0c").pluck(:created_at)
 
=> [Tue, 08 Mar 2022 19:24:04.766282000 UTC +00:00, Tue, 08 Mar 2022 19:23:50.398927000 UTC +00:00]

As it can be easily noticed, event was broadcasted only once (trace: "2a563c5791afb4a0cc6abc5a6b7994303ee5aeda58b8576209052389433b7a0d") but bsa-secure service produced two traces of processing this same event.

We agreed to solve outcome of this problem with idempotency, yet it would be beneficial to get rid of the source of this duplication.