Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batcher: Make the confirmation of successful messages more intuitive #2750

Open
paul-butcher opened this issue Nov 6, 2024 · 1 comment
Open

Comments

@paul-butcher
Copy link
Contributor

paul-butcher commented Nov 6, 2024

This business with the indices is a relic of the SQS Stream driven version. It's a bit tricky to untangle, so I've left it in place.

It might be easier to rewrite this later on when we no longer use that approach.

Originally posted by @paul-butcher in #2735 (comment)

See also Slack

How it works:

  • Take a list of messages, use that to create a corresponding list of path strings
  • create selectors for all the paths, associate each selector with a position in the list
  • bundle up the selectors (the point of the batcher), associate each batch with a list of positions of paths in the list
  • try sending on each batch (again the point of the batcher), if it fails, return the list positions that fail
  • Iterate over the message list. If any message in the list has the same position as one of the paths that failed, remove that message from the output list
  • Finally, emit the successful messages (or rather, the messages that did not fail), which SQSStream will then delete from the input queue.
    ...
    The way I think it should work, is that it could conceal the message within a generic, and pass that along a Flow When it succeeds, it emits whatever succeeds.
@paul-butcher
Copy link
Contributor Author

The above statement about passing something along a Flow supposes that we continue using Akka for this, which currently looks a bit unnecessary and also contrary to the fast-start requirement of Lambda.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant