Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PSC-STM-A3: Add Storage for PSC Stream Timepoint #242

Open
tiredpixel opened this issue Feb 28, 2024 · 0 comments
Open

PSC-STM-A3: Add Storage for PSC Stream Timepoint #242

tiredpixel opened this issue Feb 28, 2024 · 0 comments
Assignees

Comments

@tiredpixel
Copy link
Contributor

When using the PSC Stream API, it requires either:

  • A timepoint pointer of the last record seen through the API, which will then return records seen since here
  • Nothing sent through, which will obtain any new records from now
stream_client.read_stream(timepoint:)

This timepoint is important to ensure that if there is any restart of requests (eg after a timeout or failure), that no records were lost in the meantime.

This needs to be:

  • Retrieved when starting up, to read latest timepoint
  • Stored after a new record has been processed

A simple key/value pair is sufficient for storage and retrieval of this, so potentially Redis would be a good choice if that is available. However, note that if this is on a single EC2 instance it is a single point of failure.

Estimate: 6 hours

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant