Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Introduce setting for max record age on targets #2692

Open
dluo-sig opened this issue Sep 30, 2024 · 1 comment
Open

feat: Introduce setting for max record age on targets #2692

dluo-sig opened this issue Sep 30, 2024 · 1 comment
Labels
kind/Feature New feature or request valuestream/SDK

Comments

@dluo-sig
Copy link

dluo-sig commented Sep 30, 2024

Feature scope

Taps (catalog, state, stream maps, tests, etc.)

Description

https://github.com/meltano/sdk/blob/main/singer_sdk/target_base.py#L53

_MAX_RECORD_AGE_IN_MINUTES is hard-coded and does not allow for configuration. I'm not sure why this would be enforced, as I would normally expect that the records that I'm pulling would be consistent as of the time of the original query. Maybe allow 0 or -1 to indicate to never time out. That would let things operate on configured batch size. This would also allow for some performance improvements as we have noticed that upsert performance is better than linear with the number of records.

@dluo-sig dluo-sig added kind/Feature New feature or request valuestream/SDK labels Sep 30, 2024
@dluo-sig
Copy link
Author

It seems like it was changed from 30 to 5 here. #1064

Not sure if I'm understanding correctly, but based on this comment, it seems like the batch flush operation is a performance penalty as opposed to finishing up the batch? I do see this happen every 5 minutes and it takes around 40 seconds before I see the next log message on my target starting to process the batch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Feature New feature or request valuestream/SDK
Projects
None yet
Development

No branches or pull requests

1 participant