Skip to content
This repository has been archived by the owner on Jun 7, 2024. It is now read-only.

Timelines: old topic gets removed immediately if retention_time = -1 (infinite) #1121

Open
eupestov opened this issue Nov 29, 2019 · 5 comments

Comments

@eupestov
Copy link

eupestov commented Nov 29, 2019

When a new timeline is created for an event type Nakadi sets the 'clean at' date for the previous timeline based on the event type retention time. After the retention period is over Nakadi deletes the old topic.
But if the event type's retention time is set to -1 the old topic is removed almost immediately because of the way the cleantAt date is calculated:

final Date cleanupDate = new Date(System.currentTimeMillis() + retentionTime);

In our setup we use -1 for indefinite retention, although it is not described in Nakadi docs. So either Nakadi should not allow -1 for retention time or clean up should not happen.

@adyach
Copy link
Member

adyach commented Nov 29, 2019

@eupestov thank you for filing an issue.

just to clarify -1 is a valid kafka retention.ms https://kafka.apache.org/documentation/#topicconfigs

@eupestov
Copy link
Author

just to clarify -1 is a valid kafka retention.ms

Yeah, that why we start using it. But is it to be officially supported by Nakadi? Taking into account it's future support for other messaging backends and other features like timelines.

@adyach
Copy link
Member

adyach commented Nov 29, 2019

Yeah, that why we start using it. But is it to be officially supported by Nakadi? Taking into account it's future support for other messaging backends and other features like timelines.

it is gonna be discussed internally (I will update on that soon), from my point of view it makes, because kafka supports it.

@mneverov
Copy link
Contributor

hi, I'd like to work on this

@mneverov
Copy link
Contributor

PR #1135

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants