Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework Sotw Cache interface #6

Closed

Conversation

valerian-roche
Copy link
Collaborator

Datadog fork PR matching envoyproxy#584

It has been raised that the addition of StreamState within the Cache interface is breaking the separation of concern between the server side managing the xds protocol itself and the cache which is only in charge of returning responses based on watches
PR envoyproxy#577 has been opened proposing to fully remove any client state from the CreateWatch interface

This PR is an alternate proposal based on a few concerns/aspects:

  • one of the argument on removing it is that the sotw protocol is stateless. It actually is not, and is explicitly visible in this PR: the lastResponse latched value as well as the version hack when the resource list is changed very well show that it is stateful
  • in cases outside of lds and cds, it is clearly stated that the control-plane is not expected to send back all resources when a single one change. This is highly critical in the case of ads with eds, as there is a likelihood of both having a lot of resources and a high churn. The implementation sending everything each time is technically tolerated, but is pushing on the data-plane work that could be done at low cost on the control-plane
  • the current support of both delta and sotw, for both linear and simple cache means that every change has to be done 4 times, with four very different implementations, each one supporting only a subset of use cases (e.g. linear does not support cds/lds as it doesn't handle wildcard in this case, and sotw does not work properly as resource list is improperly maintained). I strongly believed that both implementation should converge. The only remaining differences should be in the messages themselves, abstracted in the server part. Having a common interface for sotw and delta is a first step for this. It would also allow in the future to make snapshot far more efficient by supporting a linear version per snapshot

In this context, this PR is changing:

  • creates a ClientState within the cache package, driven by the data needed from the cache perspective to be able to answer a client request
  • simplifies streamState
    • makes it compatible with the new ClientState interface
    • removes the IsFirst notion, as the protocol states it should come from the nonce in the request
    • removes the duplicate/parallel knownResourceNames and use the delta one instead. It will also properly carry the version in a future PR

…interface is breaking the separation of concern between the server side managing the xds protocol itself and the cache which is only in charge of returning responses based on watches

PR envoyproxy#577 has been opened proposing to fully remove any client state from the CreateWatch interface

This commit is updating the Cache interface based on a few concerns/aspects:
- one of the argument on removing it is that the sotw protocol is stateless. It actually is not, and is explicitly visible in this PR: the lastResponse latched value as well as the version hack when the resource list is changed very well show that it is stateful
- in cases outside of lds and cds, it is clearly stated that the control-plane is not expected to send back all resources when a single one change. This is highly critical in the case of ads with eds, as there is a likelihood of both having a lot of resources and a high churn. The implementation sending everything each time is technically tolerated, but is pushing on the data-plane work that could be done at low cost on the control-plane
- the current support of both delta and sotw, for both linear and simple cache means that every change has to be done 4 times, with four very different implementations, each one supporting only a subset of use cases (e.g. linear does not support cds/lds as it doesn't handle wildcard in this case, and sotw does not work properly as resource list is improperly maintained). I strongly believed that both implementation should converge. The only remaining differences should be in the messages themselves, abstracted in the server part. Having a common interface for sotw and delta is a first step for this. It would also allow in the future to make snapshot far more efficient by supporting a linear version per snapshot

In this context, this commit is changing:
- creates a SubscriptionState within the cache package, driven by the data needed from the cache perspective to be able to answer a client request
- simplifies streamState
  - makes it compatible with the new SubscriptionState interface
  - removes the IsFirst notion, as the protocol states it should come from the nonce in the request
  - removes the duplicate/parallel knownResourceNames and use the delta one instead. It will also properly carry the version in a future PR

Signed-off-by: Valerian Roche <[email protected]>
Rename KnownResources to ACKedResources to better reflect the change

Signed-off-by: Valerian Roche <[email protected]>
pkg/cache/v3/cache.go Outdated Show resolved Hide resolved
pkg/cache/v3/cache.go Outdated Show resolved Hide resolved
pkg/cache/v3/cache.go Outdated Show resolved Hide resolved
pkg/cache/v3/cache.go Show resolved Hide resolved
pkg/cache/v3/linear.go Show resolved Hide resolved
pkg/server/delta/v3/server.go Outdated Show resolved Hide resolved
pkg/server/sotw/v3/ads.go Outdated Show resolved Hide resolved
pkg/server/sotw/v3/server.go Outdated Show resolved Hide resolved
pkg/server/sotw/v3/xds.go Outdated Show resolved Hide resolved
pkg/server/stream/v3/subscription.go Outdated Show resolved Hide resolved
@valerian-roche valerian-roche force-pushed the vr/interface branch 2 times, most recently from 5943db4 to 33d2fa9 Compare January 17, 2024 03:13
pkg/cache/v3/cache.go Show resolved Hide resolved
pkg/cache/v3/linear.go Show resolved Hide resolved
pkg/server/sotw/v3/server.go Outdated Show resolved Hide resolved
@atollena
Copy link

I guess you can close this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants