Should we provide guarantees over our Go API? What about events?

[richardcase]

Started from discussion on #7786.

Currently, CAPI only really considers its CRDs part of its API guarantees. And from the discussion on the pr:

I think the discussion is fundamentally "do we want to guarantee API stability for events" (and what does API stability for events exactly mean) and not just "how do we get the apidiff to fail" or "how do we define the events in our API package so they are technically a Go API".

(thanks @sbueringer)

[richardcase]

Based on the discussion on the PR, this question is wider than just Events and is probably "do we provide guarantees for our Go Package API to consumers"......which I'm guessing is part of the reason apidiff was enabled in the first place. But as pointed out, "getting apidiff to fail" isn't the whole story, but it's still part of the story.

[fabriziopandini]

We are already providing guarantees on our GO APIs, as described in the contributor guide; also, with 1.0 we decided to decuple semantic versioning used for our releases from API versioning in order to allow the project to evolve with fewer frictions.

WRT to the question is about guarantees on events, which have never been formalized before, in my humble opinion this falls more on the API guarantees bucket than in the golang guarantee bucket, no matter if by the opportunity we are going to use apidiff to enforce both (but frankly speaking I will keep them separated for sake of simplicity in reading the signal)

[sbueringer]

+1

I general we provide various guarantees for the CRDs / CRD Go API types, other Go types in API packages and even public Go packages in general.

The guarantees for public packages differ depending on Go package. The general guarantees with a few exceptions are that we don't do breaking changes to types in public Go packages without previous deprecation (deprecation for ~ a Cluster API minor release).

Exceptions are:

• API go packages have stronger guarantees
• Some Go packages have weaker guarantees, this is then defined in a local README, e.g. https://github.com/kubernetes-sigs/cluster-api/blob/main/cmd/clusterctl/README.md
• Sometimes we directly changed a function signature if it seemed the more reasonable thing to do. These where cases where we wanted to do minor modifications to a function signature. The alternative would have been:
  • Introduce a new func with a new name and the new signature
  • Deprecate the old func
  • Remove the old func
  • Create a func with the new signature but the old name
  • Deprecate the func with the new name
  • Remove the func with the name name
  • => This just doesn't seem to be worth it if you e.g. only want to drop a parameter and it's less disruption for consumers if we just change the signature directly

So in general we try to strike a pragmatic balance between providing stability to API consumers fo the Cluster API repository and doing changes in a reasonable way. As far as I'm aware there were no complaints about how we handle it currently for the last few months / years.

[sbueringer]

Coming back to events. As far as I can tell we currently don't provide guarantees about things like events, logs and metrics.

As far as I'm aware k/k only has guarantees on metrics (but they also have some sort of alpha/beta/GA framework for metrics).

I'm not sure if it is a good idea to provide guarantees on events, this seems pretty similar to logs to me. I also wonder how a stability guarantee for events look like:

• the object we emit it for
• the type (warning, ..)
• the reason and message
• at which time, under which circumstances and how often we emit it
  • This might make it very hard / impossible to change our Reconcilers going forward.

I think we have our CRDs as APIs for a reason and that is that users can rely on a stable API when they use Cluster API while they don't have to care about the implementation behind the APIs and we are flexible to change/improve it over time. I wonder if guaranteeing stability of events might be one step to far where we "lock down" the implementation of our APIs (the reconcilers) too far.

[richardcase]

Thanks for the replied @fabriziopandini & @sbueringer . Great answers and i think that has clarified categorically what we guarantee with our Go API.

I had missed this crucial part in the docs:

These guarantees extend to all code exposed in our Go Module, including types from dependencies in public APIs. Types and functions not in public APIs are not considered part of the guarantee.

[richardcase]

With events specifically.

this falls more on the API guarantees bucket than in the golang guarantee bucket

I also think this falls more into the API guantee bucket.

I'm not sure if it is a good idea to provide guarantees on events, this seems pretty similar to logs to me.

I've also been thinking about this. I have a slightly different take. For me i see logs and events as different because the consumers/usage of each is different. But perhaps my thinking is wrong 😉 I'll try to explain.

In Kubernetes event reasons are exported constants, so these can be imported by other applications and acted upon. I couldn't imagine that Kubernetes would change the Failed container event reason as it would probably break a whole bunch of stuff internally and externally with alerting systems etc.

I see stable event reasons as easier for systems to reason & integrate with compared to logs that have been traditionally more free form (obviously structured logging does help here).

I'll put together a WIP PR in CAPA to see what it would look like, this may help discussion.

[sbueringer]

Yup agree. Events are definitely more structured than logs and thus easier to provide guarantees for.

I'm not super concerned about guaranteeing a stable object / type / reason for an event. We should be probably be more flexible with message.

	// This should be a short, machine understandable string that gives the reason
	// for the transition into the object's current status.
	// TODO: provide exact specification for format.
	// +optional
	Reason string `json:"reason,omitempty" protobuf:"bytes,3,opt,name=reason"`

	// A human-readable description of the status of this operation.
	// TODO: decide on maximum length.
	// +optional
	Message string `json:"message,omitempty" protobuf:"bytes,4,opt,name=message"`

The most tricky part for me is to guarantee when exactly an event is emitted as this basically guarantees that we don't change part of our reconcile code. But this might differ between various events. Maybe it's all pretty straightforward.

[sbueringer]

One additional point. I think we should think about the "lifecycle" of an event:

• How to we introduce a new event (e.g. do we need some sort of alpha/beta/GA)
• How do we deprecate and remove an event

Basically how do we communicate to our users the stability / state of the event

[richardcase]

Good points @sbueringer, especially around the lifecycle.