Customize sync option values to be more granular than true/false/confirm

[andrii-korotkov-verkada]

Summary

I suggest to do adding more possible values and deprecating some of existing ones (but still keeping them around for backwards compatibility). The completely new values can be like if-requested for Prune and Replace (e.g. by checking a checkbox in UI or by autosync with a corresponding options) or if-immutable-fields-updated or if-apply-failed-for-any-reason for Replace. To have a more self-explanatory system, true/false should be deprecated in favor of always/never.

Motivation

A discussion on #14161 and #20720 surfaced the limitations of the existing sync with replace. As it is, replace can be quite dangerous. It's inflexible - when you request sync with replace from UI, all out-of-sync resources would get replaced even if they don't have to be. Controlling permissions require either changes to RBAC (using existing delete permissions is not good enough) or a different sync options system. Given that it's more of a safety than security concern, the later can be a better solution.

Proposal

All sync option keys (like Prune and Replace) would support the following values:

• always - same as true
• never - same as false
• confirm - stays the same
• if-requested - same as true if the corresponding option was set in UI/CLI or by autosync config.
• (deprecated) true - stays the same
• (deprecated) false - stays the same

Replace and Force would also support:

• if-immutable-fields-updated - same as true if the first or dry-run apply failed due to an attempt to update immutable fields.
• if-apply-failed-for-any-reason - same as true if the first or dry-run apply failed for any reason.

Setting options on the application level would still take effect on all resources (if not overridden).

The default value for the keys supporting a setting via UI/CLI would be if-requested to begin with, but should be configurable via a configmap option like controller.sync.replace.default, so people who want to restrict dangerous options by default can do so, while still having a flexibility to use annotations to override the default.
The default value for keys not configurable like that the default value would be always or never, same as current default values of true or false.
I hope this gives enough controls to solve the problem well while keeping backwards compatibility and flexibility.