Change cache scope dynamically

[Mugenor]

Feature Request

Describe the problem you need a feature to resolve.

We are creating multi-tenant K8S cluster and using HNC to structure tenants namespaces. We have a single root namespace, and a subnamespace for each tenant. Tenants can create their own subnamespaces if they need (for app deployment or testing).

We are working on an operator that manages CRs in tenants namespaces, meaning that it has to work with all namespaces in the hierarchy that is changes constantly throughout a day.

The current options we have:

1. Configure operator cache to monitor the whole cluster. But we are constrained in terms of RBAC, so we can give it access only to the tenant NS hierarchy.
2. It's possible to configure operator to monitor subset of NSs, the problem is that this subset is static and have to be configured at the startup, so it requires us to restart the operator every time namespace is created or deleted. While, this is technically working, it's very noisy and makes the operator restart pretty often.
3. It's possible to create a cache wrapper that will restart just the cache instance in runtime when namespace is created or deleted and use it with NewCache option. Again, it's pretty noisy and while it helps to avoid operator restarts, it triggers reconciliation of everything, logs pollutes with errors from the controller runtime because of stopped cache.
4. Write our own implementation of cache that will satisfy our requirements. This is error prone and it looks like it's not possible to reuse some code bits related to cache from the library, because almost everything is private, meaning that it will require huge duplication of code that already exists in the framework.

So, none of the options is really perfect.

Describe the solution you'd like.

It would be nice to have some kind of API to tweak scope of the cache.

I think it's reasonable to make the users code to decide when and how the cache scope should change. For example, I as a operator developer can decide how I watch for events in cluster and depending on custom conditions change update the cache scope.

I assume namespace granularity would suffice most of use cases.

A lot of operator implementations allows to specify the static list of watched namespaces, but not dynamic because of this issue. It will enable them for more flexible use.

P.S.: Looks like it's already implemented in java version of operator sdk, so it would be nice to have for go version as well.