HashiCorp VaultSecretProvider Reauthentication and Max Token TTL

[danielhass]

Dear Maintainers,

I'm currently running into an issue with the VaultSecretProvider that from my current understanding is subject to a design flaw.

We are using the AppRole authentication method that allows the Vault client to generate a token with a max TTL of 60m. After the Kafka Connect startup the VaultSecretProvider works fine for 60 minutes and afterwards starts to throw the following error:

WARN Failed to renew the Kerberos ticket (io.lenses.connect.secrets.async.AsyncFunctionLoop:43)
   com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
   Response body: {"errors":["permission denied"]}

From my understanding this is caused by the AsyncFunctionLoop that tries to renew the Vault token on a fixed period (default 60000ms). Due to the fact that our current AppRole is setup to generate non periodic tokens (which is the default), these tokens are not renewable indefinitely.

Docs Ref HashiCorp: https://www.vaultproject.io/docs/concepts/tokens#periodic-tokens

For me the current VaultSecretProvider looks like it only supports the usage of periodic tokens (which are kind a special in the Vault universe). Standard service tokens that are subject to a max TTL should all run into the same problem sooner or later.

We have several Vault Agents running in our infrastructure which I would consider somethign like a "reference implementation" for automatic daemon-like token renewals. These agents renew the service tokens until there max TTL is reached and issue a reauthentication afterwards to acquire a fresh token. This however doesn't seem to be possible with the VaultSecretProvider provider as it only authentications once on the providers configure() call.

I would like to get some insights from other users and/or the maintainers on this problem and how they handle it. I would like to make sure that I'm not missing something here.

[usopp]

Same problem here. We use approle auth and running in the same error.

[mcl085]

Hi, I have the same problem here. I'm using Kubernetes role implementation.

Does anybody have some insights about that?

[2024-01-23 18:02:48,461] WARN Failed to run function Vault Token Renewal (io.lenses.connect.secrets.async.AsyncFunctionLoop:44)
io.github.jopenlibs.vault.VaultException: Vault responded with HTTP status code: 403
Response body: {"errors":["permission denied"]}