spring-cloud-starter-aws-secrets-manager-config does not work with spring-cloud-starter-kubernetes-client-config

[dibyajyotibehera]

I am trying to use aws-secrets-manager for secrets and kubernetes-client-config for app configuration.
aws-secrets-manager works fine in isolation , however when i add the kubernetes-client-config dependency the app fails.
aws-secrets-manager for some reason tries to read a non existent secret (/secret/null_kubernetes) and fails with below error

aws-sdk-java-1651158580786 is not authorized to perform: secretsmanager:GetSecretValue on resource: /secret/null_kubernetes because no identity-based policy allows the secretsmanager:GetSecretValue action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException;

[maciejwalkowiak]

Do you mean to load this secret /secret/null_kubernetes or its an unwanted side effect of adding kubernetes-client-config? I guess the latter but want to make sure.

[dibyajyotibehera]

yes it is unwanted side effect and application fails to load due to that.
I have replicated the issue in a branch here - https://github.com/dibyajyotibehera/spring-cloud-k8s-demo/blob/aws_secrets_issue/app-deployment.yaml

what i am trying to do is load the aws secret from config map instead from app. yaml.

my contents of config map -

apiVersion: v1
kind: ConfigMap
metadata:
    name: spring-cloud-k8s-demo
data:
    application.yaml: |-
        messageProp: k8s config map7
        spring:
            config:
                    import: aws-secretsmanager:test-secret

this fails with error -

2022-04-29 03:15:07.445 INFO [spring-cloud-k8s-demo,,] 1 --- [ main] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-configmap.spring-cloud-k8s-demo.default'}]
2022-04-29 03:15:07.452 INFO [spring-cloud-k8s-demo,,] 1 --- [ main] s.AwsSecretsManagerPropertySourceLocator : Loading secrets from AWS Secret Manager secret with name: /secret/null_kubernetes, optional: false
2022-04-29 03:15:08.330 ERROR [spring-cloud-k8s-demo,,] 1 --- [ main] o.s.boot.SpringApplication : Application run failed

io.awspring.cloud.secretsmanager.AwsSecretsManagerPropertySources$AwsSecretsManagerPropertySourceNotFoundException: com.amazonaws.services.secretsmanager.model.ResourceNotFoundException: Secrets Manager can't find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 0270f925-cd16-4bb3-b0e2-8047af614bba; Proxy: null)

[dibyajyotibehera]

Thanks @maciejwalkowiak . I added bootstrap.properties . app now seems to get past the failure , however it does not seem to be trying to load the secret mentioned in configmap. Is this supported ? . I can see other secret name in the logs but not the one i mentioned in configmap.

2022-04-29 04:55:49.028  INFO 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Loading secrets from AWS Secret Manager secret with name: /secret/null_kubernetes, optional: true
2022-04-29 04:55:51.102  WARN 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Unable to load AWS secret from /secret/null_kubernetes. Secrets Manager can't find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 605cbeac-8654-48b2-9162-ac28743a290e; Proxy: null)
2022-04-29 04:55:51.103  INFO 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Loading secrets from AWS Secret Manager secret with name: /secret/null, optional: true
2022-04-29 04:55:51.259  WARN 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Unable to load AWS secret from /secret/null. Secrets Manager can't find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 38989748-d2f1-4eea-8ad4-a315b5931901; Proxy: null)
2022-04-29 04:55:51.260  INFO 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Loading secrets from AWS Secret Manager secret with name: /secret/application_kubernetes, optional: true
2022-04-29 04:55:51.434  WARN 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Unable to load AWS secret from /secret/application_kubernetes. Secrets Manager can't find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: c32a2747-fb6e-4f37-aa66-a03b499ee9f2; Proxy: null)
2022-04-29 04:55:51.435  INFO 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Loading secrets from AWS Secret Manager secret with name: /secret/application, optional: true
2022-04-29 04:55:51.729  WARN 1 --- [           main] s.AwsSecretsManagerPropertySourceLocator : Unable to load AWS secret from /secret/application. Secrets Manager can't find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 98afe79a-2ecc-4ec2-983d-92142aa5015d; Proxy: null)
2022-04-29 04:55:52.563  INFO 1 --- [           main] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-configmap.spring-cloud-k8s-demo.default'}]

Config map props does seem to get loaded but much later in the startup sequence . It does load up other props from config map but not aws secrets .

2022-04-29 04:55:56.666 DEBUG 1 --- [ main] ySourcesPropertyResolver$DefaultResolver : Found key 'messageProp' in PropertySource 'bootstrapProperties-configmap.spring-cloud-k8s-demo.default' with value of type String

[dibyajyotibehera]

@maciejwalkowiak - could you please confirm if aws-secrets-manager properties are supported via configmaps or not . Thanks.

[maciejwalkowiak]

To be honest I am not sure and don't have time to verify. Do other spring.config.import implementations work if you put them to K8s Config Map? My gut feeling is that config maps are resolved after spring.config.import so it's already too late.

[dibyajyotibehera]

Yes , thats what the logs indicate . Just wanted to check though in case there are any known work around for this . Thanks for clarifying

[dibyajyotibehera]

@maciejwalkowiak - I can attempt a PR for disabling spring-cloud-starter-bootstrap for 2.4 . Is there any good place in the code base i can start looking at ?