-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[falcoctl artifact follow]: can't handle or refresh ECR token after initial artifact pull #325
Comments
Hi @CarpathianUA, thanks for the detailed issue. Falcoctl caches the authentication token, and it seems that is not able to refresh it after the expiration. By disabling the internal cache it works, but at every request it will authenticate to the remote repository. The authentication is based on https://github.com/oras-project/oras-credentials-go module. I will take some time to investigate further. |
Hi @alacuku , thank you for the quick reply! I'll really appreciate this needed functionality to be implemented in a next |
Hey @CarpathianUA, the fix is here #326. I tested it locally, but would really appreciate it if you could test it in your environment. |
@alacuku I can confirm that in the latest |
@CarpathianUA CarpathianUA May I ask how did You modify the helm chart in order to add the amazon-ecr-credential-helper for installer and follower containers? Did You fork/clone the repo and modified? |
Hi, you don't need to modify anything - official chart allows to add init and sidecars containers, as well as to add custom volumes and volume mounts. So you can download ecr binary with init container under the path with shared volume, and then mount this volume to install and follow containers. |
@CarpathianUA - My faloctl was able to pull rules from ECR but after some time, I started getting errors with empty credentials to ECR. After recreating Service Account and restarting pod, it started working. Is it possible that it is connected to the issue mentioned above by You? on which level did You setup the AWS_ECR_DISABLE_CACHE? Just on for falcoctl or for artifacts or for config? |
@robert-pudlowski-mox
for both |
What happened:
falcoctl artifact follow
is not able to refresh an ECR auth token when working withamazon-ecr-credential-helper
binary.What you expected to happen:
falcoctl artifact follow
is able to leverageamazon-ecr-credential-helper
binary correctly to auth against ECR while following an artifact since 12 hours after initial auth against ECR.How to reproduce it (as minimally and precisely as possible):
Environment: AWS EKS,
falco
deployed with a Helm chart. Latest version ofamazon-ecr-credential-helper
added to bothfalcoctl-artifact-install
andfalcoctl-artifact-follow
containers. IAM role forfalco
service account is bound to the pod with all necessary permissions to pull the artifact from ECR, as well as to get auth token (ecr:GetAuthorizationToken
to*
), no static credentials and tokens are in place.falcoctl-artifact-install
initContainer is able to pull an artifact.falcoctl-artifact-follow
sidecar container is following an artifact, and can pull it during 12 hours. After 12 hoursfalcoctl-artifact-follow
can't pull the artifact:What's interesting, that when I
exec
intofalcoctl-artifact-follow
container, I'm able to pull an artifact even after 12 hours with/usr/bin/falcoctl-bin registry pull <REDACTED>.dkr.ecr.eu-west-1.amazonaws.com/falco-rules:master
, so it means that at leastregistry pull
can work correctly withamazon-ecr-credential-helper
besides the fact, that initial auth was 12 hours ago (default and maximum time of ECR token lifecycle before expiration), and can retrieve a new token viaamazon-ecr-credential-helper
:Configuration that I use:
.docker/config.json
:Caching of token is disabled, sdk load config as well. I've tried with cache and sdk load config enabled as well - behaviour for
falcoctl-artifact-follow
is still the same:AWS IAM role policy for service account:
~/.ecr/log/ecr-login.log
. I'm expecting at least some records there as well whenfalcoctl-artifact-follow
is expected to retrieve new token after 12 hours:I've tried all possible combinations of
.docker/config.json
, env vars likeAWS_ECR_DISABLE_CACHE
andAWS_SDK_LOAD_CONFIG
- no luck.Regarding the fact that
falcoctl registry pull
can correctly leverageamazon-ecr-credential-helper
even after 12 hours and pull an artifact (and it means that leveraging ofamazon-ecr-credential-helper
forfalcoctl registry pull
works as expected) , I assume that the issue is behindfalcoctl artifact follow
controller / method specifically.@alacuku Would appreciate any suggestions and solutions there. Thanks in advance!
The text was updated successfully, but these errors were encountered: