From 9b337f8ef95e1ae6db036febeb65ea7167ad8b9b Mon Sep 17 00:00:00 2001 From: Anton Gilgur Date: Fri, 20 Sep 2024 13:59:51 -0400 Subject: [PATCH] docs: copy-edit of "Configuring S3" section - this is a partial cherry-pick of 19b2322e26127145a895ec46931efebfd1b0f3b5, specifically only the docs changes to _existing_ features - they are a substantial improvement over the previous docs, IMO Co-authored-by: Raymond Chow Signed-off-by: Anton Gilgur --- docs/configure-artifact-repository.md | 122 +++++++++++++++++++------- 1 file changed, 91 insertions(+), 31 deletions(-) diff --git a/docs/configure-artifact-repository.md b/docs/configure-artifact-repository.md index 8c4f227301f5..70f9f62defd0 100644 --- a/docs/configure-artifact-repository.md +++ b/docs/configure-artifact-repository.md @@ -75,16 +75,20 @@ artifacts: ## Configuring AWS S3 -Create your bucket and access keys for the bucket. AWS access keys have the same -permissions as the user they are associated with. In particular, you cannot -create access keys with reduced scope. If you want to limit the permissions for -an access key, you will need to create a user with just the permissions you want -to associate with the access key. Otherwise, you can just create an access key -using your existing user account. +First, create a bucket: ```bash -$ export mybucket=bucket249 -$ cat > policy.json < policy.json < policy.json < access-key.json ``` -If you do not have Artifact Garbage Collection configured, you should remove `s3:DeleteObject` from the list of Actions above. +If you do not have [Artifact Garbage Collection](walk-through/artifacts.md#artifact-garbage-collection) configured, you should remove `s3:DeleteObject` from the list of Actions above. -NOTE: if you want argo to figure out which region your buckets belong in, you -must additionally set the following statement policy. Otherwise, you must -specify a bucket region in your workflow configuration. + -```json - { - "Effect":"Allow", - "Action":[ - "s3:GetBucketLocation" - ], - "Resource":"arn:aws:s3:::*" - } - ... -``` +!!! Note "Region discovery" + + Argo can discover the region of your buckets with the additional policy below. + Without this, you must specify the region in your artifact configuration. + + ```json + { + "Effect":"Allow", + "Action":[ + "s3:GetBucketLocation" + ], + "Resource":"arn:aws:s3:::*" + } + ... + ``` + + + +#### AWS S3 IRSA -### AWS S3 IRSA +IAM Roles for Service Accounts (IRSA) is the recommended Kubernetes native mechanism to authenticate to S3. +If you are using EKS, follow [the IRSA setup guide](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). +If not, follow the [Pod Identity Webhook self-hosted setup guide](https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md). -If you wish to use S3 IRSA instead of passing in an `accessKey` and `secretKey`, you need to annotate the service account of both the running workflow (in order to save logs/artifacts) and the argo-server pod (in order to retrieve the logs/artifacts). +With the bucket and policy as described above, create an IAM role and add the policy: + +```bash +aws iam create-role --role-name $mybucket-role +aws iam put-role-policy --role-name $mybucket-user --policy-name $mybucket-policy --policy-document file://policy.json +``` + +Attach this IAM role to a service account with an annotation: ```yaml apiVersion: v1 kind: ServiceAccount metadata: annotations: - eks.amazonaws.com/role-arn: arn:aws:iam::012345678901:role/mybucket + eks.amazonaws.com/role-arn: arn:aws:iam::012345678901:role/mybucket-role name: myserviceaccount namespace: mynamespace ``` +Use the service account in a workflow: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Workflow +spec: + serviceAccountName: myserviceaccount +``` + +#### AWS S3 with IAM Access Keys + +!!! Note "Least privilege user" + To reduce the privileges of an access key, create a user with only the necessary permissions and no more. + +With the bucket and policy described above, create an IAM user and add the policy: + +```bash +aws iam create-user --user-name $mybucket-user +aws iam put-user-policy --user-name $mybucket-user --policy-name $mybucket-policy --policy-document file://policy.json +aws iam create-access-key --user-name $mybucket-user > access-key.json +``` + +Configure an artifact with the access keys: + +```yaml +artifacts: + - name: my-output-artifact + path: /my-output-artifact + s3: + endpoint: s3.amazonaws.com + bucket: my-s3-bucket + key: path/in/bucket/my-output-artifact.tgz + # The following fields are secret selectors. + # They reference the k8s secret named 'my-s3-credentials'. + # This secret is expected to have the keys 'accessKey' and 'secretKey', + # containing the base64 encoded credentials to the bucket. + accessKeySecret: + name: my-s3-credentials + key: accessKey + secretKeySecret: + name: my-s3-credentials + key: secretKey +``` + ## Configuring GCS (Google Cloud Storage) Create a bucket from the GCP Console @@ -319,7 +379,7 @@ metadata: namespace: rrsa-demo annotations: pod-identity.alibabacloud.com/role-name: $your_ram_role_name - + --- apiVersion: v1 kind: ConfigMap @@ -385,7 +445,7 @@ artifacts: # useSDKCreds: true accountKeySecret: name: my-azure-storage-credentials - key: account-access-key + key: account-access-key ``` If `useSDKCreds` is set to `true`, then the `accountKeySecret` value is not