<Assume_role_credentials> throw MissingCredentialsError #246

purva1192 · 2018-10-12T20:21:24Z

I am running fluentd in a Kubernetes cluster. When configuring the s3_out plugin, instead of providing the access_key and the secret_key, I used the <assume_role_credentials>. My s3 configuration looks like this -

<match s3.logs>
  @type s3
  @log_level debug
  <assume_role_credentials>
    role_arn arn:aws:iam::1111111111111:role/fluentd-test-role
    role_session_name Admin
  </assume_role_credentials>
  s3_bucket "sumo-log"
  s3_region "us-west-2"
  path logs/
  time_slice_format %Y%m%d%H%M
  utc
  auto_create_bucket true
</match>

However, I get the following error

2018-10-12 18:50:25 +0000 [info]: #0 starting fluentd worker pid=30 ppid=1 worker=0
2018-10-12 18:50:25 +0000 [error]: #0 unexpected error error_class=Aws::Errors::MissingCredentialsError error="unable to sign request without credentials set"
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/signature_v4.rb:72:in `sign_request'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/signature_v4.rb:112:in `apply_signature'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/signature_v4.rb:65:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/retry_errors.rb:138:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/query/handler.rb:28:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/request.rb:70:in `send_request'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-sts/client.rb:486:in `assume_role'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/assume_role_credentials.rb:49:in `refresh'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/refreshing_credentials.rb:20:in `initialize'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/assume_role_credentials.rb:40:in `initialize'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-s3-1.1.4/lib/fluent/plugin/out_s3.rb:426:in `new'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-s3-1.1.4/lib/fluent/plugin/out_s3.rb:426:in `setup_credentials'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-s3-1.1.4/lib/fluent/plugin/out_s3.rb:198:in `start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:165:in `block in start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:153:in `each'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:153:in `block in lifecycle'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:140:in `each'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:140:in `lifecycle'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:164:in `start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/engine.rb:274:in `start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/engine.rb:219:in `run'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:774:in `run_engine'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:523:in `block in run_worker'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:699:in `main_process'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:518:in `run_worker'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/bin/fluentd:8:in `<top (required)>'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/local/bin/fluentd:22:in `load'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/local/bin/fluentd:22:in `<main>'
2018-10-12 18:50:25 +0000 [error]: #0 unexpected error error_class=Aws::Errors::MissingCredentialsError error="unable to sign request without credentials set"
  2018-10-12 18:50:25 +0000 [error]: #0 suppressed same stacktrace
2018-10-12 18:50:25 +0000 [info]: Worker 0 finished unexpectedly with status 1

I should be able to access AWS without providing the access key and secret key.

Can anybody provide more details about it?

The text was updated successfully, but these errors were encountered:

repeatedly · 2018-10-15T08:03:05Z

Does anyone have an idea for this error?
On my environment, <assume_role_credentials> works corretly and I first see "unable to sign request without credentials set" error.
So I'm not sure how to resolve this error.
Missing credentials information on the node?

jiachenzhong · 2018-10-17T07:17:13Z

I am also the same mistake, only AK+SK can use.

Sphyrna-029 · 2019-01-25T21:41:10Z

Was a solution ever found for this? Experiencing the same issue when trying to use instance profile credentials.

repeatedly · 2019-01-28T23:06:48Z

We need help from AWS expert to investigate deeply.
For example, this error happens when no ~/.aws/credentials and no aws related environment variable.
I'm not sure full spec of AWS client but this comments show how to process credentials: https://github.com/aws/aws-sdk-ruby/blob/97b28ccf18558fc908fd56f52741cf3329de9869/gems/aws-sdk-core/lib/aws-sdk-core/plugins/credentials_configuration.rb#L42

Sphyrna-029 · 2019-01-29T04:40:09Z

Solved my issue today. Did not have enough permissions for the plugin to access the bucket. Minimum requirements seem to be list bucket and put.

repeatedly · 2019-01-29T09:00:34Z

For that case, the error should be AccessDenied. Did you get MissingCredentialsError instead of AccessDenied with lack of permission?

bhaveshmodiatekaplus · 2019-11-28T07:46:43Z

After giving ListBucket and putobject permission also , we are facing same issue.
Can any one please tell minimum permission required. Thanks in advance :)

dmanchikalapudi · 2020-08-11T18:55:20Z

Did anyone figure out the issue? I see the same error trying to push logs to cloudwatch. I am running fluentd as a sidecar in fargate pod.

mayankcpdixit · 2020-08-13T09:26:51Z

Workaround I used:
I replaced Assume_role_credentials with instance_profile_credentials providing metadata ip/port for fetching default AWS creds:

<instance_profile_credentials>
  ip_address 169.254.169.254
  port 80
</instance_profile_credentials>

My pods had enough permissions. I had kube2iam(ref/more ref) installed and the pod had the right annotation.

Problem I was facing:
After ssh-ing aws s3 ls s3:// was working but the fluent-plugin-s3 plugin wasn't able to publish and threw:

unexpected error error_class=Aws::Errors::MissingCredentialsError error="unable to sign request without credentials set"

This suggests that it might not be a permission issue but some SDK usage related issue.

Hope it helps someone.
In case it still doesn't work, also check out: https://github.com/ccpgames/kubernetes-fluentd-s3

pinkavaj · 2021-02-10T12:30:12Z

In my case I have found out I need to disable containerSecurityContext in the helm chart from which is the fluentd installed to allow EC2 instance authentication.

gpodila-disc · 2021-07-17T16:09:30Z

Did anyone found the solution? Im facing the same problem but while writing into AWS Elastic search.

pawanyoda · 2021-07-30T07:10:15Z

I am facing also this error:
Without IAM configuration in fluentd.conf
Error:

[error]: #0 unexpected error error_class=Aws::Sigv4::Errors::MissingCredentialsError error="missing credentials, provide credentials with one of the following options:\n  - :access_key_id and :secret_access_key\n  - :credentials\n  - :credentials_provider"

After added assume role configuration:
Error:

[error]: #0 unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::Errors::MissingCredentialsError: unable to sign request without credentials set>"

But pod have already IAM role assigned and I can able to copy data to my bucket via aws s3 cp file s3://bucket --region us-east-1

Conf:

<match access_log>
  @type copy
  <store>
    @type elasticsearch_dynamic
    host "host"
    port "80"
    scheme http
    ssl_verify false
    type_name index_name
    logstash_format true
    logstash_prefix index_name
    logstash_dateformat %Y-%m-%d
    include_tag_key true
    time_key time
    time_key_format "%FT%T.%L%z"
    reload_connections false
  </store>
  <store>
    @type s3
    @log_level debug
    s3_bucket "bucketname"
    s3_region "us-east-1"
    path logs/
    s3_object_key_format "%{path}/${tag}/%Y-%m-%d/%{time_slice}_%{index}.%{file_extension}"
    storage_class STANDARD                
    utc true
    <buffer tag,time>
      @type memory
      chunk_limit_size 8MB
      total_limit_size 512MB
      chunk_full_threshold 0.9
      compress gzip
      timekey_wait 1m
      timekey_use_utc true
      flush_mode interval
      flush_interval 60s
      flush_thread_count 2
    </buffer>
    <format>
      @type json
    </format>                
  </store>                            
</match>

yangjiel · 2023-02-16T04:37:24Z

I suppose even without any credential given, the plugin should not crash the fluentd.

sparampalli · 2023-09-19T10:00:12Z

Do we have any solutions for this issue? Use of roles is more secure than access keys.

repeatedly added the help wanted Need help from users label Oct 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

<Assume_role_credentials> throw MissingCredentialsError #246

<Assume_role_credentials> throw MissingCredentialsError #246

purva1192 commented Oct 12, 2018

repeatedly commented Oct 15, 2018 •

edited

Loading

jiachenzhong commented Oct 17, 2018

Sphyrna-029 commented Jan 25, 2019

repeatedly commented Jan 28, 2019 •

edited

Loading

Sphyrna-029 commented Jan 29, 2019

repeatedly commented Jan 29, 2019

bhaveshmodiatekaplus commented Nov 28, 2019

dmanchikalapudi commented Aug 11, 2020 •

edited

Loading

mayankcpdixit commented Aug 13, 2020 •

edited

Loading

pinkavaj commented Feb 10, 2021

gpodila-disc commented Jul 17, 2021

pawanyoda commented Jul 30, 2021 •

edited

Loading

yangjiel commented Feb 16, 2023

sparampalli commented Sep 19, 2023

<Assume_role_credentials> throw MissingCredentialsError #246

<Assume_role_credentials> throw MissingCredentialsError #246

Comments

purva1192 commented Oct 12, 2018

repeatedly commented Oct 15, 2018 • edited Loading

jiachenzhong commented Oct 17, 2018

Sphyrna-029 commented Jan 25, 2019

repeatedly commented Jan 28, 2019 • edited Loading

Sphyrna-029 commented Jan 29, 2019

repeatedly commented Jan 29, 2019

bhaveshmodiatekaplus commented Nov 28, 2019

dmanchikalapudi commented Aug 11, 2020 • edited Loading

mayankcpdixit commented Aug 13, 2020 • edited Loading

pinkavaj commented Feb 10, 2021

gpodila-disc commented Jul 17, 2021

pawanyoda commented Jul 30, 2021 • edited Loading

yangjiel commented Feb 16, 2023

sparampalli commented Sep 19, 2023

repeatedly commented Oct 15, 2018 •

edited

Loading

repeatedly commented Jan 28, 2019 •

edited

Loading

dmanchikalapudi commented Aug 11, 2020 •

edited

Loading

mayankcpdixit commented Aug 13, 2020 •

edited

Loading

pawanyoda commented Jul 30, 2021 •

edited

Loading