[aws]: inspector2/guarduty/securityhub - no role_arn or session_token

[jdnurmi]

Integration Name

AWS [aws]

Dataset Name

guardduty

Integration Version

v2.22.1

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

(Cloud)

Software/API Version

No response

Error Message

{"log.level":"error","@timestamp":"2024-08-13T19:58:17.344Z","message":"Error while processing http request: failed to collect first response: failed to execute http POST: server responded with status code 403: {\"message\":\"The security token included in the request is invalid.\"}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-31962c63-cb07-4dfd-8008-c45dda472331","type":"httpjson"},"log":{"source":"httpjson-31962c63-cb07-4dfd-8008-c45dda472331"},"log.origin":{"file.line":181,"file.name":"httpjson/input.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.run.func1"},"log.logger":"input.httpjson-cursor","service.name":"filebeat","id":"httpjson-aws.guardduty-23af25d5-57be-472e-84d1-85d7bea197ff","input_source":"https://guardduty.us-east-2.amazonaws.com/detector/18c0b8b5740bb519ba47e3361517d029/findings","input_url":"https://guardduty.us-east-2.amazonaws.com/detector/18c0b8b5740bb519ba47e3361517d029/findings","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Same for inspector2 and security hub.

Best I can determine, if

integrations/packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs

Line 37 in e6362aa

value: '[[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate ($now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate ($now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.{{tld}}\n" (sprintf "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'

is the source of signing, that's wrong for ephemeral credentials (missing x-amz-security-token / SESSION_TOKEN) , which I discovered while trying to use an assumed role (from valid instance credentials) - it doesn't appear that signing method takes role_arn into account at all either.

Event Original

No response

What did you do?

Nodes in EKS, have a valid pod role assigned; Want this integration to use those credentials to assume role_arn into a foreign (in-org) account to access securityhub and other services.

What did you see?

Nothing, but digging the logs yielded the above

What did you expect to see?

Happy security dashboards!

Anything else?

No response

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[riezebosch]

Facing the same issue in custom logs integration:

Timestamp	event.dataset	Message
11:43:04.152	elastic_agent.filebeat	[elastic_agent.filebeat][error] Error while processing http request: failed to collect first response: failed to execute http POST: server responded with status code 403: {"message":"The security token included in the request is invalid."}

[kcreddy]

This is applicable for following datastreams: SecurityHub (Findings and Insights), Inspector, and GuardDuty.

Signing with temporary credentials is not supported out-of-the-box for HTTPJSON input. It would need a rewrite with chain step calls.

A similar idea with CEL input is discussed here: elastic/beats#40762 (comment). At each interval, the CEL program could make STS API calls to request temporary credentials at the beginning of execution, and then use those temporary credentials for subsequent calls to SecurityHub or Inspector. Alternately, we could have AWS SDK handle the temporary credentials refresh, like how assume_role.expiry_window works with AWS S3 input. The SDK option is not available in CEL input yet.

Outlining some options:

1. Migrate to CEL input and handle temporary credentials by making calls to STS API at the beginning of execution.
2. Migrate to CEL input and allow the SDK to handle the credentials expiry (CEL input enhancement needed).
3. Use httpjson to first make STS API call to get temporary credentials and use them inside the chain step (corresponding data API calls).

Option 2 could be ideal way to solve this allowing SDK to control the credential refresh. But it would need a prior enhancement to CEL input to make AWS SDK handle it. Options 1 and 3 are faster to implement.

[kcreddy]

Hey @andrewkroh, WDYT of the above approaches as it seems to be similar scenario noted in elastic/beats#40762 (comment)