AWS S3 Input custom endpoint handling broken in 8.15

[strawgate]

8.15 no longer has the fix for AWS S3 endpoint handling that was present in 8.14

8.15:

beats/x-pack/filebeat/input/awss3/input.go

Lines 51 to 60 in 2f0dda8

	if config.AWSConfig.Endpoint != "" {
	// Add a custom endpointResolver to the awsConfig so that all the requests are routed to this endpoint
	awsConfig.EndpointResolverWithOptions = awssdk.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (awssdk.Endpoint, error) {
	return awssdk.Endpoint{
	PartitionID: "aws",
	URL: config.AWSConfig.Endpoint,
	SigningRegion: awsConfig.Region,
	}, nil
	})
	}

8.14:

beats/x-pack/filebeat/input/awss3/input.go

Lines 83 to 104 in 7b6cfad

	if config.AWSConfig.Endpoint != "" {
	// Parse a URL for the host regardless of it missing the scheme
	endpointUri, err := url.Parse(config.AWSConfig.Endpoint)
	if err != nil {
	return nil, fmt.Errorf("failed to parse endpoint: %w", err)
	}
	// For backwards compat:
	// If the endpoint does not start with S3, we will use the endpoint resolver to make all SDK requests use the specified endpoint
	// If the endpoint does start with S3, we will use the default resolver uses the endpoint field but can replace s3 with the desired service name like sqs
	if !strings.HasPrefix(endpointUri.Hostname(), "s3") {
	awsConfig.EndpointResolverWithOptions = awssdk.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (awssdk.Endpoint, error) {
	return awssdk.Endpoint{
	PartitionID: "aws",
	Source: awssdk.EndpointSourceCustom,
	URL: config.AWSConfig.Endpoint,
	SigningRegion: awsConfig.Region,
	}, nil
	})
	}
	}

In AWS, the endpoint field is supposed to act kind of like a "base url" where service URLs are built using the value in the endpoint field. So when the SQS client makes a request, an endpoint field of s3.us-east1.amazonaws.com is transformed into sqs.us-east1.amazonaws.com, etc.

The 8.15 code forces all endpoints to use the value in the endpoint field instead of relying on the resolver to use the endpoint to "build" each service's endpoint (s3, sqs, etc). In the example above, this would cause the SQS client to directly query s3.us-east1.amazonaws.com

Even the 8.14 code has an issue that crops up with some customers. We should likely switch to only using a custom endpoint resolver when a user explicitly tells us to, for example by introducing a new setting called "static endpoint" or something similar, that when set to true, sets the endpoint resolver as it is set currently. This would be a breaking change.

An alternative would be introducing a setting called, "dynamic_endpoint" or something similar which, when set, sets the endpoint field without using a resolver.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

#39709 was only in 8.14 and #39722 to bring it to main was never merged or backported.

[strawgate]

#39709 was only in 8.14 and #39722 to bring it to main was never merged or backported.

39722 was not supposed to merge, it was just an example for how to include it post s3/sqs refactor, ill close the PR but add that as a comment

[predogma]

Issue was hit in version Elasticsearch 8.15.2, with at least these integrations.

• Crowdstrike (FDR) 1.42.2
  section "Collect CrowdStrike Falcon Data Replicator logs (input: aws-s3) > Falcon Data Replicator logs"

• Cisco Umbrella 1.26.2.
  section "Collect logs from Cisco Umbrella > Cisco Umbrella Logs"

Workaround required deleting the value located in the "Endpoint" field

Are there any other integrations that we should be aware of that could be impacted?

[cmacknz]

Any integration that can use the aws-s3 input would be affected, here's a preliminary list I generated quickly:

❯ rg -g 'manifest.yml' 'aws-s3' -l --sort=path
packages/amazon_security_lake/data_stream/event/manifest.yml
packages/amazon_security_lake/manifest.yml
packages/aws/data_stream/apigateway_logs/manifest.yml
packages/aws/data_stream/cloudfront_logs/manifest.yml
packages/aws/data_stream/cloudtrail/manifest.yml
packages/aws/data_stream/ec2_logs/manifest.yml
packages/aws/data_stream/elb_logs/manifest.yml
packages/aws/data_stream/emr_logs/manifest.yml
packages/aws/data_stream/firewall_logs/manifest.yml
packages/aws/data_stream/guardduty/manifest.yml
packages/aws/data_stream/route53_resolver_logs/manifest.yml
packages/aws/data_stream/s3access/manifest.yml
packages/aws/data_stream/vpcflow/manifest.yml
packages/aws/data_stream/waf/manifest.yml
packages/aws/manifest.yml
packages/aws_bedrock/data_stream/invocation/manifest.yml
packages/aws_bedrock/manifest.yml
packages/aws_logs/data_stream/generic/manifest.yml
packages/aws_logs/manifest.yml
packages/canva/data_stream/audit/manifest.yml
packages/canva/manifest.yml
packages/carbon_black_cloud/data_stream/alert/manifest.yml
packages/carbon_black_cloud/data_stream/alert_v7/manifest.yml
packages/carbon_black_cloud/data_stream/endpoint_event/manifest.yml
packages/carbon_black_cloud/data_stream/watchlist_hit/manifest.yml
packages/carbon_black_cloud/manifest.yml
packages/cisco_umbrella/data_stream/log/manifest.yml
packages/cisco_umbrella/manifest.yml
packages/cloudflare_logpush/data_stream/access_request/manifest.yml
packages/cloudflare_logpush/data_stream/audit/manifest.yml
packages/cloudflare_logpush/data_stream/casb/manifest.yml
packages/cloudflare_logpush/data_stream/device_posture/manifest.yml
packages/cloudflare_logpush/data_stream/dns/manifest.yml
packages/cloudflare_logpush/data_stream/dns_firewall/manifest.yml
packages/cloudflare_logpush/data_stream/firewall_event/manifest.yml
packages/cloudflare_logpush/data_stream/gateway_dns/manifest.yml
packages/cloudflare_logpush/data_stream/gateway_http/manifest.yml
packages/cloudflare_logpush/data_stream/gateway_network/manifest.yml
packages/cloudflare_logpush/data_stream/http_request/manifest.yml
packages/cloudflare_logpush/data_stream/magic_ids/manifest.yml
packages/cloudflare_logpush/data_stream/nel_report/manifest.yml
packages/cloudflare_logpush/data_stream/network_analytics/manifest.yml
packages/cloudflare_logpush/data_stream/network_session/manifest.yml
packages/cloudflare_logpush/data_stream/sinkhole_http/manifest.yml
packages/cloudflare_logpush/data_stream/spectrum_event/manifest.yml
packages/cloudflare_logpush/data_stream/workers_trace/manifest.yml
packages/cloudflare_logpush/manifest.yml
packages/crowdstrike/data_stream/fdr/manifest.yml
packages/crowdstrike/manifest.yml
packages/f5_bigip/data_stream/log/manifest.yml
packages/f5_bigip/manifest.yml
packages/imperva_cloud_waf/data_stream/event/manifest.yml
packages/imperva_cloud_waf/manifest.yml
packages/jamf_protect/data_stream/alerts/manifest.yml
packages/jamf_protect/data_stream/telemetry/manifest.yml
packages/jamf_protect/data_stream/telemetry_legacy/manifest.yml
packages/jamf_protect/data_stream/web_threat_events/manifest.yml
packages/jamf_protect/data_stream/web_traffic_events/manifest.yml
packages/jamf_protect/manifest.yml
packages/lyve_cloud/data_stream/audit/manifest.yml
packages/lyve_cloud/manifest.yml
packages/sentinel_one_cloud_funnel/data_stream/event/manifest.yml
packages/sentinel_one_cloud_funnel/manifest.yml
packages/sublime_security/data_stream/audit/manifest.yml
packages/sublime_security/data_stream/email_message/manifest.yml
packages/sublime_security/data_stream/message_event/manifest.yml
packages/sublime_security/manifest.yml
packages/symantec_endpoint_security/data_stream/event/manifest.yml
packages/symantec_endpoint_security/manifest.yml
packages/tanium/data_stream/action_history/manifest.yml
packages/tanium/data_stream/client_status/manifest.yml
packages/tanium/data_stream/discover/manifest.yml
packages/tanium/data_stream/endpoint_config/manifest.yml
packages/tanium/data_stream/reporting/manifest.yml
packages/tanium/data_stream/threat_response/manifest.yml
packages/tanium/manifest.yml
packages/trellix_edr_cloud/data_stream/event/manifest.yml
packages/trellix_edr_cloud/manifest.yml

</details?