AWS_S3 source: Add metrics support

[GGonzalezGomez]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

AWS metric streams allows to send metrics to S3 (through firehose) in otel format or json. The use case would be vector picking up these metric files and processing them in the pipeline.

Attempted Solutions

We use aws_s3 source for logs, picking up log files through SQS messages. This is working for us properly. So we would also like to extend this for metrics as well as AWS Cloudwatch can dump metrics into S3 as well.

Proposal

AWS_S3 source already allows to pick up log files. In the documentation output is mentioned only for logs, so the idea would be to be able to extend this functionality for metrics as well. That way you may either read a bunch of files from S3 for processing or read an SQS queue for new file notifications and keep sending metrics through S3.
Again, this is already available for logs, just extending the functionality for metric signal as well.

References

No response

Version

0.33 and 0.37

[jszwedko]

The aws_s3 sink can already decode metrics, but only if they are encoded in one of the currently supported metric codecs. These would be: native_json, native, and influxdb. I think this feature could be supported by:

• Adding an otlp codec that could decode OTLP events
• Adding an aws_metric_json codec that understands how to decode the AWS metric JSON format

[GGonzalezGomez]

That makes sense, if it accepts already metrics having support to at least OTLP (AWS can output both 0.7 and 1.0 format) would be fantastic.

Thanks!