Faraday/Typhoeus HTTP adapter support (potentially as a plugin)

[ziggythehamster]

Is your feature request related to a problem? Please describe.
Net::HTTP doesn't get very good parallel throughput because it's totally in Ruby and doesn't benefit from being able to manage its own I/O or threading in the same way that a C extension would. The AWS SDK supports pluggable handlers, and this was brought up in #301, but in SDKv3 I don't know what the current pattern for implementing such a handler would look like.

In particular, I'm issuing a ton of S3 API calls in threads, and throughput maxes out at a relatively low number of threads because everything is having to go through the Ruby interpreter, so there are only so many tasks that can run concurrently.

Describe the solution you'd like
I would love official support for Faraday (for multiple pluggable backends) or Typhoeus (which is what I would use, as it's a wrapper for cURL), possibly in a plugin. Typhoeus, in particular, would also work for H2 and H3/QUIC since it is using cURL internally. Alternatively, having documentation for how a custom handler might be implemented in SDKv3 would enable me to write my own handler.

Describe alternatives you've considered
I've thought about spawning multiple processes instead of multiple threads, but it would most likely perform worse without spawning processes and threads, and this would be a ton of work to get working properly - most likely more than writing a HTTP handler on top of Typhoeus.

I've also considered JRuby (whose Net::HTTP does hit a code path that runs in compiled code), but we don't use JRuby for anything else (and our other apps won't run under JRuby), so a broader solution that benefits more than this one-off thing would be preferred.

• 👋 I may be able to implement this feature request

Additional context
I am an Amazon employee (not in AWS), and so may be able to get the time to implement this and help you ship it.

[pschrammel]

Can only second this, there are awesome adapters for high throughput IO like https://github.com/socketry/async-http-faraday . THX

[github-actions]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

[baxang]

https://github.com/saluzafa/async-aws-ruby seems close to what the OP suggested albeit neither Faraday nor Typhoeus.

[mullermp]

I think an async adapter is a great idea. For JSON/XML parsing, we have a concept of "engines" where we load engines based on what dependencies are bundled/installed with the SDK. I think we can similarly apply the engines concept to HTTP clients, checking for async, Faraday, Typhoeus, etc, and using that client. It may have unintended consequences though.

[ziggythehamster]

async-aws probably does the trick, assuming it works with the latest versions of the AWS SDK and @ioquatix's Async library.

That said, the library is relatively simple (I hadn't realized it was quite that easy to write an adapter since the docs don't go into much detail), so if it doesn't, it should be pretty easy to fix it.

And for that matter, a Typhoeus adapter would apparently be easy as well.