Is there a way to achieve some sort of autoscaling / adjustment to adjust number of fibers to achieve the best performance?

[ioquatix]

Considering typical async server implementations:

Async do
  endpoint.accept do |peer|
    while request = peer.read(...)
      response = process(request)
      peer.write(response)
    end
  end
end

We are concerned with whether endpoint.accept should limit the number of connections to improve the performance of the system. More specifically, in the event loop, you will have several fibers - one calling accept in a loop, and one per incoming connection. We are concerned about whether it's possible to end up in a situation where there are too many fibers.

[ioquatix]

In any system design like this, it's certainly possible to become CPU bound. If 10,000 clients connect and none of them write a request, at some point when they do all write requests, the server may suddenly become overloaded. If this is the kind of real world load you are trying to serve, you only really have two options: either accept the latency due to the overloaded server, or provision more hardware.

However, in typical real world scenarios, it's more likely that you will experience incrementally increasing load. Because the fibers are cooperative, if a fiber is CPU bound in generating a response, the fiber which is calling accept would not be resumed for a while. In a system with multiple server processes, this gives time for another server process to accept the connection. Because of this, incoming requests will naturally load balance across all available hardware.