High latency with Linux clients - or "It's always the Nagle's Algorithm"

[ansiwen]

I'm running a REST API over a Cohttp server with TLS in a mirage unikernel, and I was investigating a suspiciously high latency (~40ms roundtrip) over sub-ms network when sending several requests in a persistent TLS connection from an arbitrary https client on Linux. The same did not appear, when doing the requests from MacOS. Looking at the network traces I saw the following issue:

The server sends the response in small chunks (HTTP response line, headers and body in three separate chunks), as expected with the default value for ~flush of true. The first chunk is in frame 36. The delay between frame 36 and 37 comes from the Delayed ACK from the client, which is activated by default on Linux. Then the server is not doing anything, but just waiting for the ACK, before sending the rest of the response.

To summarize my observations (thanks to @hannesm for making it more concise):

1. I can observe the behaviour with a server on mirage-tcpip
2. I can observe the behaviour with a server on Linux TCP/IP stack
3. I can not observe the behaviour with a client that either is macOS or delayed ACK is disabled (TCP_QUICKACK socket option on the client socket) (I still see the double roundtrip ACK dance for a single response, but it's very fast.)

After some research I learned about Nagle's Algorithm, which is implemented in the Linux TCP/IP Stack as well as the MirageOS TCP/IP stack and would expose exactly this behaviour IIUC. On Linux (and other Unixes) you can disable the Nagle's Algorithm with the TCP_NODELAY socket option, which is often done for servers. The Mirage TCP/IP stack offers the write_nodelay functions for circumvent the buffering, but Conduit doesn't expose these. The Go language for example sets the TCP_NODELAY option by default to avoid such problems.

I tried to set ~flush to false, but it didn't seem to change anything.

I see two options to solve the problem here:

1. Disable the Nagle's Algorithm. That would still cause two small packages for one response, which is suboptimal, but at least it should be fast.
2. Send the whole response in one package in the same way the first chunk is sent now. Then the Nagle's algorithm would probably not delay anything, but I'm not 100% sure.

Optimally I want to have both: One package for a response and deactivated Nagle. Is this already possible, and I didn't figure yet out how? And if not, what would the right approach to implement that?

Follow up of: mirleft/ocaml-tls#480

[ansiwen]

@TheLortex @haesbaert @dinosaure pinging you, because you guys helped me before with other TCP/IP related issues. 😬

[ansiwen]

I can indeed disable the Nagle's Algorithm and make the issue disappear by using a Stack wrapper like this:

module Stack_nodelay (Stack : Tcpip.Stack.V4V6) = struct
  include Stack
  module TCP = struct
    include Stack.TCP
    let write = Stack.TCP.write_nodelay
    let writev = Stack.TCP.writev_nodelay
  end
end

However, as I mentioned above, even with ~flush:false it creates three separate "tinygrams" for one response.

[ansiwen]

I found one of the problems:

ocaml-cohttp/cohttp-mirage/src/io.ml

Lines 60 to 62 in daf1323

	let flush _ =
	(* NOOP since we flush in the normal writer functions above *)
	Lwt.return_unit

Is there a reason why every write in cohttp-mirage is flushed @avsm ?