Limiting software will help hardware

[BorisPis]

Consider adding an API for users to indicate that on the send side, packets will not use variable length fields. Doing variable length field processing in hardware is more costly, and performance and scale will improve if assumptions can be made on packet header format.

In particular, consider:

• constant packet number length (e.g., 4 bytes)
• constant connection ID length for all flows sharing a UDP destination port

[nibanks]

constant packet number length (e.g., 4 bytes)

If this helps perf, I think we could definitely consider restricting packet numbers to 4 bytes. MsQuic already always uses 4 bytes.

constant connection ID length for all flows sharing a UDP destination port

On RX, we definitely can/do use the same length CIDs in MsQuic. On TX, we don't have control of the CID, so we can't place any restrictions.

[BorisPis]

An additional request is to keep the connection ID as short as possible. I speculate that different devices may have different limitations, in general the shorter the better.

[nibanks]

An additional request is to keep the connection ID as short as possible. I speculate that different devices may have different limitations, in general the shorter the better.

As previously indicated, we have no control for the TX path.

But even for the RX path, the size and composition of the CID will generally be dictated based on the load balancer model. For instance, encrypted CID LB models will require 18-byte CIDs.

So, while I agree that smaller might be better, we cannot make any assumptions or restrictions in this area, unfortunately. In fact, I think we should assume worst case here, and do what we can to optimize for it.

[nibanks]

@BorisPis @stevedoyle (and any others) for the constant packet number issue, should it be a recommendation or a requirement?

[BorisPis]

As previously indicated, we have no control for the TX path.

Right, because the client selects the CID.

But even for the RX path, the size and composition of the CID will generally be dictated based on the load balancer model. For instance, encrypted CID LB models will require 18-byte CIDs.

Right, and these will likely have slightly worse performance and scale.

for the constant packet number issue, should it be a recommendation or a requirement?

Hmm... For high speed connections using hardware, it will simplify things if this was required. But, we can live with it being only a recommendation.

In any case, I think that the kernel-driver interface should indicate whether this is enforced or not. The device's offload for a flow may look slightly differently when packet numbers are guaranteed to be 4 bytes.

[nibanks]

Based on this feedback and others I've received privately, I'm leaning towards possibly having the following restrictions to make things simpler for the hardware (and to some extent the software):

1. Require all offloaded receive direction CIDs to be the same length, globally.
   a. This is a pretty steep requirement, but I think that in practice it's likely not that big of deal. HW offload generally means server scenario. Server scenario generally means one or few unique servers, and generally they will likely all being using the same QUIC stack.
   b. By imposing this requirement, we can completely eliminate one lookup on the receive path. Without this, the CID length has to first be discovered based on the local tuple. Then that length is used to get the correct CID to then look up the rest of the info. Having a 2-step lookup is painful.
2. Restrict the possible lengths of offloaded CIDs (in both directions?) to a few possible values (0, 8, 20?) to eliminate the completely variable length nature required to be supported in HW.
   a. I'm not sure of how many and what values to restrict it to. 0 is a must. 20 (current max) is likely a must. But what values (and how many) in between? Any suggestions @BorisPis or @stevedoyle?
3. Restrict TX offload to using 4-byte packet numbers.

[BorisPis]

Require all offloaded receive direction CIDs to be the same length, globally.

That would improve receive offload, if possible please consider doing so.

Restrict the possible lengths of offloaded CIDs (in both directions?)

In general, the connection ID problem is mainly a receive side issue. On transmit, software and hardware can use a short <=4B hint to indicate the connection's parameters. This however is not possible on receive as the NIC sees the packet before software.

Restrict TX offload to using 4-byte packet numbers.

That would help as well. Even making the packet number length a constant would avoid a lookup as long as hardware knows this ahead of time.

[nibanks]

We had some discussions yesterday, and are currently leaning towards:

1. Not limiting the packet number size, because we cannot control the receiver side (from client to server) anyways, so the HW must support this regardless.
2. For the receive side, we cannot realistically limit to just a few different sizes of connection IDs, but instead we can use the first byte (as specified in quic-load-balancers) to self describe the CID length.