Reverse TLS hybrid keyshares for x25519/x448-mlkem hybrids

[bhess]

• Implements reversing the key shares according to https://datatracker.ietf.org/doc/draft-kwiatkowski-tls-ecdhe-mlkem/
• The idea is a to add an optional parameter fips_standard to kems in generate.yml. If this is set, the key shares in x25519/x448 hybrids are reversed.
• Updates the NID of the x25519_mlkem768 hybrid.
• Interop with x25519_kyber768, disabled interop with x25519_kyber512 (https://mailarchive.ietf.org/arch/msg/tls/hli5ogDbUudAA4tZXskVbOqeor4/). x25519_mlkem768 seems not available yet for interop at pq.cloudflareresearch.com
• Tried to leave the code for hybrid/composite signatures untouched as good as possible

Open point: working encoder for the "reversed" hybrid kems. @baentsch is this still be needed after #522 lands?

Fixes #503

[pi-314159]

Tested with Google-BoringSSL's X25519MLKEM768 and passed :)

[baentsch]

Thanks for the PR, @bhess! Please let me know when you consider it Ready for Review.

Interop with x25519_kyber768

I don't understand: What did you test? That the key share reversal works using Kyber768, not MLKEM768? Using which code point?

Open point: working encoder for the "reversed" hybrid kems. @baentsch is this still be needed after #522 lands?

What reading do you think #522 has on this, @bhess? The changes this PR does apply (with most externally visible relevance) to KEMs with OIDs (well, really primarily x25519mlkem768), right? #522 in turn changes the handling of KEMs without OIDs.

Tested with Google-BoringSSL's X25519MLKEM768 and passed :)

Nice, indeed. Thanks for the confirmation @pi-314159 . Just the names don't seem to align yet: Do you plan on aligning this, too @bhess?

[baentsch]

Finally, a general comment: On the positive side the approach taken in the PR avoids encoding specific algorithms' names into the provider, so good; I'm not convinced this added configuration option doesn't complicate maintainability of the code, though. Looking at the changes, this looks "brittle": Are you comfortable maintaining/supporting this going forward, @bhess? If so shouldn't there be some documentation explaining how this works? Might it not be better to simply change the order for all X-hybrids (as per suggestion by @pi-314159 I think?)?

[pi-314159]

@baentsch I asked @bhess earlier about changing the order of all X-hybrids.
In this case, we need to change about 20 codepoints in generate.yml and modify oqs_kmgmt.c/keymgmt_functions.fragment.
Changing names may not be necessary since current names are consistent with other hybrids supported by the oqs-provider.

[baentsch]

@baentsch I asked @bhess earlier about changing the order of all X-hybrids. In this case, we need to change about 20 codepoints in generate.yml and modify oqs_kmgmt.c/keymgmt_functions.fragment.

Changing 20 ids one-time doesn't sound that hard compared to having to explain and understand time and again what's going on in which cases and which code paths in hybrid vs composite vs plain PQ.

If oqsprovider remains a research vehicle, no problem -- that type of user can read the code and help themselves. But there's quite a few people that try to re-label this software as "production ready": Problem there: most of those folks have never contributed meaningfully or at all to OQS so don't know what their decisions practically imply. I'd like to ensure that the code in oqsprovider also caters to such case, i.e., that the (then growing) support obligations on the community should not be getting unmanageable (btw, there is precedent to the TAC disregarding community concerns -- and it is exactly in the area of declaring software of high quality).

Changing names may not be necessary since current names are consistent with other hybrids supported by the oqs-provider.

It surely may not be necessary. Taking the perspective of a user, wouldn't that be more consistent (across implementions and with spec), though?

[ghen2]

Tested this x25519_mlkem768 with Firefox 132 nightly and they are interoperable.
(while Firefox 130 continues to work with x25519_kyber768 on the same oqs-provider)

[bhess]

Might it not be better to simply change the order for all X-hybrids (as per suggestion by @pi-314159 I think?)?

@baentsch I asked @bhess earlier about changing the order of all X-hybrids.
In this case, we need to change about 20 codepoints in generate.yml and modify

I agree that changing all x-hybrids would be a bit simpler. Changing the codepoints would also be ok IMO.
However, we would also lose interop with Chrome&Cloudflare's (edit: and Firefox) x25519_kyber768 which will still be around for a while, as the order there is not reversed. Don't we want to keep that at least until Cloudflare and Chrome supports x25519_mlkem768? (Chrome plans to do the switch to mlkem in version 131 November).
The current implementation basically implements the logic employed in draft-kwiatkowski-tls-ecdhe-mlkem.

I don't understand: What did you test? That the key share reversal works using Kyber768, not MLKEM768? Using which code point?

No, I just added the commented-out interop test to Cloudflare again with x25519_kyber768.

Just the names don't seem to align yet: Do you plan on aligning this, too @bhess?

If we want to do this, yes. It may take me 1-2 days since I'm traveling this week. @praveksharma kindly offered help with the templating changes needed. Thanks Pravek! Feel free to give it a go. My idea was to add a property like "standard_name" to the .yml file, and to use this as name if available.
The other open point is getting the encoder for the reversed orders working again.

[bhess]

Thanks for testing with Firefox and BoringSSL @ghen2 & @pi-314159!

[ghen2]

However, we would also lose interop with Chrome&Cloudflare's (edit: and Firefox) x25519_kyber768 which will still be around for a while, as the order there is not reversed. Don't we want to keep that at least until Cloudflare and Chrome supports x25519_mlkem768? (Chrome plans to do the switch to mlkem in version 131 November).

FYI, Firefox 132 with x25519_mlkem768 will be released on October 29, and has it enabled by default for HTTP/2 (but they seem to have broken the knob for HTTP/3).

[pi-314159]

@bhess @baentsch I don't think it's a big problem

1. this is currenly for research only.
2. Chrome and Firefox will support x25519mlkem768 within two months, so we can switch early.
3. Since x25519_kyber768 will be dropped soon, I don't see the necessity to interop with Cloudflare's x25519_kyber768.

[bhess]

@bhess @baentsch I don't think it's a big problem

1. this is currenly for research only.
2. Chrome and Firefox will support x25519mlkem768 within two months, so we can switch early.
3. Since x25519_kyber768 will be dropped soon, I don't see the necessity to interop with Cloudflare's x25519_kyber768.

So why don't we do the switch (reversing all x-hybrids) in two months, i.e. in the oqs-provider release after this one? I think it doesn't hurt and is a plus to have interop of what is used now in practice. @pi-314159 I think it's not a problem if you do this differently in oqs-BoringSSL and drop interop now.

Are you comfortable maintaining/supporting this going forward, @bhess?

Sure and I'll also be happy to drop this logic when it is not used anymore. The change required is minimal.

[baentsch]

just added the commented-out interop test to Cloudflare again with x25519_kyber768.

At a different code point, though. So CF simply switched to "our" code point for x22519_kyber768 it seems but then works towards implementing MLKEM....

to add a property like "standard_name" to the .yml file

That'd also be my favourite to keep key information in one place.

@praveksharma kindly offered help with the templating changes needed

Thanks also from me @praveksharma ! Any and all additional help with this project welcome!

[ghen2]

The next revision of draft-kwiatkowski-tls-ecdhe-mlkem will also standardise p384_mlkem1024 as SecP384r1MLKEM1024 with id 0x11ED.

[bhess]

just added the commented-out interop test to Cloudflare again with x25519_kyber768.

At a different code point, though. So CF simply switched to "our" code point for x22519_kyber768 it seems but then works towards implementing MLKEM....

I don't think I changed the code point for x25519_kyber768. It was 0x6399 before and still is.
There was a codepoint as env variable defined for x25519_kyber512 which I removed, because Cloudflare discontinued support.

[bhess]

The encoders are updated to be able to reverse the key shares. Thanks @praveksharma for adding the "standard names" X25519MLKEM768 and SecP256r1MLKEM768.

[dstebila]

What's the logic with the startswith('p') test?

[bhess]

not hybrid['hybrid_group'].startswith('p') captures all hybrids with curves that don't start with 'p', i.e. not p256/p384/p521. So it sets the 'reverse' flag to 1 if the pq-kem is a fips_standard and the curve is x25519 or x448.

[dstebila]

But sometimes p256 is named SecP256. I guess the root of my question is if there's some kind of fragile implicit logic here that might not be obvious to someone else down the line.

[bhess]

I reused the logic that was already in the template for matching with startswith('p'), at least in the template the curves are not named secpXXX. But I agree this is fragile, I can change this here to explicitly match x25519 and x448.

[bhess]

Updated in 77214a4

[baentsch]

there's some kind of fragile implicit logic here that might not be obvious to someone else down the line

Here and in many other places of the code. Mea culpa.

Getting sth done with little time available always has been priority #1. I tried to be conscientious but I've been pretty much alone on this -- and did many other things in parallel (more and more people do more and more of the things I've been doing "behind the scenes" and still OQS keeps dropping feature/functions I enabled before). Hence I've been arguing/looking for "someone else" (ideally more than 1 :) -- and not down the line, but since years -- and more and more vehemently since PQCA started to market this as "product ready" despite a clear lack of technical understanding (or lack of concern for possible problems "down the line") what this really entails. Sorry, I am a fallible human with limited energy (and you know who/what has drained most of it, @dstebila ...).

[baentsch]

Thanks for all these updates, @bhess & @praveksharma !

Question to @feventura : Would you have time to check this statement for composites?

Tried to leave the code for hybrid/composite signatures untouched as good as possible

As far as the hybrid sigs are concerned, I'd argue that (continued) proper operation in the IETF PQCerts hackathon would be a good indication everything's (still) OK. @praveksharma : Are we still in contact with that team or even better, have scripting to validate all is fine?

Given how convoluted all this "keyslot logic" is (Yes, I know you said so @thb-sb...), I'd also feel much better if we had a test server to test this against (in CI). For now, the manual checks done by @ghen2 & @pi-314159 are arguably good enough. As discussed yesterday, we should be getting back to where we've been in this regard (worthwhile creating an issue if this can't be brought back in this PR, e.g., by way of testing with boringssl as discussed?).

[bhess]

Given how convoluted all this "keyslot logic" is (Yes, I know you said so @thb-sb...), I'd also feel much better if we had a test server to test this against (in CI). For now, the manual checks done by @ghen2 & @pi-314159 are arguably good enough. As discussed yesterday, we should be getting back to where we've been in this regard (worthwhile creating an issue if this can't be brought back in this PR, e.g., by way of testing with boringssl as discussed?).

Fully agree. I found the code in oqsprov_keys.c especially difficult to follow and spent quite some time for a seemingly simple task like reversing the key share. It's mixing logic between KEMs, signatures, hybrids and composites, sometimes encoding key sizes in the key structure, sometimes using keyslots and sometimes not. Better structuring this code would be helpful, but a major task.

[baentsch]

Better structuring this code would be helpful, but a major task.

Completely agree; see e.g. #375 #483