Unknown certificate type

[Ggrison]

Dear community,

I desire to make as simple VMWare virtual Ubuntu server running locally an nginx webserver using dilithium certificate and kyber. My objective is simply to do my hello world using only quantum resistant algorythm. Nothing more nothing less. The fun of it.

The configuration I am using is the following:

• Ubuntu server 24.04 LTS (https://ubuntu.com/download/server)
• OpenSSL 3.3.1 (https://openssl-library.org/source/)
• OQS-provider obviously
• Nginx 1.26 (but I'm not at this step yet so not relevant for the question I am asking here)

You will find below the exact command line that I have typed to setup a brand new installed Ubuntu Server:

Liboqs:

sudo apt install astyle cmake gcc ninja-build libssl-dev python3-pytest python3-pytest-xdist unzip xsltproc doxygen graphviz python3-yaml valgrind

cd /usr/local/src
git clone https://github.com/open-quantum-safe/liboqs
cd liboqs

mkdir build && cd build
cmake -GNinja ..
ninja
ninja run_tests
sudo ninja install

All perfect, no issue at all.

OQS-Provider:

cd /usr/local/src
git clone https://github.com/open-quantum-safe/oqs-provider
cd oqs-provider
cmake -S . -B _build && cmake --build _build && sudo cmake --install _build

cd _build && ctest --parallel 5 --rerun-failed --output-on-failure -V

All test OK, no issue at all.

OpenSSL:

cd /usr/local/src
wget https://github.com/openssl/openssl/releases/download/openssl-3.3.1/openssl-3.3.1.tar.gz
cd /usr/local/src/openssl-3.3.1
./Configure --prefix=/usr/local/ssl --openssldir=/usr/local/ssl '-Wl,--enable-new-dtags,-rpath,$(LIBRPATH)'

make -j8
make test
sudo make install
/usr/local/ssl/bin/openssl version -a

OpenSSL 3.3.1 4 Jun 2024 (Library: OpenSSL 3.3.1 4 Jun 2024)
built on: Sat Aug 24 07:57:57 2024 UTC
platform: linux-x86_64
options:  bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
OPENSSLDIR: "/usr/local/ssl"
ENGINESDIR: "/usr/local/ssl/lib64/engines-3"
MODULESDIR: "/usr/local/ssl/lib64/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0xfffa32034f8bffff:0xd19e27eb

All test OK, install OK, version OK

Copying oqsprovider.so into the correct local openssl install:

sudo cp /usr/lib/x86_64-linux-gnu/ossl-modules/oqsprovider.so /usr/local/ssl/lib64/ossl-modules/oqsprovider.so

Modifying the configuration file as explained in the documentation:

sudo nano /usr/local/ssl/openssl.cnf

[provider_sect]
default = default_sect
oqsprovider = oqsprovider_sect
[default_sect]
activate = 1
[oqsprovider_sect]
activate = 1

Checking OQS-provider is recognized and algo available:

/usr/local/ssl/bin/openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.3.1
    status: active
  oqsprovider
    name: OpenSSL OQS Provider
    version: 0.6.2-dev
    status: active

/usr/local/ssl/bin/openssl list -signature-algorithms -provider oqsprovider
  { 1.2.840.113549.1.1.1, 2.5.8.1.1, RSA, rsaEncryption } @ default
  { 1.2.840.10040.4.1, 1.2.840.10040.4.3, 1.3.14.3.2.12, 1.3.14.3.2.13, 1.3.14.3.2.27, DSA, DSA-old, DSA-SHA, DSA-SHA1, DSA-SHA1-old, dsaEncryption, dsaEncryption-old, dsaWithSHA, dsaWithSHA1, dsaWithSHA1-old } @ default
  { 1.3.101.112, ED25519 } @ default
  { 1.3.101.113, ED448 } @ default
  { 1.2.156.10197.1.301, SM2 } @ default
  ECDSA @ default
  HMAC @ default
  SIPHASH @ default
  POLY1305 @ default
  CMAC @ default
  dilithium2 @ oqsprovider
  p256_dilithium2 @ oqsprovider
  rsa3072_dilithium2 @ oqsprovider
  dilithium3 @ oqsprovider
...
  rsa3072_sphincsshake128fsimple @ oqsprovider
  mayo1 @ oqsprovider
  p256_mayo1 @ oqsprovider
  mayo2 @ oqsprovider
  p256_mayo2 @ oqsprovider
  mayo3 @ oqsprovider
  p384_mayo3 @ oqsprovider
  mayo5 @ oqsprovider
  p521_mayo5 @ oqsprovider

From my perspective, everything is green and as expected.

Let's start generating some certifiate as in the USAGE.md:

mkdir /home/kyber/kyber_cert
cd /home/kyber/kyber_cert

/usr/local/ssl/bin/openssl req -x509 -new -newkey dilithium3 -keyout dilithium3_CA.key -out dilithium3_CA.crt -nodes -subj "/CN=test CA" -days 365 -config /usr/local/ssl/openssl.cnf
/usr/local/ssl/bin/openssl genpkey -algorithm dilithium3 -out dilithium3_srv.key
/usr/local/ssl/bin/openssl req -new -newkey dilithium3 -keyout dilithium3_srv.key -out dilithium3_srv.csr -nodes -subj "/CN=test server" -config /usr/local/ssl/openssl.cnf
/usr/local/ssl/bin/openssl x509 -req -in dilithium3_srv.csr -out dilithium3_srv.crt -CA dilithium3_CA.crt -CAkey dilithium3_CA.key -CAcreateserial -days 365

ll
kyber@kyber:~/kyber_cert$ ll
total 52
drwxrwxr-x 2 kyber kyber 4096 Aug 24 08:30 ./
drwxr-x--- 5 kyber kyber 4096 Aug 24 08:27 ../
-rw-rw-r-- 1 kyber kyber 7497 Aug 24 08:30 dilithium3_CA.crt
-rw------- 1 kyber kyber 8155 Aug 24 08:30 dilithium3_CA.key
-rw-rw-r-- 1 kyber kyber   41 Aug 24 08:30 dilithium3_CA.srl
-rw-rw-r-- 1 kyber kyber 7481 Aug 24 08:30 dilithium3_srv.crt
-rw-rw-r-- 1 kyber kyber 7285 Aug 24 08:30 dilithium3_srv.csr
-rw------- 1 kyber kyber 8155 Aug 24 08:30 dilithium3_srv.key

Again, all perfect!

So, now let's make to simulate a server and a client connection as in the documentation USAGE.md:

/usr/local/ssl/bin/openssl s_server -cert dilithium3_srv.crt -key dilithium3_srv.key -www -tls1_3 -groups kyber768:frodo640shake

Using default temp DH parameters
error setting certificate
80EBCAE330700000:error:0A0000F7:SSL routines:ssl_set_cert:unknown certificate type:ssl/ssl_rsa.c:257:

Here is the drama that I am trying to debug since 3 days. I re-did these actions multiple time and looked a lot around internet.
Then I gave up and decided to re-start from scratch logging everything properly to give you all the necessary informations above.
I have tried some adjustment in environment variable, messing with openssl, try other configuration build, and so many other things... But nothing works when it comes to perform the client and server with the dilithium certificates.
I have also reviewed the different cases already closed but couldn't find a proper solution to apply to my problem.

Already thank you your assistance,

Greg

[baentsch]

@Ggrison Thanks for the full report. Before reading all details, allow me to ask whether you checked https://github.com/open-quantum-safe/oqs-demos/tree/main/nginx (and ran the ready-made dockerimage)? This "demo" has been done to facilitate experimentation such as yours...

[Ggrison]

@baentsch Hello Michael,
Yes I did and it worked that's why I got to the next step. I'm an old school type that doesn't really like docker and like real VM because I will suffer, suffer again but it will make me more knowledgeable and stronger. I don't like to not understand why it's not working but it should be.

I did not find any real VM implementing this also on internet and would like to do it and make it available to others.

Stubborn ? *insert Heisenberg meme * "You're god damn right".

Regards,

Greg

[baentsch]

I'm an old school type that doesn't really like docker and like real VM

Big smile. I'm even more old school and don't like VM either, but only real hardware :-) So, good to know our setup works to begin with... Now I read your description in detail and agree, it all looks like things should work OK. Can I safely assume you also checked the (dil) cert (for correctness/no corruption) via the openssl x509 command? Can I also assume you ran stuff using a hard-set LD_LIBRARY_PATH, OPENSSL_MODULES and OPENSSL_CONF to exclude any possibility that you're picking up the wrong libcrypto or config? Have you built/run stuff under a debugger? Building both openssl and oqsprovider with debug options alone creates a plethora of new logging capabilities that may help your "inner Heisenberg"... :-) Should any of this unearth a difference in your setup and the one in our nginx Dockerfile (that apparently works OK), please let us know (or do a PR to the documentation) to let others know....

[Ggrison]

@baentsch Thank you for your answer.

OK. Can I safely assume you also checked the (dil) cert (for correctness/no corruption) via the openssl x509 command?

Yup:

kyber@kyber:~/kyber_cert$ /usr/local/ssl/bin/openssl x509 -in dilithium3_CA.crt -noout -subject
subject=CN=test CA

kyber@kyber:~/kyber_cert$ /usr/local/ssl/bin/openssl x509 -in dilithium3_srv.crt -noout -subject
subject=CN=test server

kyber@kyber:~/kyber_cert$ /usr/local/ssl/bin/openssl x509 -in dilithium3_CA.crt -noout -text | head -20
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0e:9a:ed:e7:f8:2c:75:42:be:86:1f:cb:16:de:de:6c:87:f2:e2:6d
        Signature Algorithm: dilithium3
        Issuer: CN=test CA
        Validity
            Not Before: Aug 24 08:30:15 2024 GMT
            Not After : Aug 24 08:30:15 2025 GMT
        Subject: CN=test CA
        Subject Public Key Info:
            Public Key Algorithm: dilithium3
                dilithium3 public key:
                PQ key material:
                    4e:22:27:fa:08:e5:03:15:ec:c5:91:fd:e9:b1:54:
                    d8:15:72:d9:e8:7b:d7:9b:28:d7:01:b7:73:73:e3:
                    f4:9d:79:61:66:62:7f:d2:b4:50:f1:25:c0:73:fa:
                    9a:cd:ac:fe:a3:0c:09:0a:87:a8:61:11:77:5e:4f:
                    7a:83:1b:74:d6:b3:65:77:01:bd:32:e8:d5:be:30:
kyber@kyber:~/kyber_cert$ /usr/local/ssl/bin/openssl x509 -in dilithium3_srv.crt -noout -text | head -20
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            5a:6c:11:41:67:46:bb:e9:2d:a4:2f:d6:30:18:80:c9:c5:ce:9e:6a
        Signature Algorithm: dilithium3
        Issuer: CN=test CA
        Validity
            Not Before: Aug 24 08:30:30 2024 GMT
            Not After : Aug 24 08:30:30 2025 GMT
        Subject: CN=test server
        Subject Public Key Info:
            Public Key Algorithm: dilithium3
                dilithium3 public key:
                PQ key material:
                    b7:64:88:9a:1c:40:bf:6b:2a:0a:bf:78:4d:c2:e3:
                    c6:e9:b5:b8:f8:08:f0:55:e7:d1:d0:02:94:ee:e4:
                    0d:43:77:00:e8:de:19:60:4d:d6:3b:7d:7c:7a:60:
                    11:a0:4a:31:2f:87:f4:b7:1c:9b:10:03:29:fe:c3:
                    db:26:0b:44:99:63:67:15:49:dd:9f:6d:51:b8:b8:

Can I also assume you ran stuff using a hard-set LD_LIBRARY_PATH, OPENSSL_MODULES and OPENSSL_CONF to exclude any possibility that you're picking up the wrong libcrypto or config?

I did that in my debugging activities but not yet here as I wanted something clean to follow you rinstructions, so it's done now :

export LD_LIBRARY_PATH=/usr/local/ssl/lib64
export OPENSSL_MODULES=/usr/local/ssl/lib64/ossl-modules
export OPENSSL_CONF=/usr/local/ssl/openssl.cnf
kyber@kyber:/usr/local/ssl$ env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/usr/local/ssl/lib64

kyber@kyber:/usr/local/ssl$ env | grep OPENSSL_MODULES
OPENSSL_MODULES=/usr/local/ssl/lib64/ossl-modules

kyber@kyber:/usr/local/ssl$ env | grep OPENSSL_CONF
OPENSSL_CONF=/usr/local/ssl/openssl.cnf

Running the openssl server with these hard setted env variable I've got same error (which is a good and a bad news at the same time) :

kyber@kyber:~/kyber_cert$ /usr/local/ssl/bin/openssl s_server -cert /home/kyber/kyber_cert/dilithium3_srv.crt -key /home/kyber/kyber_cert/dilithium3_srv.key -www -tls1_3 -groups kyber768:frodo640shake
Using default temp DH parameters
error setting certificate
80BB5D0DA7760000:error:0A0000F7:SSL routines:ssl_set_cert:unknown certificate type:ssl/ssl_rsa.c:257:

Have you built/run stuff under a debugger? Building both openssl and oqsprovider with debug options alone creates a plethora of new logging capabilities that may help your "inner Heisenberg"... :-) Should any of this unearth a difference in your setup and the one in our nginx Dockerfile (that apparently works OK),

This is where I putted my limit, I did not wanted to do any debugging as my C skills died a long time ago, I can read some normal code but not advanced code like in these libraries.

please let us know (or do a PR to the documentation) to let others know....

Well yes, you can count on me. I wish to build a script to be run as a one-shot from a brand new ubuntu server install to do the setup of post-quantum hello world. I'm not there yet but I wish I can avoid some people to internally die like I am here.

Thank you,

Greg

[baentsch]

I wish to build a script to be run as a one-shot from a brand new ubuntu server

Sounds like a definitely doable thing (I suppose it should run an nginx server, then, basically mimicking the setup of test.openquantumsafe.org)? If you don't feel like banging your head alone, feel free to share your code/repo and I'll take a look/lend a hand.

[Ggrison]

Thank you for your help. I will continue with my shovel, even buy a second one if need be.

For your info, I have tried to generate RSA certificate and of course it's working like a charm with openssl s_server.

kyber@kyber:~/kyber_cert$ /usr/local/ssl/bin/openssl s_server -cert /home/kyber/kyber_cert/rsa_srv.crt -key /home/kyber/kyber_cert/rsa_srv.key -www -tls1_3 -groups kyber768:frodo640shake -debug -security_debug_verbose -provider oqsprovider
Using default temp DH parameters
Security callback: Certificate chain EE key=, bits=2048, security bits=112: yes
Security callback: Certificate chain CA digest=RSA-SHA256, security bits=128: yes
ACCEPT

So I tempted my chance with some Falcon certificate but it gives me the same issue as with dilithium. Wanted to make sure that Dilithium wasn't victim of bug only affecting him.

Debug verbose with openssl s_server are not as talkative as I wanted them to be...

kyber@kyber:/usr/local/src/openssl-3.3.1$ /usr/local/ssl/bin/openssl s_server -cert /home/kyber/kyber_cert/dilithium3_srv.crt -key /home/kyber/kyber_cert/dilithium3_srv.key -www -tls1_3 -groups kyber768:frodo640shake -debug -security_debug_verbose -provider oqsprovider -status_verbose -msg
Using default temp DH parameters
Security callback: Certificate chain EE key=, bits=192, security bits=192: yes
Security callback: Certificate chain CA digest=dilithium3, security bits=192: yes
error setting certificate
801BA206807B0000:error:0A0000F7:SSL routines:ssl_set_cert:unknown certificate type:ssl/ssl_rsa.c:257:

kyber@kyber:~/kyber_cert$ /usr/local/ssl/bin/openssl s_server -cert /home/kyber/kyber_cert/falcon_srv.crt -key /home/kyber/kyber_cert/falcon_srv.key -www -tls1_3 -groups kyber768:frodo640shake -debug -security_debug_verbose -provider oqsprovider
Using default temp DH parameters
Security callback: Certificate chain EE key=, bits=256, security bits=256: yes
Security callback: Certificate chain CA digest=falcon1024, security bits=256: yes
error setting certificate
807B3007DC770000:error:0A0000F7:SSL routines:ssl_set_cert:unknown certificate type:ssl/ssl_rsa.c:257:

Thank you for your help proposal, I can share with you my Virtual Machine through a big zip file (using VMWare Workstation Player)

Regards,

[baentsch]

Before shipping around large VM images, can I ask you to set the OPENSSL_ROOT_DIR cmake var when building oqsprovider? All your problems are consistent with your setup using a libssl with a version below 3.2 (which cannot handle provider based sigs for TLS 13 ops, only for doing certs), I.e., the system libssl, and I didn't see that config in your build script...

[Ggrison]

Before shipping around large VM images, can I ask you to set the OPENSSL_ROOT_DIR cmake var when building oqsprovider? All your problems are consistent with your setup using a libssl with a version below 3.2 (which cannot handle provider based sigs for TLS 13 ops, only for doing certs), I.e., the system libssl, and I didn't see that config in your build script...

Step 1:

cmake -DOPENSSL_ROOT_DIR=/usr/local/ssl -S . -B _build && cmake --build _build && sudo cmake --install _build

cd _build && ctest --parallel 5 --rerun-failed --output-on-failure -V
...
100% tests passed, 0 tests failed out of 6

Total Test time (real) =  13.94 sec

Step 2:

kyber@kyber:/usr/local/src/oqs-provider/_build$ /usr/local/ssl/bin/openssl s_server -cert /home/kyber/kyber_cert/dilithium3_srv.crt -key /home/kyber/kyber_cert/dilithium3_srv.key -www -tls1_3 -groups kyber768:frodo640shake -debug -security_debug_verbose -provider oqsprovider
Using default temp DH parameters
Security callback: Certificate chain EE key=, bits=192, security bits=192: yes
Security callback: Certificate chain CA digest=dilithium3, security bits=192: yes
ACCEPT

It's beautiful.

I will now continue with nginx implementation.

Thank you. THANK YOU.

Do you want that we add something in the documentation between this error and this solution ?
If yes then tell me what the process is.

Regards,

Greg

[baentsch]

Do you want that we add something in the documentation between this error and this solution ?

Sure -- feel free to do a PR adding wording to the documentation in a place where you deem it missing/might help others/would have helped you most. Please add to your commit the tag "[skip ci]" to avoid wasting CI test cycles (if indeed it only is a documentation change).

All that said, this option is visible in the "courtesy" build script... Maybe a more clear reference in the documentation to that script could be sensible, too (?).

I will now continue with nginx implementation.

Allow me to suggest again using our Dockerfile as a starting point (that also btw contains the reference to OPENSSL_ROOT_DIR): It encodes many hours of my time banging my head so other don't have to again :)

[baentsch]

Closing due to inactivity. Tagging @Ggrison as a gentle reminder to contribute documentation as per discussion above to avoid others hitting the same problem again.