vhost is in error-state (red) after successful receive of certificate (because of not jet created .der-file)

[tuxmi]

Hi ZoeyVid

I found an other potential bug. The creation of the ".der"-file seems a bit slower as the check for that file happens. This leads to a faulty-vhost (red). Here it should maybe wait a few seconds (or have something like a grace period) to recheck for this file, before doing this check. The full logs you see below

Currently my process to get a new certificate looks like this:

1. Hit "Add Proxy-Host"
2. Enter the details / FQDN / and Proxy destination
3. Under "TLS" -> Request new certificate & ToS
4. Hit "Save" -> Window closes after a few seconds -> vhost is now in error-state
5. Deactivate and Reactivate the vhost over the NPMplus webUI
6. Now it's working

For testing i changed the key-type in /etc/certbot.ini to RSA.

Successfully received certificate.
Certificate is saved at: /data/tls/certbot/live/npm-29/fullchain.pem
Key is saved at:         /data/tls/certbot/live/npm-29/privkey.pem
This certificate expires on 2025-04-01.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[Global   ] › ⬤  debug     CMD: openssl x509 -in /data/tls/certbot/live/npm-29/fullchain.pem -subject -noout
[Global   ] › ⬤  debug     CMD: openssl x509 -in /data/tls/certbot/live/npm-29/fullchain.pem -issuer -noout
[Global   ] › ⬤  debug     CMD: openssl x509 -in /data/tls/certbot/live/npm-29/fullchain.pem -dates -noout
[Global   ] › ⬤  debug     CMD: certbot-ocsp-fetcher.sh -c /data/tls/certbot -o /data/tls/certbot/live --no-reload-webserver --quiet
[Global   ] › ⬤  debug     CMD: nginx -tq
[Global   ] › ⬤  debug     CMD: certbot-ocsp-fetcher.sh -c /data/tls/certbot -o /data/tls/certbot/live --no-reload-webserver --quiet
[Global   ] › ⬤  debug     CMD: nginx -tq
[Nginx    ] › ✖  error     nginx: [emerg] BIO_new_file("/data/tls/certbot/live/npm-29.der") failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/data/tls/certbot/live/npm-29.der, rb) error:10000080:BIO routines::no such file)
nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed

[Global   ] › ⬤  debug     CMD: certbot-ocsp-fetcher.sh -c /data/tls/certbot -o /data/tls/certbot/live --no-reload-webserver --quiet
[Global   ] › ⬤  debug     CMD: nginx -tq
[Nginx    ] › ℹ  info      Reloading Nginx
[Global   ] › ⬤  debug     CMD: nginx -s reload
2025/01/01 22:51:48 [warn] 1432#1432: deleting socket /run/nginx-16.sock
[Global   ] › ⬤  debug     CMD: certbot-ocsp-fetcher.sh -c /data/tls/certbot -o /data/tls/certbot/live --no-reload-webserver --quiet
[Global   ] › ⬤  debug     CMD: nginx -tq
[Nginx    ] › ℹ  info      Reloading Nginx
[Global   ] › ⬤  debug     CMD: nginx -s reload
[Global   ] › ⬤  debug     CMD: certbot-ocsp-fetcher.sh -c /data/tls/certbot -o /data/tls/certbot/live --no-reload-webserver --quiet
[Global   ] › ⬤  debug     CMD: nginx -tq
[Global   ] › ⬤  debug     CMD: certbot-ocsp-fetcher.sh -c /data/tls/certbot -o /data/tls/certbot/live --no-reload-webserver --quiet
[Global   ] › ⬤  debug     CMD: nginx -tq
[Global   ] › ⬤  debug     CMD: certbot-ocsp-fetcher.sh -c /data/tls/certbot -o /data/tls/certbot/live --no-reload-webserver --quiet
[Global   ] › ⬤  debug     CMD: nginx -tq
[Nginx    ] › ℹ  info      Reloading Nginx
[Global   ] › ⬤  debug     CMD: nginx -s reload

[tuxmi]

From what we can see in the logs above, NPMplus (specifically the certbot-ocsp-fetcher.sh script) is generating a local OCSP response in DER format and placing it in my example to "/data/tls/certbot/live/npm-29.der". This .der file is then referenced by the Nginx config via the ssl_stapling_file directive to enable OCSP stapling.

However, it appears there is something like a race condition in which the .der file hasn’t finished generating by the time NPMplus tries to read it - leading to BIO_new_file(...No such file or directory) errors.

Essentially, Nginx is being reloaded (and config-checked) before the .der file fully exists on disk.

[Zoey2936]

does this also happen when creating certs using letsencrypt/zerossl?

[Zoey2936]

it looks more like the CA does not return the stapling.der file in the first few seconds after the cert was created, since the script itself runs between cert creation and config creation, so the der file should exist if the CA returns it. so I think the CA is "too slow"

[tuxmi]

From my research today, the .der file for OCSP stapling typically isn’t sent directly by the CA as part of the ACME certificate issuance. Instead, here’s the usual flow:

1. Certificate Issuance:

   • The ACME server (FreeIPA/Dogtag in my case) issues the certificate in PEM format (or a chain of PEM certs).
   • That completes the ACME transaction, but no .der file is provided at this point.

2. OCSP Fetch Step (Local Script):

   • After the certificate is created, a script (in our casecertbot-ocsp-fetcher.sh) queries the OCSP endpoint of the CA.
   • The CA’s OCSP endpoint returns an OCSP response (if it’s available yet - some CA's also need here a few seconds to generate it on CA side) in DER format.
   • The script then writes that DER-encoded response to /data/tls/certbot/live/npm-xx.der.
   • Nginx then uses this .der file (stapling file) to provide OCSP stapling.

What the problem could be :

• Right after a brand-new cert is issued, the CA’s OCSP responder might not immediately have the updated OCSP data for that certificate.
• If the script queries the OCSP endpoint too soon, it may return an error or “unknown” status. In that scenario, no valid .der response file is written.
• As a result, Nginx’s config test fails when it tries to read the .der file that isn’t there yet.

I think a quick fix would be, to add a short grace period or retry after cert issuance—giving the CA’s OCSP responder time to recognize the new certificate - before we generate or validate the .der file.

Maybe something like :

for i in {1..4}; do
  if [ -f "/data/tls/certbot/live/npm-xx.der" ]; then
    echo ".der file is ready!"
    break
  else
    echo "Waiting for the .der file to appear..."
    sleep 2
  fi
done

As a quick workaround, I currently set ACME_OCSP_STAPLING=false to prevent the creation of the .der files. Also to not have validation Problems with our internal-sites because of the missing OSCP validation responds I also set ACME_MUST_STAPLE=false that no more status_request is set to the certificates.

(Must-Staple is an X.509 extension (tlsfeature with value status_request), which tells clients (browsers) they must see a valid OCSP staple respond in the TLS handshake or they should reject the connection.)

[Zoey2936]

thats what I wrote above, the issue is the CA which return the stapling response to slow, no Issue Bug of NPMplus

[Zoey2936]

can you maybe adjust the /app/internal/nginx.js file inside the container to remove the --quiet argument from the ocsp fetcher script, change the function type from execFile to execfg, restart npmplus and create a new host with a new cert, then the script would output some more information

[Zoey2936]

does the issue still exist, if not the information I asked above this comment would be helpful

[tuxmi]

Hey Zoey, yes it still exists, but currently I am working on my cyber-sec study exams and (sadly) don't have time for debugging, developing or engineering my stuff. I hope i will continuous in the next few weeks. :)