Authentication Failure after cluster certs renewal

[bj9306]

We have renewed cluster and client certs recently and we are observing some authentication failures after that. we ran below command multiple times. we are seeing empty "Acceptable client certificate CA names" sometimes and we are getting intermittent publish and subscribe failures. any idea why this is `happening?`

openssl s_client -connect broker1-kafka-load-eastus.att.io:443

Attempt1

Acceptable client certificate CA names
C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
O = io.strimzi, CN = clients-ca v0

Attempt2

CONNECTED(00000005)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 334 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

[scholzj]

It is hard to help just from what you shared. But it looks like there is no SSL on the other side of the OpenSSL connection.

[bj9306]

if you see this response, you can find it shows acceptable client certs.

Attempt1

Acceptable client certificate CA names
C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
O = io.strimzi, CN = clients-ca v0

[scholzj]

But these are from the client side, not from the server side, or?

[bj9306]

yes from client side. broker1-kafka-load-eastus.att.io is one of the broker in our cluster.

openssl s_client -connect broker1-kafka-load-eastus.att.io:443

[scholzj]

Yeah, so, why does it not show any certificates presented by the server? Mine OpenSSL IMHO normally does that. So that would IMHO suggest some networking issue?

[bj9306]

i am fully blank on whats happening on this. some of the publish attempts are successful while others are failing with same secret.