"SSL_read: unexpected eof while reading" in Ubuntu 22.04/openssl 3.0.2

[drbugfinder-work]

Describe the bug

Hi,

with the switch from Ubuntu 20.04 to Ubuntu 22.04 (which includes the transition from openssl 1.1.1f to openssl 3.0.2), we are getting the following error message every few seconds with a Forward input (from Fluent Bit) with TLS/SSL encryption:

│ 2022-07-25 09:09:34 +0000 [warn]: #0 close socket due to unexpected ssl error: SSL_read: unexpected eof while reading
│ 2022-07-25 09:09:42 +0000 [warn]: #0 close socket due to unexpected ssl error: SSL_read: unexpected eof while reading

Maybe it is related to curl/curl#5138 (curl/curl#5138 (comment))

To me it looks like fluentd was not updated to use OpenSSL 3, like described here: openssl/openssl@0cd2ee6#diff-9a86c74704f3d66bc69ba9cf9dc3168eea957103bcc36f1c2e3a1f05b10188a1R166

We cannot see any impact on message handling from Fluent Bit to td-agent. Setting TLS version (of td-agent) did not change the SSL EOF warnings.

We tried with td-agent 4.3.2 and the pre-release version of 4.4.0 and got the same warning.

To Reproduce

• Install fluentd/td-agent 4.3.2 on Ubuntu 22.04
• Create a config with a Forward Input (TLS encrypted)
• Send logs from Fluent Bit to this instance

Expected behavior

No SSL error

Your Environment

- Fluentd version: v1.14.6
- TD Agent version: 4.3.2
- Operating system: Ubuntu 22.04
- Kernel version: running inside container / Container Host: 5.10.123

Your Configuration

Use Forward plugin with SSL/TLS encryption

Your Error Log

[warn]: #0 close socket due to unexpected ssl error: SSL_read: unexpected eof while reading

Additional context

Opened same ticket in td-agent fluent/fluent-package-builder#406

[ashie]

To me it looks like fluentd was not updated to use OpenSSL 3, like described here: openssl/openssl@0cd2ee6#diff-9a86c74704f3d66bc69ba9cf9dc3168eea957103bcc36f1c2e3a1f05b10188a1R166

I don't think so. Fluentd seems to use OpenSSL 3 properly, this is the why it shows SSL_read: unexpected eof while reading error.
Although I don't yet understand well the issue, according the document you mentioned, this error will be detected only when application uses OpenSSL 3.0 or later, or 1.1.1e. 1.1.1f reverts the fix to keep backward compatibility, so that if fluentd uses 1.1.1 series except 1.1.1e, it won't be shown.

Probably the root cause is that the peer application (Fluent Bit) doesn't send close_notify.
OpenSSL 1.1.1 series ignores the error but OpenSSL 3.0 series strictly detect the error, this is the why you see the error.
I'm also using fluentd on Ubuntu 22.04 with fluentd peers but I don't yet get such error.

On OpenSSL 3.0, setting SSL_OP_IGNORE_UNEXPECTED_EOF to oepnssl context seems to be able ignoring this.
https://www.openssl.org/docs/man3.0/man3/SSL_CTX_set_options.html#SSL_OP_IGNORE_UNEXPECTED_EOF
But probably it should be fixed by the peer application.

[drbugfinder-work]

I see. So you mean we should address this issue in Fluent Bit?

[ashie]

I see. So you mean we should address this issue in Fluent Bit?

I'm not sure yet, but probably it should be.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[drbugfinder-work]

still open

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[Gradlon]

What is the state on this issue?

[soulseekah]

Using the faraday-typhoeus HTTP backend helps with the OpenSearch plugin.

fluent/fluent-plugin-opensearch#128 (comment)

[kenhys]

The issue was already forwarded to Fluent Bit, but it was closed without any action.

fluent/fluent-bit#5791

Nothing we can do further more, so I'll close this issue.