Use OpenSSL 3.0.x in our binary builds

[gpshead]

Feature or enhancement

We currently use OpenSSL 1.1.1 series in our Windows and macOS binary builds.

Per https://www.openssl.org/source/, that is only supported through September of 2023.

Thus we need to switch to a supported version of OpenSSL before 3.12 is released. (And likely consider moving 3.11 to use it if deemed feasible)

There are a pile of bugs related to OpenSSL 3 that may or may not be blockers:

• Windows support for OpenSSL 3.0 #90728
• python 3.11 test_ssl failure with OpenSSL 3.0 on Windows #101401
• FIPS usedforsecurity flag is no longer functional with OpenSSL 3.0.0 #90307
• ssl module with OpenSSL 3.0 does not throw SSLEOFError on unclean shutdown #95494
• ... edit this list to link to others ...

We have a longer term desire to not be so beholden to OpenSSL at all. But this issue is being filed as a practical response to untangling that not being likely feasible before 3.12beta1.

Linked PRs

• gh-99079: Update Windows build to use OpenSSL 3.0.9 #106649
• [3.12] gh-99079: Update Windows build to use OpenSSL 3.0.9 (GH-106649) #106680
• [3.11] gh-99079: Update Windows build to use OpenSSL 3.0.9 (GH-106649) #106761
• [3.12] gh-99079: Update macOS installer to use OpenSSL 3.0.9. #107472
• gh-99079: Update macOS installer to use OpenSSL 3.0.9. #107474
• [3.11] gh-99079: Update macOS installer to use OpenSSL 3.0.9. #107476
• [3.12] gh-99079: add What's New item #107481
• [3.11] gh-99079: add What's New item #107482

[corona10]

@gpshead Out of curiosity, Did we have any discussion about using boringSSL by vendoring them to the CPython repo?

[gpshead]

Did we have any discussion about using boringSSL by vendoring them to the CPython repo?

I'm not sure how well that would work out. boringSSL isn't really intended to be used by other projects, it doesn't guarantee a stable API. Even for security updates. And it lacks some interfaces and features.

test_ssl.py needs significant modification to "pass" when linked with boringSSL and various pieces of the _ssl extension code need modifications as well as workarounds to disable unsupported features in some places. We carry patches on our Python runtime internally at Google that does some this, but they wouldn't be acceptable as is upstream as they disable features that need to work for global users and skip some tests rather than making them work. If we want to consider official boringSSL support in the CPython repo as well that should get tracked in another issue.

[corona10]

it doesn't guarantee a stable AP

Oh, I thought that we can manage the boringSSL as the vendored library likewise libexpat or mimalloc.

We carry patches on our Python runtime internally at Google that does some this, but they wouldn't be acceptable as is upstream as they disable features that need to work for global users and skip some tests rather than making them work

Oh got it..

If we want to consider official boringSSL support in the CPython repo as well that should get tracked in another issue.

Yeah and we also need to write PEP also if we try to do, but we may need to know the possibility of the porting boringSSL into our repository first.

[gpshead]

it doesn't guarantee a stable AP

Oh, I thought that we can manage the boringSSL as the vendored library likewise libexpat or mimalloc.

We could and would if we did this. It's mostly noteworthy as a risk as it could involve some refactoring larger than we like to do within security fix releases several years down the line in order to pull in other updates.

[h-vetinari]

Did we have any discussion about using boringSSL by vendoring them to the CPython repo?

Now on DPO!

Moving away from OpenSSL completely has some large maintenance benefits (but IMO even larger downsides too).

Moving to boringSSL sounds like a very bad proposition to my mind. OpenSSL is a heavy dependency, but it goes very far out of its way in terms of stability (which nowadays includes being buildable everywhere¹), e.g. it promises:

• No API or ABI breaking changes are allowed in a minor or patch release.
• No existing public interface can be modified except where changes are unlikely to break source compatibility [...]
• No existing public interface can be removed until its replacement has been in place in an LTS stable release. The original interface must also have been documented as deprecated for at least 5 years.

I don't see sufficiently huge issues with OpenSSL (or sufficiently huge improvements with boringSSL) that a switch would look even remotely reasonable from my POV.

Footnotes

1. not least still being on C89 to support some ultra-exotic platforms without modern compilers ↩

[mzhao-dev]

I built python 3.11.1 with OPENSSL 3.0.7 on windows 10 platform, when I run test_ssl.py, it shows no OPENSSL_Applink, error and crashed.
I confirmed that applink.c file was added and complied, so I have no idea about that issue. I think maybe need some other code change about that?

#101401

[gpshead]

Marking this as a release blocker again because we really should do this before beta1 so that people test against a relevant representation of what the next release will contain.

[oraluben]

Can cpython limit openssl3 version? Some version of openssl3 (>3.0.8 && <= 3.1.0) breaks debuging libcrypto on Apple Silicon: openssl/openssl#20188

Since libcrypto is required by fundamental modules like _ssl and _hashlib in cpython, this totally breaks debugging cpython on Apple chips. I think version restriction is worth considering.

Update: This is not fixed in any release branch of openssl. According to other comments, not only debugging is affected, but I have no problem normally running with those openssl versions.

Update 2: Given the condition where this error would be triggered (compile from source, Apple chip, use openssl >= 3.1, debugging with LLDB), I guess a warning message would be sufficient, maybe near the The necessary bits to build these optional modules were not found: after build?

[arhadthedev]

@ronaldoussoren, @ned-deily here is a macOS-specific OpenSSL issue reported in the previous comment by @oraluben.

[h-vetinari]

Update: This is not fixed in any release branch of openssl.

This is fixed in a branch (master), but not in any released version. I made a comment in the issue if the fix can be backported to the next 3.1.x release, because OpenSSL 3.2 will probably still take a while.

[gpshead]

it looks like they've at least got a pending PR that'll backport it to 3.1, but it isn't clear to me exactly which version the problem was introduced in. @oraluben mentions >3.0.8 as okay but from the linked openssl issue it is not obvious to anyone not intimately familiar with the openssl workflow or codebase which versions which offending changes landed in (read: I'm not going to be that person).

Regardless, a configure.ac check on Darwin to avoid a specific version range makes sense.

[ned-deily]

I'm not sure why we should call this transitory issue out. Presumably it affects other projects than cPython as well and, in the end, we have no control over what version of OpenSSL or equivalent that builders or distributors use other than the python.org macOS installers which currently don't use OpenSSL 3.x anyway.

[oraluben]

but it isn't clear to me exactly which version the problem was introduced in

Well, I also misunderstood in my first comment. Here's my current understanding:

tldr: now, and hopefully only openssl 3.1.0 is affected.

Some instruction probing logic breaks debugging in darwin-aarch64, these commits are in the dev branch of openssl (master, SVE: openssl/openssl@b1b2146, RNG: openssl/openssl@efa1f22), and have been cherry-picked to 3.1 release branch (openssl-3.1, SVE: openssl/openssl@a14eff6, RNG: openssl/openssl@45c74dea). They haven't been cherry-picked to 3.0, so openssl 3.0.x is not affected.

The fix (openssl/openssl@52a3814) is also in master, I submitted a PR to cherry-pick it in 3.1 (openssl/openssl#20948), if it's merged, the only affected version would be 3.1.0. I guess 3.0.x would be ok forever, if openssl decided to apply those instruction tests I expect they won't forget the corresponding fix.

[arhadthedev]

My pardon for an attempt to reassign this issue to 3.13, I missed that it's a release blocker for 3.12.

However, will it be possible to resolve the child gh-90728 tomorrow at best, the next week at worst?

[gpshead]

deferring this until after beta1.

[ned-deily]

What's the status of macOS? Python 3.12 and 3.13 are now built with OpenSSL 3.0 on Windows, but OpenSSL 1.1 on macOS.

See comment above. Currently, AFAIK, we are now using OpenSSL 3.x for all macOS Github CI builds and buildbots. We also plan to provide OpenSSL 3.x for the Windows and macOS 3.12.0rc1 and future 3.11.x release binaries. (What is in build-installer.py has no effect on macOS CI or buildbots.) Are you seeing something different, @vstinner?

[vstinner]

(What is in build-installer.py has no effect on macOS CI or buildbots.)

The issue title is "Use OpenSSL 3.0.x in our binary builds": build-installer.py is used to build the macOS binary distributed on python.org, no?

[vstinner]

We also plan to provide OpenSSL 3.x for the Windows and macOS 3.12.0rc1 and future 3.11.x release binaries.

Ok, great.

[ned-deily]

I think we are done here: as of 3.12.0rc1 and 3.11.5, Windows builds and downloads from python.org for Windows and macOS should be using OpenSSL 3.0 (currently 3.0.9).

[lennin-cp]

What's the plan to update OpenSSL in supported versions 3.8, 3.9, and 3.10?

[zware]

What's the plan to update OpenSSL in supported versions 3.8, 3.9, and 3.10?

None: binaries are no longer produced for these versions, and the code is already compatible (enough).

[gamer191]

Any plans to update to Openssl 3.1.x or 3.2.x?

[gpshead]

Probably not until OpenSSL declares another Long Term Support version as 3.0.x is per https://www.openssl.org/policies/releasestrat.html