armcow64: dockerized

[DerLinkman]

Description:

This PR adds the ARM64 compatibility for our lovely mailcow: dockerized.

ARM64 does not need a manual configuration change in the mailcow Configs, all changes to be ARM64 compatible (not much) are included into this PR.

Phase Status:

1. Port mailcow to ARM64:
2. Test all components of mailcow on a ARM64 Server:
3. Publish Images for public tests:
4. Update Documentation Pages for ARM64:
5. Merges and final Tests:

Latest Events:

• August 7st 2023: We stumbled across a big issue with decrypting migrated e-mails with the new Alpine based dovecot images. Older E-Mails are not decryptable, new once are. It's not 100% clear why this happens but right know it looks like a non completed implementation of OpenSSL within Dovecot which is causing this strange behaviour as the new OS Versions (Alpine 3.17+ and Debian 12+ are now shipped with OpenSSL 3.X instead of 1.1.1x)
• August 2nd 2023: Within the latest Dev Build from Rspamd (3.6) the issue with Hyperscan/Vectorscan has finally been solved. Now we can concentrate on optimizing the rest.
• May 5th 2023: We've published the first ARM64 images (for testing to Docker Hub) which means that you can now test the ARM64 Port of mailcow. Simply clone mailcow as you normally would on a ARM64 Device and switch the branch with git checkout feat/arm64-cow. To generate/update your mailcow please use the --dev Parameter alongside the desired script to keep the current branch checked out instead of the master/nightly branch.

Estimated Release:

TBA

[BarsMonster]

Just tried to test it on Oracle cloud instance with fresh installation and got: " docker.errors.DockerException: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'" which is probably caused by incorrect urllib3 version inside "mailcowdockerized-dockerapi-mailcow-1". Not sure if it is caused by any ARM64 changes or it's an overall issue.

[DerLinkman]

DockerAPI is now running again it was indeed a urllib3 issue.

Can you try it again?

[BarsMonster]

DockerAPI is now running again it was indeed a urllib3 issue.

Can you try it again?

Yep, can confirm that 'chunked' issue is fixed.
Update: It looks like everything works smoothly. Thanks for making it possible on ARM64!

[BarsMonster]

It seem almost all functionality is working, except rspamd HyperScan library.
I am getting "bad hs database in /var/lib/rspamd/"[....] on each restart, even if I delete *.hs cache files - they are recreated and still invalid. Probably caused by architecture mismatch.

[rflrkn]

Just making sure: Can I update Mailcow on ARM64 using the normal update.sh-Script (with the --dev Parameter, obviously)?

[BarsMonster]

Just making sure: Can I update Mailcow on ARM64 using the normal update.sh-Script (with the --dev Parameter, obviously)?

worked for me

[DerLinkman]

It seem almost all functionality is working, except rspamd HyperScan library.

I am getting "bad hs database in /var/lib/rspamd/"[....] on each restart, even if I delete *.hs cache files - they are recreated and still invalid. Probably caused by architecture mismatch.

Jup also saw that during my testing phases. Don't know how to proceed with that. Yes we could deactivate the hyperscan functionality on ARM64 devices but that seems more like a workaround to me.

[BarsMonster]

It seem almost all functionality is working, except rspamd HyperScan library.
I am getting "bad hs database in /var/lib/rspamd/"[....] on each restart, even if I delete *.hs cache files - they are recreated and still invalid. Probably caused by architecture mismatch.

Jup also saw that during my testing phases. Don't know how to proceed with that. Yes we could deactivate the hyperscan functionality on ARM64 devices but that seems more like a workaround to me.

My understanding is that HyperScan is Intel's contribution, so supporting it on ARM64 is on us. I tried to contact rspamd developers with no luck. In my view disabling it is a valid short-term solution, as there is slower PCRE fallback. I was not able to disable it in a way that suppresses the error, "disable_hyperscan" option does not cut it.

I.e. currently I see this error in logs, but fallback works. I.e. seems no functionality is lost (just performance impact + errors in log).

Update: Apparently it must be disabled at compile time with -DENABLE_HYPERSCAN=OFF

[BarsMonster]

Yes we could deactivate the hyperscan functionality on ARM64 devices but that seems more like a workaround to me.

Apparently it is (theoretically) possible to use ARM64-aware fork of hyperscan: https://github.com/VectorCamp/vectorscan

Without replacing the library - HyperScan must be disabled with PCRE fallback to scare less people with unfixable error message. In HyperScan official repo Intel officially declined to support or merge in ARM64 support (which is not surprising).

[jfreax]

Tried it on Hetzner and it works great! Thanks for that.

There were only a small hiccup with the restore helper script. It tries to fetch mailcow/backup:latest which is only available for x86 at the moment. But tag 1.2 is indeed available as arm64. Changing the version on top of ./helper-scripts/backup_and_restore.sh did the trick. Restoring went fine afterwards.

[DerLinkman]

isable it in a way that suppresses the error, "disable_hyperscan" option does not cut it.

I.e. currently I see this error in logs, but fallback works. I.e. seems no functionality is lost (just performance impact + errors in log).

Update: Apparently it must be disabled at compile time with -DENABLE_HYPERSCAN=OFF

Or we keep this error and state it as won't fix. I don't want to disable this feature for x86 too (our bigger userbase).

Theoretically we could compile it for ourself but i don't see the necessarity here...

[BarsMonster]

@DerLinkman Unfortunately, it has more impact than just error message. These HyperScan files are continuously re-generated, which takes CPU time and disk space.

The only solution is -DENABLE_HYPERSCAN=OFF added conditionally, only for ARM64 (or for all platforms which are not x86 / x86_64). This would also enable it later to work on RiskV servers, which will come few years down the line.

[DerLinkman]

Is Vectorscan compatible with x86 or is that only compatible for ARM(64)?

[BarsMonster]

@DerLinkman
Vectorscan claim to fame is portability, so it is definitely not just ARM64.

But x86_32 is not supported: "32-bit x86 is not supported by Vectorscan, we do not test or include it in the CI builds, please use Hyperscan instead if you need 32-bit x86."

x86_64 Seems to be supported, few bugs were found and fixed.

[DerLinkman]

It seem almost all functionality is working, except rspamd HyperScan library. I am getting "bad hs database in /var/lib/rspamd/"[....] on each restart, even if I delete *.hs cache files - they are recreated and still invalid. Probably caused by architecture mismatch.

Let's see what it brings:

rspamd/rspamd#4493

[Vcele]

Should one currently use the arm64-Version? Are there expected security problems, or severe functionality problems?

[BarsMonster]

Should one currently use the arm64-Version? Are there expected security problems, or severe functionality problems?

I am using it at the moment. No issues found that affect functionality. rspamd performance can be reduced, but for <1'000'000 emails per day it should be ok.

[FieryCod]

Getting an error for SSL certificate. Is it connected with ARM64 or the user error on my side?

blockether-mail-acme-mailcow-1  | Traceback (most recent call last):
blockether-mail-acme-mailcow-1  |   File "/usr/bin/acme-tiny", line 8, in <module>
blockether-mail-acme-mailcow-1  |     sys.exit(main())
blockether-mail-acme-mailcow-1  |   File "/usr/lib/python3.10/site-packages/acme_tiny.py", line 195, in main
blockether-mail-acme-mailcow-1  |     signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact, check_port=args.check_port)
blockether-mail-acme-mailcow-1  |   File "/usr/lib/python3.10/site-packages/acme_tiny.py", line 90, in get_crt
blockether-mail-acme-mailcow-1  |     out = _cmd(["openssl", "req", "-in", csr, "-noout", "-text"], err_msg="Error loading {0}".format(csr))
blockether-mail-acme-mailcow-1  |   File "/usr/lib/python3.10/site-packages/acme_tiny.py", line 28, in _cmd
blockether-mail-acme-mailcow-1  |     raise IOError("{0}\n{1}".format(err_msg, err))
blockether-mail-acme-mailcow-1  | OSError: Error loading /var/lib/acme/mail.blockether.io/acme.csr
blockether-mail-acme-mailcow-1  | b'Unable to load X509 request\n5810779DFFFF0000:error:0480006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:763:Expecting: CERTIFICATE REQUEST\n'

[deanpcmad]

Have just tried the branch on a clean Ubuntu 22.04 ARM server on Hetner Cloud and I'm having issues. It keeps getting stuck when trying to connect to the database, but the database is online:

mailcowdockerized-watchdog-mailcow-1   | Waiting for SQL...
mailcowdockerized-postfix-mailcow-1    | Waiting for database to come up...
mailcowdockerized-php-fpm-mailcow-1    | Waiting for SQL...
mailcowdockerized-sogo-mailcow-1       | Waiting for database to come up...
mailcowdockerized-dovecot-mailcow-1    | Waiting for database to come up...
mailcowdockerized-watchdog-mailcow-1   | Waiting for SQL...
mailcowdockerized-rspamd-mailcow-1     | Waiting for PHP on port 9001...
mailcowdockerized-postfix-mailcow-1    | Waiting for database to come up...
mailcowdockerized-php-fpm-mailcow-1    | Waiting for SQL...
mailcowdockerized-sogo-mailcow-1       | Waiting for database to come up...
mailcowdockerized-dovecot-mailcow-1    | Waiting for database to come up...
mailcowdockerized-watchdog-mailcow-1   | Waiting for SQL...
mailcowdockerized-postfix-mailcow-1    | Waiting for database to come up...
mailcowdockerized-php-fpm-mailcow-1    | Waiting for SQL...
mailcowdockerized-sogo-mailcow-1       | Waiting for database to come up...
mailcowdockerized-dovecot-mailcow-1    | Waiting for database to come up...
mailcowdockerized-rspamd-mailcow-1     | Waiting for PHP on port 9001...
mailcowdockerized-watchdog-mailcow-1   | Waiting for SQL...

Any ideas?

[BarsMonster]

Any ideas?

I am pretty sure it is not related to ARM64. I would investigate using regular mailcow troubleshooting. i.e. inspect SQL logs in more details, and try to connect manually from SQL docker / from postfix docker.

[deanpcmad]

You're right, I ended up completely removing all volumes and re-starting. All seems well!
Will this be in the next version?

[BarsMonster]

@DerLinkman I see that there was a commit in rspamd on 25th of May to try to fix hyperscan, docker image mailcow/rspamd though is still from 12th. I assume the straightforward way to test it is to update docker image to current version and test?

Also, hope to see this pushed to mainline for mailcow. Even if we have to disable hyperscan when building rspamd as a temporary measure. My server is working flawlessly in the last 2 months, with no updates though :-)

[DerLinkman]

I see that there was a commit in rspamd on 25th of May to try to fix hyperscan, docker image mailcow/rspamd though is still from 12th. I assume the straightforward way to test it is to update docker image to current version and test?

Just tested the most current Rspamd Repo Package (from July 4th) and the issue still persists im afraid.

Sorry but this will delay the ARM64 thing as i don't want to disable Hyperscan for Rspamd as it unessecarily complicates the building process for now.

[DerLinkman]

New PR: #5587 (final one to be merged)