ARM64 Support

[extremeshok]

I have extensive experience with docker on arm, so before I get to work on this, has anyone got mailcow-dockerized ported or running on arm ?

Did a quick search through the repo and I do not see any references to arm.

Does the repo owner have any issues against arm support being added ?

Thanks

[mkuron]

A Raspberry Pi doesn’t have enough memory. You need around 1GB if you disable ClamAV or 2GB if you don’t. Recent Odroid models should be sufficiently equipped though.

Please give it a try and open a pull request if it works out. You‘ll need to swap out the base images and use arm instead of amd64 repositories where available and compile yourself where not. Just don’t duplicate all the Dockerfiles as that would create a maintenance nightmare.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[MatthiasLohr]

@extremeshok Did anything happen here?

[David91919]

RAM issue on RPI is solved by now, would be great to see MC on RPI

[andryyy]

RAM is/was the smallest problem.
Maintenance is another, but the biggest problem is missing software for ARM.

I won't maintain mailcow on ARM, but feel free to port it and create a PR etc. - I think SOGo won't work reliable on ARM though.

[Braintelligence]

Some people even have issues with 4GB RAM if they use ClamAV and SOLR, so I wouldn't say that RPi4 would have none at all.

[Adorfer]

Until ARM-based 1U servers (with decent amount of RAM) become largely availabe at reasonable cost, ARM is not a suitable platform for mailservers on (small) enterprise level.

[bolet]

I have the mailcow stack running on armhf (odroid hc1, 8-core, 2gb), with, as expected, the usual resource hungry services disabled. It consumes less than 1gb, and I plan to run it on a (small) swarm for better resilience (with glusterfs as persistent storage). I'd argue that arm is a perfect platform for small traffic servers, which is my case.

The major part was rebasing debian/strecth-slim and ubuntu/bionic services on debian/buster-slim (the original distros were lacking armhf versions of some required packages). The other part was building mariadb:10.3, which also resulted in rebasing on debian/buster-slim. I believe the rebase on a common distro would benefit the consistency of the project.

Anyway, I'd be glad to share the patch if anyone is still interested.

[Adorfer]

offereing "state of the art" anti-spam and FTS are the key features of mailcow (and to offer it as a real alternative to hosting at gmail/M365 etc).
getting rid of that sounds like offering a modding/patch for a semi-truck getting rid of the hook and any lockable doors. Of course it's still a truck, but does not fit in most use cases.

[bolet]

Let's not be dictators! What about letting users decide how they want to configure mailcow? Isn't it one of the freedoms permitted by "free software"? :-)

I'll personally be happy to enable all the features when I switch to a higher end (arm64 based) server, but until then I still prefer using mailcow stripped from a few features than configuring a mail server from scratch (which is what I've been running for about 20 years).

[luckyluc74]

@bolet Would be great, if you could share the patch for mailcow stack for armhf. And I have to agree with you that there are users that are very interested to see mailcow stack running on armhf/aarch64, i am one of those users. I am currently Running 3 raspberry pi 4 (4GB) in swarm and would be nice to run arm version of the mailcow stack on it.

Very recently Ubuntu released version 19.10 with raspberry pi 4 support ( aarch64) . Would be great to also see a aarch64 version of mailcow.

[bolet]

@luckyluc74 Here is the patch : buster+arm.zip

Sogo image from mailcow runs but is broken because it still pulls amd64 code (I keep getting 502 errors).

This project builds sogo from sources and is a good starting point. But then I found out that debian:buster includes pretty recent (4.0.7) Sogo packages, so here is my patch to this last project.

All pieces seem to work now, but I haven't yet taken the time to put them all together. I'll keep this bug posted as I move on, but feel free to help :-)

[luckyluc74]

@bolet thank you for sharing the patch and extra info. That SOHO is not working is ok for me, because I am going to use Roundcube instead of SOHO anyway.

[bolet]

@luckyluc74 I since forgot about this patch. It has to be applied to julienfastre project which results in a working arm sogo build. Then use it with the rest of mailcow.

Since then I also tested mailu which is more lightweight than mailcow, and more fit to rpi like servers.

[luckyluc74]

@bolet thanks for the update and information, appreciated! I have to say I like Mailcow. But mailu is also nice.

[Xeroxxx]

So now (acutally since May 2020) Raspberry Pi 4 with 8GB and VMware ESXi on ARM is available since October I would be happy to see offical support :)

Works patched within a ARM VM like a charm except SoGo (patch above) :)

[bonanza123]

@Xeroxxx I am trying to run mailcow on RPi4 (8GB). Could you link me to the correct docker images to use?

[bolet]

@bonanza123 As far as I know, there are no working mailcow images for any ARM platform. My previous patches still need to be applied and the image built on your device.

The maintainers of mailcow don't seem eager to include ARM support, and though I am happy to participate here and there in various projects, I don't have the resources to maintain an up to date fork of mailcow. This unfortunate situation is common in open source projects...

PS: With Apple's M1 chip showing ARM can outperform Intel/AMD chips (at equal watts), there is hope for a growing consideration for ARM.

[mkuron]

The maintainers of mailcow don't seem eager to include ARM support

The only place you could currently run it on right now is pretty much a Raspberry Pi at home, and we really don‘t encourage running a mail server at home. ARM rackmount or cloud servers are still not common (and not cheaper or higher performance than x86). As long as nobody makes a compelling case beyond "nice to have", I still see no reason to add ARM support to Mailcow. It would take additional effort for testing, and as long as nobody is going to use it productively, that‘s not worth it.

With Apple's M1 chip showing ARM can outperform Intel/AMD chips (at equal watts), there is hope for a growing consideration for ARM.

Not a typical Linux machine, unfortunately. It‘s a very nice chip though and I hope we will at some point see similar chip designs in the server market.

[bolet]

we really don‘t encourage running a mail server at home

That's a sensible opinion, but please have some consideration for ppl with a different one. My experience running a home mail server has been fine for over 20 years (with admittedly a friend running a secondary MX).

It would take additional effort for testing

I can understand that. Maybe a solution would be to discard official support for ARM, and let the community take care of this specific platform? I'd argue the whole project would benefit from this. It's a somewhat similar situation to supporting more compilers, it usually ends up making the code more robust.

In my case, though mailcow was my 1st choice, I ended up installing mailu on my home ARM swarm, and I got quite familiar with it (I even patched most images). Months later, I had to deploy a mail server for my company, and because my knowledge of mailu, I also installed it there even though we had a high end x86 server. Don't you I think it's a pity?

[MatthiasHertel]

The maintainers of mailcow don't seem eager to include ARM support

The only place you could currently run it on right now is pretty much a Raspberry Pi at home, and we really don‘t encourage running a mail server at home. ARM rackmount or cloud servers are still not common (and not cheaper or higher performance than x86). As long as nobody makes a compelling case beyond "nice to have", I still see no reason to add ARM support to Mailcow. It would take additional effort for testing, and as long as nobody is going to use it productively, that‘s not worth it.

With Apple's M1 chip showing ARM can outperform Intel/AMD chips (at equal watts), there is hope for a growing consideration for ARM.

Not a typical Linux machine, unfortunately. It‘s a very nice chip though and I hope we will at some point see similar chip designs in the server market.

@mkuron

i would be very happy about the possibility to deploy mailcow on arm - short - thumbs up and feature request for arm! it is absolutely nice to have for me!

[maroci]

I'm adding my wish to other requests to have armhf image of MailCow. :) I really hope to get one to test soon. Thank you

[alfonsrv]

Any more recent patch @bolet? Possibly maybe even with SOGO recompiled? 🚀

[bolet]

As I said earlier, I can't afford to maintain a fork of mailcow for arm.
If one day the official mailcow project wants to support arm, I'll be happy to participate.
The main hurdle is the inclusion of pre-compiled amd64 binaries in the build process.
Just compiling those binaries at build-time is all it takes to have mailcow working on all architectures.

[tehsunnliu]

+1 for ARM

[maroci]

@bolet I would like to learn to do what you said. I don't have knowledge to do that no though though.
If anyone has time and patience to teach me how, I would like to give a try.
Even few links to a tutorial.

[alfonsrv]

Unfortunately this is the only book that teaches you what there is to do to make mailcow-dockerized run on ARM.
I think honestly @bolet's patch should still work fine and produce up-to-date containers.

[axkng]

Is there any update on this from the developers?

ARM is gaining more and more momentum in the datacenter.
Graviton instances on AWS have high sales and Oracle is offering a free tier that gives you a lot power of on ARM at zero cost.
So I would say this will be a major thing in the near future.

[jebabin]

ARM64 is now in beta if you missed it: https://mailcow.email/posts/2023/arm64-open-beta/

[pedroluccasc]

any ETA to stable?

[w4sb0y]

Hello there, I'm trying to run mailcow arm64 on a raspberry pi 4, sending w/ mailgun. I can send, but cannot recive emails, and we pretty much eliminated the possible problems. This popped up in the postfix logs. Is it possible, that a library isn't arm compatible?

[KeldorDE]

Since today all mails are rejected without changing anything. In the rsapmd history all mails are rejected with the status "ratelimit". The ratelimit for each domain are disabled.

I could only stop the behavior by removing the content of the file data/config/rspamd/override.d/ratelimit.conf completely.
I am not sure if the problem is related to the ARM version.

[APoniatowski]

no idea if this is related to the arm64 image, but after an update today, my UI broke (luckily accessing mailboxes via clients still work), but SOGo keeps restarting with exec /docker-entrypoint.sh: exec format error and nginx as well with ping: bad address 'sogo', so the UI refuses to work
in other words,
sogo fails -> exec /docker-entrypoint.sh: exec format error -> restart
nginx fails -> ping: bad address 'sogo' -> restarts
whole frontend stops working, due to reverse-proxy restarting
[UPDATE] - after a long period of time, the admin UI came back up, but SOGo stopped working. still not sure if it is an ARM64 image issue, or is AMD64 also affected

[DerLinkman]

Hi guys 👋🏻

Just a small "update" which is no update actually. The development of this has staled as we encountered a critical issue which is also regarding all x86 mailcow instances in the future. It's about the change of openssl from 1.1.X to 3.X and dovecot's encryption behavior of old mails, as for some reasons the old mails won't get decrypted with the exact same key with OS running the newer openssl Version (Debian 12, Alpine 3.17+ and so on).

So yeah that is the current stopping point here. If anyone have an idea what we could do or even better has a solution for this feel free to contribute that here.

In worst case we have to encrypt the mails first to decrypt them on openssl3 with a newly generated keypair automatically during some update process.

Let's hope the best 🤞🏻

[jebabin]

I was starting to look at this openssl issue and found you very recent message on the dovecot mailing list https://dovecot.org/mailman3/archives/list/[email protected]/thread/QCY226M4JCDDMQ2KDNMNTR3XSTOSQOPS/
I was wondering if there is a mailcow issue open about this (I did not found one) ? Some people might be interrested by tracking the resolution

[DerLinkman]

Haha that was me :)

So yeah it is known. No a solution is near the horizon but not clear when it will drop.

Let me quickly sum up what happend in the past week:

We finally analysed the issue and tracked it down. It is indeed a openssl 3.0 issue as my first speculation guessed. Dovecot or to be exact the libcrypt Module is not OpenSSL 3.0 compatible yet, that's the reason for it's strange behavior.

A fix is already upon the horizon, as i already asked the Alpine Devs (we want to switch to Alpine Base for dovecot with ARM64) to adapt a specific patch which allows the Crypt Module to be compliant with openssl 3.X (see here: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15365)

Until this isn't fixed we won't continue the development. We could use Debian's Repo Versions of dovecot but that would only temporarily fix the issue, so we skip that.

[DerLinkman]

Good news everyone! The Issue and it's workaround approach has been merged within Alpines latest Edge release
(see https://gitlab.alpinelinux.org/alpine/aports/-/commit/815bb154f5337786fa025d420a0399696dfc0149).

I already tested it and it works! All mails from openssl 1.1.1x can now be decrypted again. But that still does not mean we're stable enough to release it.

There are still tests + the documentation left before this thing is going into prod.

[squadramunter]

@DerLinkman So this means there is still a change of getting ARM64 support in 2023 as a stable release? Referring to this news post: https://mailcow.email/posts/2023/arm64-delay/

[DerLinkman]

I don't think so. We now might have anything done to be compatible on ARM64 and x86 but we need to finalize the docs + do tests with all final components.

[Knight1]

I am encountering the "no private key available" issue with my mailcow.

I recently set up this instance from the "nightly" build on September 25th, and I migrated all emails to this mailcow instance using imapcopy. Initially, it ran on dovecot:1.25 for 3 months until I updated it to branch feat/arm64 with update.sh and --nightly flag tonight due to issue #5596.

Everything was running smoothly until I was not able to read emails in sogo, despite the fix in the Alpine Edge Version.
I

• Confirmed that the fixed version was installed; the running version was 2.3.20-r17, and the fix was included in 2.3.20-r15.
• Built the mailcow-dovecot container locally from alpine:edge.
• Updated the container post-deployment with nightly-20231122.

The only solution was to revert the Dovecot container image to version nightly-1.25 from the nightly-20231122 in nightly branch, and it started working again. Currently, it's running on Dovecot Version 2.3.20-r10 with Alpine 3.18.

Anything I missed or do I need to migrate again with the patched version?

[DerLinkman]

The problem is: it's really hard to reproduce as it works with my tests I did in the past. It even worked after I switched my staging mailcow to the separate ARM branch. So it basically ran on the same Maschine. However, it is important to work with a existing mailcow flawlessly as we are changing the basic image for dovecot for all users not only ARM64 users.

Could you share more details about how you exactly set the builds up and migrated your mails? It needs the same decrypting key to work, but that is also a dependency of the current migration plan.

[Knight1]

Hey Niklas!

We used https://mailcow.email/posts/2023/arm64-open-beta/ and setup a fresh new Host and migrated with imapcopy from non mailcow Provider.

So the Encryption Key was generated with the version that is not compatible with the current stable system. So we basically bootstrapped a host from the nightly build with the broken encryption setup and i tried to migrate to the encryption setup that currently works. So this might be the core Problem here.

Strange thing i missed in my post was that i got an email while the host was on the new dovecot version. It could read that email because my mac fetched that email and still has it locally. But sogo now cannot display that same Message with the old dovecot version because "no private key available". Which is totally Strange because the key was never touched.

[DerLinkman]

Hi,

ok. If i get you right you've setup the nightly version before we introduced the change of Dovecot with the OpenSSL 3.0 Patch (which was on October 24th 2023). Yeah then it's needed to update the images again. But i would recommend building the images by yourself for now.

You can do that while you copy the docker-compose.override.yml from the Folder /opt/mailcow-dockerized/helper-scripts/docker-compose.override.yml.d/BUILD_FLAGS to your mailcow Root directory. If you made custom changes inside your docker-compose.override.yml already you might adapt those building flags to your existing config.

My tests including the latest staging branch for x86 and then switching to the newly ARM64 branch followed by building the images on the machine to have the images with the exact state of the development.

[pkprotoplasm]

In case anybody else will be upgrading from the old beta arm64 stuff -- I was able to avoid encryption issues by decrypting the emails prior to performing the upgrade to the current nightly functionality. The steps to do so are documented in https://docs.mailcow.email/manual-guides/Dovecot/u_e-dovecot-mail-crypt/ (remove the -regextype egrep option to find for busybox compatibility). After the upgrade, you may encrypt the emails again using the steps on that page as well, and this could be done on a live system as long as your server has the capacity to do so.

And if you've upgraded already and need an emergency rollback to perform the decryption, you could temporarily change the dovecot image version in docker-compose.yml to mailcow/dovecot:nightly-20231011, decrypt, then change back to current, and encrypt.

Beware the risks when fussing about with docker stuff.

[rflrkn]

I tried doing that (using mailcow/dovecot:nightly-20231011), but got the following error:

Fatal: fs_init() failed: compress: Compression method 'lz4' not supported.

[DerLinkman]

That version is very old.

Try the version: mailcow/dovecot:nightly-20231122

[rflrkn]

I used that version because of the recommendation made by @pkprotoplasm - with the latest version I cannot encrypt my mails properly any longer. (I can't read them using SOGo or any other client at least.)

[DerLinkman]

Introduced within 2024-01

[aleksasiriski]

I've just migrated my mailcow-dockerized setup from x86 to arm (on Hetzner, using helper script backup-and-restore) and SOGo works perfectly but the admin UI redirects to /debug and is completely blank (white)?

[DerLinkman]

I've just migrated my mailcow-dockerized setup from x86 to arm (on Hetzner, using helper script backup-and-restore) and SOGo works perfectly but the admin UI redirects to /debug and is completely blank (white)?

Sounds like a non Update problem maybe head over to https://t.me/mailcow to get help there.

[aleksasiriski]

Unfortunately I don't have telegram (I would prefer to not have to install it), I've opened #5643 instead. It's unrelated to the backup-and-restore, since it happens on a clean install as well (following the install instructions from the docs).

[KeldorDE]

It's just a small visual bug, but the Arch column still shows the warning symbol behind the aarch64 on the current release version like the nightly build :)

[maxkratz]

It's just a small visual bug, but the Arch column still shows the warning symbol behind the aarch64 on the current release version like the nightly build :)

As far as I understand this comment, this is desired behavior.

[DerLinkman]

It's just a small visual bug, but the Arch column still shows the warning symbol behind the aarch64 on the current release version like the nightly build :)

Yes that is correct. As there are maybe some small bugs left behind we keep it for the next 1-2 releases until we drop it. Or short: just like @maxkratz said!