freeipa not starting on podman 5.2.2

[leonidas-o]

When using podman version 4.9.4-rhel the container starts without any issues

podman logs freeipa-server
mode of '/tmp/var' changed from 1755 (rwxr-xr-t) to 0755 (rwxr-xr-x)
Sat Dec  7 13:43:21 UTC 2024 /usr/sbin/ipa-server-configure-first upgrade
FreeIPA server is already configured, starting the services.
Sat Dec  7 13:44:08 UTC 2024 /usr/sbin/ipa-server-configure-first update-self-ip-address
freeipa.my-domain.com has address 10.1.1.16
FreeIPA server started.

But when using the same systemd file or the same podman run command with podman version 5.2.2. The container gets killed after x seconds. Interesting to see, if using -e DEBUG_NO_EXIT=1 the container logs stop like shown below. Freeipa even seems to work, I could open the UI, login etc. Without the DEBUG_NO_EXIT flag, it gets killed all the time.

podman logs -f freeipa-server

mode of '/tmp/var' changed from 1755 (rwxr-xr-t) to 0755 (rwxr-xr-x)
systemd 252-46.el9_5.2.0.1 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.
Queued start job for default target Minimal target for containerized FreeIPA server.
Sat Dec  7 13:29:25 UTC 2024 /usr/sbin/ipa-server-configure-first upgrade
/etc/resolv.conf /data/etc/resolv.conf.ipa differ: byte 13, line 1
/usr/sbin/ipa-server-configure-first: line 142: /etc/resolv.conf: Read-only file system

ExecStart inside systemd file

ExecStart=/usr/bin/podman run \
        --cidfile=%t/%n.ctr-id \
        --cgroups=no-conmon \
        --rm \
        --sdnotify=conmon \
        -d \
        --replace \
        -e IPA_SERVER_IP=10.1.1.16 \
        -p 53:53/udp \
        -p 53:53 \
        -p 80:80 \
        -p 443:443 \
        -p 389:389 \
        -p 636:636 \
        -p 88:88 \
        -p 464:464 \
        -p 88:88/udp \
        -p 464:464/udp \
        -p 123:123/udp \
        --name freeipa-server \
        --hostname freeipa.my-domain.com \
        --read-only \
        -v /srv/freeipa/ipa-data:/data:Z \
        freeipa/freeipa-server:rocky-9 --no-ntp

Couldn't find anything in the podman release notes, which could help here. Anything I can do, or try? Appreciate any help.

[adelton]

Does the FreeIPA container have DNS server configured? If that's the case, the podman parameters should likely include --dns=127.0.0.1. That will in turn cause the update of the /etc/resolv.conf (which fails in your case) to be skipped.

As for why it works with one podman version and not the other -- is that on exactly the same host, or on a different machine? If the machines are different, is it possible that their /etc/resolv.conf (on the host) differ to start with?

[leonidas-o]

Sorry for the late reply. Freeipa is running rootless using the "freeipa-server:rocky-9" image on a VM. The VM has a self hosted DNS server configured. I entered the freeipa container and the /etc/resolv.conf contains two entries. I haven't used the --dns=... arg.

/etc/resolv.conf on host

nameserver 10.1.1.12

/etc/resolv.conf inside freeipa container

nameserver 10.0.2.3
nameserver 10.1.1.12

Yes, it's the exact same VM. I've just done a dnf upgrade and that's it.

[adelton]

So the DNS server is running outside of the FreeIPA container.

Hmm, our logic in https://github.com/freeipa/freeipa-container/blob/master/ipa-server-configure-first#L235 might be a bit broken for that case.

If you remove the /etc/resolv.conf.ipa file (or move it to a different name), can I assume that the problem goes away?

[leonidas-o]

The DNS server is running on a totally different VM.
I wonder where the 10.0.2.3 entry is coming from?
Nevertheless, what should I try? Update the host system (podman 5.2.2), then enter the freeipa container and remove what file exactly? /etc/resolv.conf OR /data/etc/resolv.conf.ipa

[adelton]

The 10.0.2.3 comes from slirp4netns(1) (or its pasta equivalent).

Remove the /data/etc/resolv.conf.ipa file. We create it after the initial ipa-server-install to signal to subsequent container starts that the nameserver in /etc/resolv.conf should be set to 127.0.0.1. That is not desired in your case and we will look into fixing that logic and that assumption, but removing that file should allow your FreeIPA container to start and run without choking on the step of updating the /etc/resolv.conf.

[leonidas-o]

Okay, I renamed that file to /data/etc/resolv.conf.ipa_BACKUP and it is working now. Thank you!
So I guess in one of the next releases this will be fixed and until then, I just have to check, that /data/etc/resolv.conf.ipa shouldn't exist (if a self hosted dns server is used outside freeipa container)?

[adelton]

Right.

The /data/etc/resolv.conf.ipa will not get recreated on an already-configured FreeIPA container, it would only be created if you created a new FreeIPA server / replica.

I filed #644 to track the future around resolv.conf.

[adelton]

But I would also point out -- do you need that -e IPA_SERVER_IP=10.1.1.16 at all?

Assuming that is an IP address of the VM and assuming that does not change, once the FreeIPA's DNS record is in the DNS zone, you can probably remove that parameter. That would also prevent the resolv.conf logic to kick in.

[adelton]

Last but not least -- if you want to have the resolv.conf content under control, you should be able to use --dns=10.1.1.12 -- that should fixate it to the value you want, without podman adding other records.

[leonidas-o]

@adelton accordig to -e IPA_SERVER_IP=10.1.1.16, yes it is the VM's ip, and no, actually it shouldn't change, but it could somewhere in the future. This way I immediately see that it is used in there. If I remove it, don't know If I could run one day in an issue with that.
Right now, I can't tell what the best approach would be, removing the IPA_SERVER env var, setting --dns or simply making a note, if the server does not come up check/remove /data/etc/resolv.conf.ipa file.