Wrong date/time in docker container

[ToonMeynen]

Hello,
I can't join my network. I get following error.
zerotier-one | connect: Operation timed out
I believe that part of the problem is that my clock/date is incorrect in the docker container.

docker-compose file:

version: '3'
services:
  zerotier:
    image: "zyclonite/zerotier:router"
    container_name: zerotier-one
    devices:
      - /dev/net/tun
    network_mode: host
    volumes:
      - '/var/lib/zerotier-one:/var/lib/zerotier-one'
      - '/etc/timezone:/etc/timezone:ro'
      - '/etc/localtime:/etc/localtime:ro'
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
      - NET_RAW
    restart: unless-stopped
    environment:
      - TZ=Europe/Brussels
      - PUID=999
      - PGID=994
      - ZEROTIER_ONE_LOCAL_PHYS=eth0
      - ZEROTIER_ONE_USE_IPTABLES_NFT=false
      - ZEROTIER_ONE_GATEWAY_MODE=inbound
      - ZEROTIER_ONE_NETWORK_IDS=MYIDXXXXXX

Logs when I run the container

pi@raspberrypi:~/ZeroTier $ docker-compose up
zerotier-one is up-to-date
Attaching to zerotier-one
zerotier-one | changed ownership of '/var/lib/zerotier-one/peers.d/778cde7190.peer' to 999:994
zerotier-one | changed ownership of '/var/lib/zerotier-one/peers.d/cafe9efeb9.peer' to 999:994
zerotier-one | changed ownership of '/var/lib/zerotier-one/peers.d/62f865ae71.peer' to 999:994
zerotier-one | changed ownership of '/var/lib/zerotier-one/peers.d/cafe04eba9.peer' to 999:994
zerotier-one | Sun Jan  0 00:00:00  1900 - launching ZeroTier-One in routing mode
zerotier-one | adding iptables rules for inbound traffic (ZeroTier to local interfaces eth0)
zerotier-one | Sun Jan  0 00:00:00  1900 - ZeroTier daemon is running as process 16
zerotier-one | connect: Operation timed out
zerotier-one | connect: Operation timed out
zerotier-one | connect: Operation timed out

[Paraphraser]

Here's what I'd suggest. First, make these edits:

You won't find the timezone/localtime volumes paths in the reference service definition. I don't have those lines in my own service definition and my container still has the correct local time:

I also don't see bad timestamps in my log:

I couldn't replicate the problem of bad timestamps when I added those timezone/localtime volumes paths on my test system so I can't say whether they are causing your container to show the bad timestamps. All I can really say is that the ZeroTier container does show correct time without those timezone/localtime volumes paths.

But I doubt that that's implicated in your timeouts.

As for changing ZEROTIER_ONE_USE_IPTABLES_NFT to true, that comes from Environment variables:

Your logs show you are running on a Raspberry Pi but there's no indication of which version of Raspberry Pi OS you are running so I'd like to make it clear that, while the words "This is needed on Raspberry Pi Bullseye" might sound like it only applies to Bullseye, it really means that it has only been tested on Raspberry Pi OS Bullseye. Setting this true might also be needed on Buster and Bookend.

It was never clear why trying to set up the IPTables rules by calling iptables didn't work on the test system (a Raspberry Pi running Bullseye), while calling iptables-nft did work. The variable defaults to false to maintain backwards compatibility but it's likely that setting it true will actually work in most if not all situations, as well as on the Pi where it is required.

However, iptables/iptables-nft is only used to set up local NAT rules so I doubt that that's implicated in your timeouts.

---

I've tested various hypotheses trying to replicate what you're seeing, including:

1. Letting ZeroTier initialise on the Pi, taking it down, going to ZeroTier Central and deleting (rather than authorising) the client, then going back to the Pi and bringing up the container again. In theory, the client would not be able to join the ZeroTier Cloud network.
2. Re-initialising the client with a clean slate, authorising it in ZeroTier Central, taking the client down, adding net-filter rules to block port 9993, and starting the client again. In theory, blocking port 9993 should confuse it a bit but it keeps working.

Basically, no matter what I do, I can't get my client to time-out like that.

What do you see in ZeroTier Central for this host?

I assume this host has been authorised?

I think what I would do is start from a clean slate:

1. Terminate the container.
2. Erase /var/lib/zerotier-one.
3. Go to ZeroTier Central and delete the authorisation.
4. Bring up the container again.
5. Go to ZeroTier Central and authorise the new client.

If that doesn't help then I'd be asking questions like:

1. Is a local firewall blocking traffic (eg port 9993)?
2. Does ZEROTIER_ONE_LOCAL_PHYS actually match the Pi's physical interfaces?

[Paraphraser]

Returning to the timezone topic…

Although I understand what the pattern of mapping /etc/timezone and /etc/localtime is trying to achieve, and while I'm not going to claim that that approach will never work, every time I've chased down why a container is not showing local time, the cause has turned out to be the absence of the tzdata package in the underlying image. If you run docker exec zerotier-one apk list tzdata you'll see tzdata is installed so just providing a valid TZ via a docker/docker-compose environment variable gets the job done.

On Discord someone once explained to me that they had mapped /etc/timezone and /etc/localtime into all their containers. Their goal was to try to get the containers to follow the Pi automatically and they couldn't understand why it sometimes seemed to work, sometimes didn't and, occasionally, could even cause a container to crash. My guess was that the situations where it seemed to work would have responded to TZ, those where it didn't were probably explained by the lack of tzdata, and those cases where containers crashed would need more research.

I think the best general-purpose solution to the TZ= problem is this:

1. Change all your service definitions to use:

   - TZ=${TZ:-Etc/UTC}

2. In the same directory where your docker-compose.yml is located, create the "hidden" file .env with the content:

   TZ=Europe/Brussels
   

---

Just having the correct timezone inside a container doesn't guarantee that the container's time will always match the host's. Whenever a sudo apt upgrade on your Pi installs a new tzdata, your per-container tzdata files are going to be out-of-sync until all the Dockerhub images you rely on are updated and/or your re-run your local Dockerfiles.

Being out-of-sync usually doesn't matter because it's probably not your timezone that was being updated. However, it's something to be aware of if your country changes when you start or stop daylight saving. Any time politicians fuzt with timezones, there's always a risk of incorrect time calculations until everything catches up.