balenaEngine and NetworkManager forward chain racing issues

[majorz]

A NetworkManager profile that has Internet sharing enabled adds a number of iptables rules. Depending on when balenaEngine adds its iptables rules it may add them before or after NetworkManager.

If NetworkManagers iptables rules are added first in the FORWARD chain, that will make containers not be able to reach other devices on the network that has Internet sharing provided to. If balenaEngine adds its rules first, containers will be able to reach the other devices on that network.

Depending on whether NM or balenaEngine adds its rules first, the default policy for the FORWARD chain could be set differently as well.

The quick workaround for this is to add a rule from inside a container like: iptables -I FORWARD 1 -d <NETWORK> -o <IFACE> -j ACCEPT.

Preferably this can be also solved inside the host OS - for example balenaEngine could be started after NetworkManager finishes interface initialization. That will also give the ability for containers to bind to interfaces which with the current setup may or may not be initialized.

Another approach could be Prevent Docker from manipulating iptables. This means that another service (e.g. supervisor) has to add those rules instead.

Something else that could be investigated is what the behavior would be when using nftables as it is more flexible and supports priorities.