Consul docker image doesn't work with hyperctl or docker+runv

[svarlamov]

I've been struggling to debug an issue with the standard consul image when running on docker+runv and/or pure hyperctl stack.

First of all, when running docker+runc (on the same fresh Ubuntu 16.04 bare metal box for all these examples -- I also tried on centos 7 bare metal and two other ubuntu 16.04 bare metal boxes), I get the following successful start:

root@hyperdev-02:~# docker run -d   --name consul   -p "8500:8500"   -h "consul"   consul agent -server -bootstrap -client "0.0.0.0"
a991885eddd23412dfceb3b5bbd56d1e52b0ef67e29ae54405e3012da2576c7b
root@hyperdev-02:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                      NAMES
a991885eddd2        consul              "docker-entrypoint..."   45 seconds ago      Up 21 seconds       8300-8302/tcp, 8301-8302/udp, 8600/tcp, 8600/udp, 0.0.0.0:8500->8500/tcp   consul
root@hyperdev-02:~# docker logs a991885eddd2
bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.1.0'
           Node ID: 'fb56a8fd-1dcc-70ae-231e-19f4fc012a14'
         Node name: 'consul'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: true)
       Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 172.17.0.2 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/05/27 11:45:24 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:fb56a8fd-1dcc-70ae-231e-19f4fc012a14 Address:172.17.0.2:8300}]
    2018/05/27 11:45:24 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state (Leader: "")
    2018/05/27 11:45:24 [INFO] serf: EventMemberJoin: consul.dc1 172.17.0.2
    2018/05/27 11:45:24 [INFO] serf: EventMemberJoin: consul 172.17.0.2
    2018/05/27 11:45:24 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
    2018/05/27 11:45:24 [INFO] consul: Adding LAN server consul (Addr: tcp/172.17.0.2:8300) (DC: dc1)
    2018/05/27 11:45:24 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
    2018/05/27 11:45:24 [INFO] consul: Handled member-join event for server "consul.dc1" in area "wan"
    2018/05/27 11:45:24 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
    2018/05/27 11:45:24 [INFO] agent: started state syncer
    2018/05/27 11:45:30 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2018/05/27 11:45:30 [INFO] raft: Node at 172.17.0.2:8300 [Candidate] entering Candidate state in term 2
    2018/05/27 11:45:30 [INFO] raft: Election won. Tally: 1
    2018/05/27 11:45:30 [INFO] raft: Node at 172.17.0.2:8300 [Leader] entering Leader state
    2018/05/27 11:45:30 [INFO] consul: cluster leadership acquired
    2018/05/27 11:45:30 [INFO] consul: New leader elected: consul
    2018/05/27 11:45:30 [INFO] consul: member 'consul' joined, marking health alive
    2018/05/27 11:45:30 [INFO] agent: Synced node info

And subsequently, the consul server works as expected.

Now, with docker+runv, I run the same docker run ... command as above, but I get the following output in the docker logs and the container exits:

root@hyperdev-02:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                         PORTS               NAMES
bcadbb56da7c        consul              "docker-entrypoint..."   10 seconds ago      Exited (1) 6 seconds ago                           consul
root@hyperdev-02:~# docker logs bcadbb56da7c
bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul server: Failed to start Raft: invalid argument

The process (and container) exit with the mysterious "Failed to start Consul server" error...

Trying it with hyperctl (slightly different command as hyperctl doesn't support the -h host flag):

root@hyperdev-02:~# hyperctl run -d --name consul --publish 8500:8500 consul agent -server -bootstrap -client "0.0.0.0"
Using default tag: latest
latest: Pulling from library/consul
ff3a5c916c92: Pull complete 
4f7ceec429a3: Pull complete 
479d56dd1427: Pull complete 
fe7ac0cf7b6b: Pull complete 
e2db5deeb099: Pull complete 
Digest: sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249
Status: Downloaded newer image for consul:latest
sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249: Pulling from library/consul
Digest: sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249
Status: Downloaded newer image for consul@sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249
POD id is consul
Time to run a POD is 35350 ms
root@hyperdev-02:~# hyperctl list
POD ID              POD Name            VM name             Status
consul              consul              vm-taqgLYUXKv       running
root@hyperdev-02:~# hyperctl logs consul
bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul server: Failed to start Raft: invalid argument

The first thing that I noticed is that when running with runv, the PORTS in the docker ps command are missing... So I have also tried running the container with --net=host which doesn't seem to work with runv for whatever reason... And then I also tried to just --publish all of the ports that are exposed in the consul:latest Dockerfile, but that yielded the same error...

On a separate note, it'd be great to have some sort of a lookup table/doc article with a list of known incompatabilities+workarounds for normal docker containers running on the runv stack... Maybe this exists, but I haven't been able to uncover it. Thanks!

[svarlamov]

I have also tried the exact 'fix' described in the following reddit post for consul on windows -- which is very similar in that it's VM-based -- but I get the same error as before...

https://www.reddit.com/r/docker/comments/89nhkw/i_am_trying_to_run_a_containerized_consul_on/

-p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302/udp -p 8302:8302 -p 8400:8400 -p 8500:8500 -p 53:53/udp