Docker Swarm: Postgres doesn't stop initing #5361

makinbacon21 · 2024-10-01T14:45:13Z

makinbacon21
Oct 1, 2024

Steps to reproduce

Set up Docker Swarm Nextcloud AIO manually
Try to fire up stack

Expected behavior

The container should register as Running and show up to other containers on the Docker network.

Actual behavior

The container stays in the Starting state despite initialization completing and the port being open.

Other information

Host OS

Rocky Linux 9

Output of `sudo docker info`

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 57
  Running: 49
  Paused: 0
  Stopped: 8
 Images: 38
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: xw1flp89uz2tuuwn7wxg8zxyr
  Is Manager: true
  ClusterID: 63fiz27ge6qp594g2vcctftpm
  Managers: 1
  Nodes: 3
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: REDACTED
  Manager Addresses:
   REDACTED
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-427.37.1.el9_4.x86_64
 Operating System: Rocky Linux 9.4 (Blue Onyx)
 OSType: linux
 Architecture: x86_64
 CPUs: 10
 Total Memory: 70.17GiB
 Name: gull
 ID: 900eee34-1bca-4b2e-b61d-ffe38e9100c8
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker run command or docker-compose file that you used

services:
  nextcloud-aio-apache:
    cap_drop:
      - NET_RAW
    depends_on:
      - nextcloud-aio-collabora
      - nextcloud-aio-nextcloud
      - nextcloud-aio-notify-push
      - nextcloud-aio-onlyoffice
      - nextcloud-aio-talk
    deploy:
      labels:
        - 'traefik.enable=true'
        - 'traefik.docker.network=traefik'
        - 'traefik.http.routers.nextcloud.entrypoints=https'
        - 'traefik.http.routers.nextcloud.rule=Host(`nextcloud.sccs.swarthmore.edu`)'
        - 'traefik.http.routers.nextcloud.tls=true'
        - 'traefik.http.routers.nextcloud.tls.certresolver=letsEncrypt'
        - 'traefik.http.services.nextcloud.loadbalancer.server.port=11000'

      placement:
        constraints:
          - node.role == manager
    ports:
      - 11000:11000/tcp
      - 11000:11000/udp
    environment:
      APACHE_HOST: nextcloud-aio-apache
      APACHE_MAX_SIZE: 10737418240
      APACHE_MAX_TIME: 3600
      APACHE_PORT: 11000
      COLLABORA_HOST: nextcloud-aio-collabora
      NC_DOMAIN: nextcloud.sccs.swarthmore.edu
      NEXTCLOUD_HOST: nextcloud-aio-nextcloud
      NOTIFY_PUSH_HOST: nextcloud-aio-notify-push
      ONLYOFFICE_HOST: nextcloud-aio-onlyoffice
      TALK_HOST: nextcloud-aio-talk
      TZ: America/New_York
    image: nextcloud/aio-apache:20240925_080419
    init: true
    networks:
      - nextcloud-aio
      - traefik
    restart: unless-stopped
    tmpfs:
      - /var/log/supervisord
      - /var/run/supervisord
      - /usr/local/apache2/logs
      - /tmp
      - /home/www-data
    volumes:
      - nextcloud_aio_nextcloud:/var/www/html:ro
      - nextcloud_aio_apache:/mnt/data

  nextcloud-aio-clamav:
    cap_drop:
      - NET_RAW
    environment:
      CLAMD_STARTUP_TIMEOUT: 90
      MAX_SIZE: 1000G
      TZ: America/New_York
    image: nextcloud/aio-clamav:20240925_080419
    init: false
    networks:
      nextcloud-aio: null
    restart: unless-stopped
    tmpfs:
      - /var/lock
      - /var/log/clamav
      - /tmp
    volumes:
      - nextcloud_aio_clamav:/var/lib/clamav

  nextcloud-aio-collabora:
    cap_add:
      - MKNOD
      - SYS_ADMIN
    cap_drop:
      - NET_RAW
    environment:
      DONT_GEN_SSL_CERT: "1"
      TZ: America/New_York
      aliasgroup1: https://nextcloud.sccs.swarthmore.edu:443
      dictionaries: de_DE en_GB en_US es_ES fr_FR it nl pt_BR pt_PT ru
      extra_params: --o:ssl.enable=false --o:ssl.termination=true --o:mount_jail_tree=false --o:logging.level=warning --o:home_mode.enable=true --o:security.seccomp=true --o:remote_font_config.url=https://nextcloud.sccs.swarthmore.edu/apps/richdocuments/settings/fonts.json
      server_name: nextcloud.sccs.swarthmore.edu
    image: nextcloud/aio-collabora:20240925_080419
    init: true
    networks:
      nextcloud-aio: null
    restart: unless-stopped

  nextcloud-aio-database:
    cap_drop:
      - NET_RAW
    environment:
      PGTZ: America/New_York
      POSTGRES_DB: nextcloud_database
      POSTGRES_PASSWORD: REDACTED
      POSTGRES_USER: nextcloud
      TZ: America/New_York
    image: nextcloud/aio-postgresql:20240925_080419
    init: true
    networks:
      - nextcloud-aio
    restart: unless-stopped
    shm_size: "268435456"
    stop_grace_period: 30m0s
    tmpfs:
      - /var/run/postgresql
    volumes:
      - nextcloud_aio_database:/var/lib/postgresql
      - nextcloud_aio_database_dump:/mnt/data

  nextcloud-aio-fulltextsearch:
    cap_drop:
      - NET_RAW
    environment:
      ES_JAVA_OPTS: -Xms512M -Xmx512M
      FULLTEXTSEARCH_PASSWORD: REDACTED
      TZ: America/New_York
      bootstrap.memory_lock: "true"
      cluster.name: nextcloud-aio
      discovery.type: single-node
      http.port: "9200"
      logger.org.elasticsearch.discovery: WARN
      xpack.license.self_generated.type: basic
      xpack.security.enabled: "false"
    image: nextcloud/aio-fulltextsearch:20240925_080419
    init: false
    networks:
      nextcloud-aio: null
    restart: unless-stopped
    volumes:
      - nextcloud_aio_elasticsearch:/usr/share/elasticsearch/data

  nextcloud-aio-imaginary:
    cap_add:
      - SYS_NICE
    cap_drop:
      - NET_RAW
    environment:
      IMAGINARY_SECRET: REDACTED
      TZ: America/New_York
    image: nextcloud/aio-imaginary:20240925_080419
    init: true
    networks:
      nextcloud-aio: null
    restart: unless-stopped
    tmpfs:
      - /tmp

  nextcloud-aio-nextcloud:
    hostname: nextcloud-aio-nextcloud
    cap_drop:
      - NET_RAW
    depends_on:
      - nextcloud-aio-clamav
      - nextcloud-aio-database
      - nextcloud-aio-fulltextsearch
      - nextcloud-aio-imaginary
      - nextcloud-aio-redis
      - nextcloud-aio-talk-recording
    deploy:
      placement:
        constraints:
          - node.role == manager
      #endpoint_mode: dnsrr
    environment:
      ADDITIONAL_APKS: imagemagick
      ADDITIONAL_PHP_EXTENSIONS: imagick
      ADMIN_PASSWORD: REDACTED
      ADMIN_USER: admin
      APACHE_PORT: "11000"
      CLAMAV_ENABLED: "yes"
      CLAMAV_HOST: nextcloud-aio-clamav
      CLAMAV_MAX_SIZE: "10737418240"
      COLLABORA_ENABLED: "yes"
      COLLABORA_HOST: nextcloud-aio-collabora
      DOCKER_SOCKET_PROXY_ENABLED: "no"
      FULLTEXTSEARCH_ENABLED: "yes"
      FULLTEXTSEARCH_HOST: nextcloud-aio-fulltextsearch
      FULLTEXTSEARCH_PASSWORD: REDACTED
      IMAGINARY_ENABLED: "yes"
      IMAGINARY_HOST: nextcloud-aio-imaginary
      IMAGINARY_SECRET: REDACTED
      INSTALL_LATEST_MAJOR: "no"
      NC_DOMAIN: nextcloud.sccs.swarthmore.edu
      NEXTCLOUD_DATA_DIR: /mnt/ncdata
      #NEXTCLOUD_MOUNT: /mnt/
      ONLYOFFICE_ENABLED: "yes"
      ONLYOFFICE_HOST: nextcloud-aio-onlyoffice
      ONLYOFFICE_SECRET: REDACTED
      OVERWRITEHOST: nextcloud.sccs.swarthmore.edu
      OVERWRITEPROTOCOL: https
      PHP_MAX_TIME: "3600"
      PHP_MEMORY_LIMIT: 2G
      PHP_UPLOAD_LIMIT: 1000G
      POSTGRES_DB: nextcloud_database
      POSTGRES_HOST: nextcloud-aio-database
      POSTGRES_PASSWORD: REDACTED
      POSTGRES_PORT: "5432"
      POSTGRES_USER: nextcloud
      RECORDING_SECRET: REDACTED
      REDIS_HOST: nextcloud-aio-redis
      REDIS_HOST_PASSWORD: REDACTED
      REMOVE_DISABLED_APPS: "yes"
      SIGNALING_SECRET: REDACTED
      STARTUP_APPS: deck twofactor_totp tasks calendar contacts notes
      TALK_ENABLED: "yes"
      TALK_PORT: "3478"
      TALK_RECORDING_ENABLED: "yes"
      TALK_RECORDING_HOST: nextcloud-aio-talk-recording
      #TRUSTED_CACERTS_DIR: /usr/local/share/ca-certificates/my-custom-ca
      TURN_SECRET: REDACTED
      TZ: America/New_York
      UPDATE_NEXTCLOUD_APPS: "yes"
      
    image: nextcloud/aio-nextcloud:20240925_080419
    init: true
    networks:
      - nextcloud-aio
    restart: unless-stopped
    stop_grace_period: 10m0s
    volumes:
      - nextcloud_aio_nextcloud:/var/www/html
      - nextcloud_aio_nextcloud_data:/mnt/ncdata

  nextcloud-aio-notify-push:
    cap_drop:
      - NET_RAW
    environment:
      NC_DOMAIN: nextcloud.sccs.swarthmore.edu
      NEXTCLOUD_HOST: nextcloud-aio-nextcloud
      POSTGRES_DB: nextcloud_database
      POSTGRES_HOST: nextcloud-aio-database
      POSTGRES_PASSWORD: REDACTED
      POSTGRES_PORT: "5432"
      POSTGRES_USER: nextcloud
      REDIS_HOST: nextcloud-aio-redis
      REDIS_HOST_PASSWORD: REDACTED
    image: nextcloud/aio-notify-push:20240925_080419
    init: true
    networks:
      nextcloud-aio: null
    read_only: true
    restart: unless-stopped
    volumes:
      - nextcloud_aio_nextcloud:/nextcloud:ro

  nextcloud-aio-onlyoffice:
    depends_on:
      - nextcloud-aio-database
    cap_drop:
      - NET_RAW
    environment:
      DB_TYPE: postgres
      DB_HOST: nextcloud-aio-database
      DB_PORT: "5432"
      DB_NAME: onlyoffice
      DB_USER: nextcloud
      DB_PWD: REDACTED
      AMQP_URI: amqp://guest:guest@nextcloud-aio-rabbitmq
      JWT_ENABLED: "true"
      JWT_HEADER: AuthorizationJwt
      JWT_SECRET: REDACTED
      TZ: America/New_York
    image: nextcloud/aio-onlyoffice:20240925_080419
    stdin_open: true
    networks:
      - nextcloud-aio
    restart: unless-stopped
    stop_grace_period: 60s
    volumes:
      - nextcloud_aio_onlyoffice:/var/lib/onlyoffice:rw

  # for openoffice, see https://github.com/ONLYOFFICE/Docker-DocumentServer/blob/master/docker-compose.yml
  nextcloud-aio-rabbitmq:
    image: rabbitmq
    restart: unless-stopped
    networks:
      - nextcloud-aio

  nextcloud-aio-redis:
    cap_drop:
      - NET_RAW
    environment:
      REDIS_HOST_PASSWORD: REDACTED
      TZ: America/New_York
    image: nextcloud/aio-redis:20240925_080419
    init: true
    networks:
      nextcloud-aio: null
    read_only: true
    restart: unless-stopped
    volumes:
      - nextcloud_aio_redis:/data

  nextcloud-aio-talk:
    cap_drop:
      - NET_RAW
    environment:
      INTERNAL_SECRET: REDACTED
      NC_DOMAIN: nextcloud.sccs.swarthmore.edu
      SIGNALING_SECRET: REDACTED
      TALK_HOST: nextcloud-aio-talk
      TALK_PORT: "3478"
      TURN_SECRET: REDACTED
      TZ: America/New_York
    image: nextcloud/aio-talk:20240925_080419
    init: true
    networks:
      nextcloud-aio: null
    ports:
      - mode: ingress
        target: 3478
        published: 3478
        protocol: tcp
      - mode: ingress
        target: 3478
        published: 3478
        protocol: udp
    restart: unless-stopped
    tmpfs:
      - /var/log/supervisord
      - /var/run/supervisord
      - /opt/eturnal/run
      - /conf
      - /tmp

  nextcloud-aio-talk-recording:
    cap_drop:
      - NET_RAW
    environment:
      INTERNAL_SECRET: REDACTED
      NC_DOMAIN: nextcloud.sccs.swarthmore.edu
      RECORDING_SECRET: REDACTED
      TZ: America/New_York
    image: nextcloud/aio-talk-recording:20240925_080419
    init: true
    networks:
      nextcloud-aio: null
    restart: unless-stopped
    shm_size: "2147483648"
    tmpfs:
      - /tmp
      - /conf

networks:
  nextcloud-aio:
    name: nextcloud-aio
    external: false
    driver: overlay
  traefik:
    external: true

volumes:
  nextcloud_aio_apache:
    name: nextcloud_aio_apache
    driver_opts:
      device: ":/volumes/nextcloud_aio_apache"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_clamav:
    name: nextcloud_aio_clamav
    driver_opts:
      device: ":/volumes/nextcloud_aio_clamav"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_database:
    name: nextcloud_aio_database
    driver_opts:
      device: ":/volumes/nextcloud_aio_database"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_database_dump:
    name: nextcloud_aio_database_dump
    driver_opts:
      device: ":/volumes/nextcloud_aio_database_dump"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_elasticsearch:
    name: nextcloud_aio_elasticsearch
    driver_opts:
      device: ":/volumes/nextcloud_aio_elasticsearch"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_nextcloud:
    name: nextcloud_aio_nextcloud
    driver_opts:
      device: ":/volumes/nextcloud_aio_nextcloud"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_nextcloud_data:
    name: nextcloud_aio_nextcloud_data
    driver_opts:
      device: ":/volumes/nextcloud_aio_nextcloud_data"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_onlyoffice:
    name: nextcloud_aio_onlyoffice
    driver_opts:
      device: ":/volumes/nextcloud_aio_onlyoffice"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs
  nextcloud_aio_redis:
    name: nextcloud_aio_redis
    driver_opts:
      device: ":/volumes/nextcloud_aio_redis"
      o: "nfsvers=4,addr=REDACTED,rw,nolock,soft"
      type: nfs

Other valuable info

+ pg_ctl stop -m fast
waiting for server to shut down....2024-10-01 10:35:23.750 EDT [14] LOG:  received fast shutdown request
2024-10-01 10:35:23.785 EDT [14] LOG:  aborting any active transactions
2024-10-01 10:35:23.788 EDT [14] LOG:  background worker "logical replication launcher" (PID 66) exited with exit code 1
2024-10-01 10:35:23.788 EDT [61] LOG:  shutting down
2024-10-01 10:35:23.819 EDT [61] LOG:  checkpoint starting: shutdown immediate
.2024-10-01 10:35:25.357 EDT [61] LOG:  checkpoint complete: wrote 762 buffers (4.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.171 s, sync=0.837 s, total=1.570 s; sync files=1108, longest=0.322 s, average=0.001 s; distance=6299 kB, estimate=6299 kB; lsn=0/1F42BD0, redo lsn=0/1F42BD0
2024-10-01 10:35:25.363 EDT [14] LOG:  database system is shut down
 done
server stopped
+ export PGPORT=5432
+ PGPORT=5432
�
x
Setting postgres values...
PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-10-01 10:35:25.612 EDT [117] LOG:  starting PostgreSQL 16.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
2024-10-01 10:35:25.612 EDT [117] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-10-01 10:35:25.612 EDT [117] LOG:  listening on IPv6 address "::", port 5432
2024-10-01 10:35:25.670 EDT [117] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-10-01 10:35:25.707 EDT [127] LOG:  database system was shut down at 2024-10-01 10:35:25 EDT
2024-10-01 10:35:26.453 EDT [117] LOG:  database system is ready to accept connections

Postgres performs migrations/restores, kills the server and restarts it, but the container remains in a "Starting" state and isn't picked up.

nc: getaddrinfo for host "nextcloud-aio-database" port 5432: Name does not resolve
Waiting for database to start...
nc: getaddrinfo for host "nextcloud-aio-database" port 5432: Name does not resolve
Waiting for database to start...
nc: getaddrinfo for host "nextcloud-aio-database" port 5432: Name does not resolve
Waiting for database to start...

Answered by makinbacon21

Oct 1, 2024

Fun fact: it seems to have uh randomly fixed itself. Probably docker garbage or some race condition. Lmao.

View full answer

szaimen · 2024-10-01T17:02:38Z

szaimen
Oct 1, 2024
Maintainer

Hi,

nextcloud-aio-database:
...
networks:
- nextcloud-aio

nextcloud-aio-imaginary:
...
networks:
nextcloud-aio: null

This looks suspicious...

0 replies

szaimen · 2024-10-01T17:14:28Z

szaimen
Oct 1, 2024
Maintainer

nc: getaddrinfo for host "nextcloud-aio-database" port 5432: Name does not resolve
Waiting for database to start.

It looks like the container cannot resolve the database container. So this is a docker networking problem. Please refer for help on this to a docker forum.

8 replies

makinbacon21 Oct 1, 2024
Author

Hence my point. It appears that in instances where the db container start.sh performs recovery/migrations/etc on startup, kills the server, and restarts it, it does not inform Docker it has completed initialization. Very strange behavior.

szaimen Oct 1, 2024
Maintainer

I dont think this is the problem. As you can see, the network connection does not seem to work. Or does docker swarm only connect the network after it thinks that it is running?

makinbacon21 Oct 1, 2024
Author

It's smelling like that. I've had phantom issues before. I will mess around with it. Overlay nets are truly obnoxious. Appreciate your time--I'll keep this updated as I triage.

makinbacon21 Oct 1, 2024
Author

Fun fact: it seems to have uh randomly fixed itself. Probably docker garbage or some race condition. Lmao.

Answer selected by makinbacon21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Swarm: Postgres doesn't stop initing #5361

{{title}}

Replies: 2 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Docker Swarm: Postgres doesn't stop initing #5361

makinbacon21 Oct 1, 2024

Steps to reproduce

Expected behavior

Actual behavior

Other information

Host OS

Output of sudo docker info

Docker run command or docker-compose file that you used

Other valuable info

Replies: 2 comments · 8 replies

szaimen Oct 1, 2024 Maintainer

szaimen Oct 1, 2024 Maintainer

makinbacon21 Oct 1, 2024 Author

szaimen Oct 1, 2024 Maintainer

makinbacon21 Oct 1, 2024 Author

makinbacon21 Oct 1, 2024 Author

makinbacon21
Oct 1, 2024

Output of `sudo docker info`

Replies: 2 comments 8 replies

szaimen
Oct 1, 2024
Maintainer

szaimen
Oct 1, 2024
Maintainer

makinbacon21 Oct 1, 2024
Author

szaimen Oct 1, 2024
Maintainer

makinbacon21 Oct 1, 2024
Author

makinbacon21 Oct 1, 2024
Author