Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot deploy on on-premise k8s #386

Open
pckroon opened this issue Oct 14, 2022 · 3 comments
Open

Cannot deploy on on-premise k8s #386

pckroon opened this issue Oct 14, 2022 · 3 comments

Comments

@pckroon
Copy link

pckroon commented Oct 14, 2022 •
edited
Loading

Hi all!

I'm running into issues installing the helm chart on my on-prem/local kubernetes cluster. I'll first dump all relevant manifests, and then describe the issue I'm running into.

Helm values 
persistence:
  enabled: true
  existingClaim: galaxy-data
metrics:
  enabled: false
s3csi:
  deploy: false
cvmfs:
  deploy: false
setupJob:
  createDatabase: true
  ttlSecondsAfterFinished: 3600
ingress:
  enabled: true
serviceAccount:
  create: true
postgresql:
  enabled: true
    #galaxyDatabasePassword: "potato"
    #postgresqlPassword: "potato"
    #postgresqlPostgresPassword: "potato"
  persistence:
    enabled: true
    storageClass: openebs-jiva-csi-sc
    size: 5Gi
refdata:
  enabled: false
influxdb:
  enabled: false
galaxy-data PVC definition 
kind: Namespace
apiVersion: v1
metadata:
  name: galaxy
  labels:
    kubernetes.io/metadata.name: galaxy
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: galaxy-data-nfs
  namespace: galaxy
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: openebs-jiva-csi-sc
  resources:
    requests:
      storage: 50Gi
---
kind: Service
apiVersion: v1
metadata:
  name: nfs-server
  namespace: galaxy
spec:
  clusterIP: 10.0.0.13
  ports:
    - name: nfs
      port: 2049
    - name: mountd
      port: 20048
    - name: rpcbind
      port: 111
  selector:
    role: nfs-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-server
  namespace: galaxy
spec:
  replicas: 1
  selector:
    matchLabels:
      role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      securityContext:
        fsGroup: 101
      containers:
      - name: nfs-server
        image: registry.k8s.io/volume-nfs:0.8
        ports:
          - name: nfs
            containerPort: 2049
          - name: mountd
            containerPort: 20048
          - name: rpcbind
            containerPort: 111
        securityContext:
          privileged: true
        volumeMounts:
          - mountPath: /exports
            name: mypvc
      volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: galaxy-data-nfs
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: galaxy-data
  namespace: galaxy
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 50Gi
  nfs:
    #server: nfs-server.galaxy.svc.cluster.local
    server: 10.0.0.13
    path: "/"
  mountOptions:
    - nfsvers=4.2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: galaxy-data
  namespace: galaxy
spec:
  accessModes:
    - ReadWriteMany
  volumeName: galaxy-data
  resources:
    requests:
      storage: 50Gi

I use the NFS backed PVC since my "standard" sc (jiva) doesn't support RWX.

Running helm install -n galaxy my-galaxy -f galaxy-values.yaml galaxy-helm/galaxy/ results in

manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: my-galaxy
LAST DEPLOYED: Fri Oct 14 16:59:25 2022
NAMESPACE: galaxy
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  http:///galaxy

Looks reasonable to me. However, the job my-galaxy-init-db fails in container galaxy-init-postgres, related to permission denied to chown 101:101 /galaxy/server/database. Easy enough to fix: wipe the namespace, recreate the NFS share, create that folder with correct ownership in the NFS share, and reinstall the helm chart.

Next issue has me stumped though kubectl logs -n galaxy jobs/my-galaxy-init-db-hb4hl -c galaxy-init-postgres:

nc: bad address 'galaxy-my-galaxy-galaxy-postgres'
waiting for galaxy-postgres service
(repeating)

And kubectl logs -n galaxy jobs/my-galaxy-init-db-hb4hl has a python traceback resulting in sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "galaxydbuser". On the NFS share, the folder server/database remains empty.
The postgres service pod is running:

NAME                                            READY   STATUS      RESTARTS   AGE
pod/galaxy-my-galaxy-galaxy-postgres-0          1/1     Running     0          17m
pod/my-galaxy-galaxy-postgres-7c897bf6c-dd7tb   1/1     Running     0          17m
pod/my-galaxy-init-mounts-monlp-mvwsp           0/4     Completed   0          17m
pod/my-galaxy-job-0-7459f74d56-c6fmb            0/1     Init:0/1    0          17m
pod/my-galaxy-nginx-8474bd9bd-qf58r             1/1     Running     0          17m
pod/my-galaxy-web-dfbddc76b-f6qf7               0/1     Init:0/1    0          17m
pod/my-galaxy-workflow-77fd776ccd-jqtfq         0/1     Init:0/1    0          17m
pod/nfs-server-69cd8d9cb9-tbjs6                 1/1     Running     0          20m

NAME                                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/galaxy-my-galaxy-galaxy-postgres          ClusterIP   10.0.49.231    <none>        5432/TCP                     17m
service/galaxy-my-galaxy-galaxy-postgres-config   ClusterIP   None           <none>        <none>                       16m
service/galaxy-my-galaxy-galaxy-postgres-repl     ClusterIP   10.0.209.18    <none>        5432/TCP                     17m
service/my-galaxy-galaxy-postgres                 ClusterIP   10.0.201.184   <none>        8080/TCP                     17m
service/my-galaxy-nginx                           ClusterIP   10.0.108.130   <none>        8000/TCP                     17m
service/my-galaxy-uwsgi                           ClusterIP   10.0.129.244   <none>        4001/TCP                     17m
service/nfs-server                                ClusterIP   10.0.0.13      <none>        2049/TCP,20048/TCP,111/TCP   20m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-galaxy-galaxy-postgres   1/1     1            1           17m
deployment.apps/my-galaxy-job-0             0/1     1            0           17m
deployment.apps/my-galaxy-nginx             1/1     1            1           17m
deployment.apps/my-galaxy-web               0/1     1            0           17m
deployment.apps/my-galaxy-workflow          0/1     1            0           17m
deployment.apps/nfs-server                  1/1     1            1           20m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/my-galaxy-galaxy-postgres-7c897bf6c   1         1         1       17m
replicaset.apps/my-galaxy-job-0-7459f74d56            1         1         0       17m
replicaset.apps/my-galaxy-nginx-8474bd9bd             1         1         1       17m
replicaset.apps/my-galaxy-web-dfbddc76b               1         1         0       17m
replicaset.apps/my-galaxy-workflow-77fd776ccd         1         1         0       17m
replicaset.apps/nfs-server-69cd8d9cb9                 1         1         1       20m

NAME                                                READY   AGE
statefulset.apps/galaxy-my-galaxy-galaxy-postgres   1/1     17m

NAME                                  SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/my-galaxy-maintenance   0 2 * * *   False     0        <none>          17m

NAME                                    COMPLETIONS   DURATION   AGE
job.batch/my-galaxy-init-db-hb4hl       0/1           17m        17m
job.batch/my-galaxy-init-mounts-monlp   1/1           5m25s      17m

NAME                                                            IMAGE                                               CLUSTER-LABEL   SERVICE-ACCOUNT   MIN-INSTANCES   AGE
operatorconfiguration.acid.zalan.do/my-galaxy-galaxy-postgres   registry.opensource.zalan.do/acid/spilo-13:2.1-p1   cluster-name    postgres-pod      -1              17m

NAME                                                        TEAM     VERSION   PODS   VOLUME   CPU-REQUEST   MEMORY-REQUEST   AGE   STATUS
postgresql.acid.zalan.do/galaxy-my-galaxy-galaxy-postgres   galaxy   13        1      5Gi                                     17m   SyncFailed
kubectl logs -n galaxy pods/galaxy-my-galaxy-galaxy-postgres-0 
2022-10-14 15:00:01,209 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2022-10-14 15:00:03,219 - bootstrapping - INFO - Could not connect to 169.254.169.254, assuming local Docker setup
2022-10-14 15:00:03,222 - bootstrapping - INFO - No meta-data available for this provider
2022-10-14 15:00:03,223 - bootstrapping - INFO - Looks like your running local
2022-10-14 15:00:03,320 - bootstrapping - INFO - Configuring pgbouncer
2022-10-14 15:00:03,320 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring log
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring pgqd
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring bootstrap
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring pam-oauth2
2022-10-14 15:00:03,323 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2022-10-14 15:00:03,323 - bootstrapping - INFO - Configuring crontab
2022-10-14 15:00:03,324 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2022-10-14 15:00:03,325 - bootstrapping - INFO - Configuring patroni
2022-10-14 15:00:03,362 - bootstrapping - INFO - Writing to file /run/postgres.yml
2022-10-14 15:00:03,365 - bootstrapping - INFO - Configuring certificate
2022-10-14 15:00:03,365 - bootstrapping - INFO - Generating ssl certificate
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring wal-e
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring standby-cluster
2022-10-14 15:00:05,136 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'.
2022-10-14 15:00:05,162 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-10-14 15:00:05,167 INFO: Lock owner: None; I am galaxy-my-galaxy-galaxy-postgres-0
2022-10-14 15:00:05,349 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgdata/pgroot/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/13/bin/pg_ctl -D /home/postgres/pgdata/pgroot/data -l logfile start

2022-10-14 15:00:08 UTC [79]: [1-1] 63497978.4f 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-10-14 15:00:08 UTC [79]: [2-1] 63497978.4f 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-10-14 15:00:08,662 INFO: postmaster pid=79
/var/run/postgresql:5432 - no response
2022-10-14 15:00:08 UTC [79]: [3-1] 63497978.4f 0     LOG:  redirecting log output to logging collector process
2022-10-14 15:00:08 UTC [79]: [4-1] 63497978.4f 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2022-10-14 15:00:09,709 INFO: establishing a new patroni connection to the postgres cluster
2022-10-14 15:00:09,772 INFO: running post_bootstrap
DO
DO
DO
CREATE EXTENSION
NOTICE:  version "1.1" of extension "pg_auth_mon" is already installed
ALTER EXTENSION
GRANT
CREATE EXTENSION
NOTICE:  version "1.3" of extension "pg_cron" is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
GRANT
CREATE EXTENSION
DO
CREATE TABLE
GRANT
ALTER TABLE
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE:  schema "zmon_utils" does not exist, skipping
DROP SCHEMA
DO
NOTICE:  language "plpythonu" does not exist, skipping
DROP LANGUAGE
NOTICE:  function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
GRANT
You are now connected to database "postgres" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
You are now connected to database "template1" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
2022-10-14 15:00:10,961 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2022-10-14 15:00:11,177 INFO: initialized a new cluster
2022-10-14 15:00:11,233 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,742 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_idle from 0 to 900
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_interval from 0 to 100
2022-10-14 15:00:11,767 INFO: Reloading PostgreSQL configuration.
server signaled
2022-10-14 15:00:12,888 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:23,334 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:33,385 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:43,384 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:53,329 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:03,387 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:04.675 35 LOG Starting pgqd 3.3
2022-10-14 15:01:04.676 35 LOG auto-detecting dbs ...
2022-10-14 15:00:01,209 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2022-10-14 15:00:03,219 - bootstrapping - INFO - Could not connect to 169.254.169.254, assuming local Docker setup
2022-10-14 15:00:03,222 - bootstrapping - INFO - No meta-data available for this provider
2022-10-14 15:00:03,223 - bootstrapping - INFO - Looks like your running local
2022-10-14 15:00:03,320 - bootstrapping - INFO - Configuring pgbouncer
2022-10-14 15:00:03,320 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring log
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring pgqd
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring bootstrap
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring pam-oauth2
2022-10-14 15:00:03,323 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2022-10-14 15:00:03,323 - bootstrapping - INFO - Configuring crontab
2022-10-14 15:00:03,324 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2022-10-14 15:00:03,325 - bootstrapping - INFO - Configuring patroni
2022-10-14 15:00:03,362 - bootstrapping - INFO - Writing to file /run/postgres.yml
2022-10-14 15:00:03,365 - bootstrapping - INFO - Configuring certificate
2022-10-14 15:00:03,365 - bootstrapping - INFO - Generating ssl certificate
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring wal-e
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring standby-cluster
2022-10-14 15:00:05,136 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'.
2022-10-14 15:00:05,162 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-10-14 15:00:05,167 INFO: Lock owner: None; I am galaxy-my-galaxy-galaxy-postgres-0
2022-10-14 15:00:05,349 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgdata/pgroot/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/13/bin/pg_ctl -D /home/postgres/pgdata/pgroot/data -l logfile start

2022-10-14 15:00:08 UTC [79]: [1-1] 63497978.4f 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-10-14 15:00:08 UTC [79]: [2-1] 63497978.4f 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-10-14 15:00:08,662 INFO: postmaster pid=79
/var/run/postgresql:5432 - no response
2022-10-14 15:00:08 UTC [79]: [3-1] 63497978.4f 0     LOG:  redirecting log output to logging collector process
2022-10-14 15:00:08 UTC [79]: [4-1] 63497978.4f 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2022-10-14 15:00:09,709 INFO: establishing a new patroni connection to the postgres cluster
2022-10-14 15:00:09,772 INFO: running post_bootstrap
DO
DO
DO
CREATE EXTENSION
NOTICE:  version "1.1" of extension "pg_auth_mon" is already installed
ALTER EXTENSION
GRANT
CREATE EXTENSION
NOTICE:  version "1.3" of extension "pg_cron" is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
GRANT
CREATE EXTENSION
DO
CREATE TABLE
GRANT
ALTER TABLE
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE:  schema "zmon_utils" does not exist, skipping
DROP SCHEMA
DO
NOTICE:  language "plpythonu" does not exist, skipping
DROP LANGUAGE
NOTICE:  function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
GRANT
You are now connected to database "postgres" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
You are now connected to database "template1" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
2022-10-14 15:00:10,961 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2022-10-14 15:00:11,177 INFO: initialized a new cluster
2022-10-14 15:00:11,233 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,742 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_idle from 0 to 900
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_interval from 0 to 100
2022-10-14 15:00:11,767 INFO: Reloading PostgreSQL configuration.
server signaled
2022-10-14 15:00:12,888 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:23,334 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:33,385 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:43,384 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:53,329 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:03,387 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:04.675 35 LOG Starting pgqd 3.3
2022-10-14 15:01:04.676 35 LOG auto-detecting dbs ...
kubectl logs -n galaxy pods/my-galaxy-galaxy-postgres-7c897bf6c-dd7tb 
time="2022-10-14T14:59:34Z" level=info msg="Spilo operator v1.7.0\n"
time="2022-10-14T14:59:34Z" level=error msg="could not create customResourceDefinition \"operatorconfigurations.acid.zalan.do\": CustomResourceDefinition.apiextensions.k8s.io \"operatorconfigurations.acid.zalan.do\" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="Parse role bindings" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="successfully parsed" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="Listening to all namespaces" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="{" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReadyWaitInterval\": 3000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReadyWaitTimeout\": 30000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ResyncPeriod\": 1800000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"RepairPeriod\": 300000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableCRDValidation\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ResourceCheckInterval\": 3000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ResourceCheckTimeout\": 600000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodLabelWaitTimeout\": 600000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodDeletionWaitTimeout\": 600000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloRunAsUser\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloRunAsGroup\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloFSGroup\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodPriorityClassName\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterDomain\": \"cluster.local\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloPrivileged\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloAllowPrivilegeEscalation\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AdditionalPodCapabilities\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterLabels\": {" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"application\": \"spilo\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   }," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InheritedLabels\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InheritedAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DownscalerAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterNameLabel\": \"cluster-name\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DeleteAnnotationDateKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DeleteAnnotationNameKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodRoleLabel\": \"spilo-role\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodToleration\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultCPURequest\": \"100m\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultCPULimit\": \"1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultMemoryLimit\": \"500Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinCPULimit\": \"250m\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinMemoryLimit\": \"250Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodEnvironmentConfigMap\": \"/\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodEnvironmentSecret\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"NodeReadinessLabel\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MaxInstances\": -1," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinInstances\": -1," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ShmVolume\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SecretNameTemplate\": \"{username}.{cluster}.credentials.{tprkind}.{tprgroup}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PamRoleName\": \"zalandos\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TeamsAPIUrl\": \"https://teams.example.com/api/\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"OAuthTokenSecretName\": \"galaxy/my-galaxy-galaxy-postgres\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InfrastructureRolesSecretName\": \"/\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InfrastructureRoles\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InfrastructureRolesDefs\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SuperUsername\": \"postgres\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReplicationUsername\": \"standby\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrAPIKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrImage\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrServerURL\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrCPURequest\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrMemoryRequest\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrCPULimit\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrMemoryLimit\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupSchedule\": \"30 00 * * *\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup:v1.7.0\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupProvider\": \"s3\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3Bucket\": \"my-bucket-url\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3Region\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3Endpoint\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3AccessKeyID\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3SecretAccessKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3SSE\": \"AES256\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupGoogleApplicationCredentials\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupJobPrefix\": \"logical-backup-\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"NumberOfInstances\": 2," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Schema\": \"pooler\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"User\": \"pooler\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-18\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Mode\": \"transaction\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MaxDBConnections\": 60," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultCPULimit\": \"1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WatchedNamespace\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"KubernetesUseConfigMaps\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EtcdHost\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-13:2.1-p1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SidecarImages\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SidecarContainers\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodServiceAccountName\": \"postgres-pod\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodServiceAccountDefinition\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"ServiceAccount\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"}}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodServiceAccountRoleBindingDefinition\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"RoleBinding\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"ClusterRole\\\",\\\"name\\\":\\\"postgres-pod\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"postgres-pod\\\"}]}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MasterPodMoveTimeout\": 1200000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DbHostedZone\": \"db.example.com\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AWSRegion\": \"eu-central-1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WALES3Bucket\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogS3Bucket\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"KubeIAMRole\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WALGSBucket\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"GCPCredentials\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WALAZStorageAccount\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AdditionalSecretMount\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AdditionalSecretMountPath\": \"/meta/credentials\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableEBSGp3Migration\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableEBSGp3MigrationMaxSize\": 1000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DebugLogging\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableDBAccess\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableTeamsAPI\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableTeamSuperuser\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TeamAdminRole\": \"admin\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"RoleDeletionSuffix\": \"_deleted\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableTeamMemberDeprecation\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableAdminRoleForUsers\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePostgresTeamCRD\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePostgresTeamCRDSuperusers\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableMasterLoadBalancer\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableReplicaLoadBalancer\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"CustomServiceAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"CustomPodAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePodAntiAffinity\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"StorageResizeMode\": \"pvc\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableLoadBalancer\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ExternalTrafficPolicy\": \"Cluster\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PDBNameFormat\": \"postgres-{cluster}-pdb\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePodDisruptionBudget\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableInitContainers\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableSidecars\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Workers\": 8," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"APIPort\": 8080," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"RingLogLines\": 100," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterHistoryEntries\": 1000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TeamAPIRoleConfiguration\": {" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"log_statement\": \"all\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   }," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodTerminateGracePeriod\": 300000000000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodManagementPolicy\": \"ordered_ready\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ProtectedRoles\": [" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"admin\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   ]," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PostgresSuperuserTeams\": [" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"postgres_superusers\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   ]," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SetMemoryRequestToLimit\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableLazySpiloUpgrade\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableCrossNamespaceSecret\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePgVersionEnvVar\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableSpiloWalPathCompat\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MajorVersionUpgradeMode\": \"off\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinimalMajorVersion\": \"9.5\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TargetMajorVersion\": \"13\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="}" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="acquiring initial list of clusters" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="added new cluster: \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="SYNC event has been queued" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T14:59:42Z" level=info msg="there are 1 clusters running" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="started working in background" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="listening on :8080" pkg=apiserver
time="2022-10-14T14:59:42Z" level=info msg="ADD event has been queued" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T14:59:42Z" level=info msg="creating pod service account \"postgres-pod\" in the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="successfully deployed the pod service account \"postgres-pod\" to the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="new node has been added: /gondor2013.bin.bioinf.nl ()" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="new node has been added: /rohan2013 ()" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="Creating the role binding \"postgres-pod\" in the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="successfully deployed the role binding for the pod service account \"postgres-pod\" to the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="syncing of the cluster started" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T14:59:42Z" level=debug msg="team API is disabled" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="team API is disabled" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="syncing secrets" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="secret galaxy/galaxydbuser.galaxy-my-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="secret galaxy/postgres.galaxy-my-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="created new secret galaxy/standby.galaxy-my-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do, namespace: galaxy, uid: df0ac207-f80e-45e5-8802-b98462095cf3" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="syncing master service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="could not find the cluster's master endpoint" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=warning msg="master is not running, generated master endpoint does not contain any addresses" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="created missing master endpoint \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="could not find the cluster's master service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="created missing master service \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=debug msg="syncing replica service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="could not find the cluster's replica endpoint" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="created missing replica endpoint \"galaxy/galaxy-my-galaxy-galaxy-postgres-repl\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="could not find the cluster's replica service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=debug msg="No load balancer created for the replica service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=info msg="created missing replica service \"galaxy/galaxy-my-galaxy-galaxy-postgres-repl\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=info msg="volume claims do not require changes" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=debug msg="syncing statefulsets" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=info msg="cluster's statefulset does not exist" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=debug msg="created new statefulset \"galaxy/galaxy-my-galaxy-galaxy-postgres\", uid: \"eb26faf0-1c3f-44bb-8af7-dbe2406a9368\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:47Z" level=debug msg="Waiting for 1 pods to become ready" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=info msg="created missing statefulset \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="making GET http request: http://10.240.206.182:8008/config" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="patching Postgres config via Patroni API on pod galaxy/galaxy-my-galaxy-galaxy-postgres-0 with following options: {\"synchronous_mode\":false,\"synchronous_mode_strict\":false}" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="making PATCH http request: http://10.240.206.182:8008/config" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="restarting Postgres server within pods" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="making GET http request: http://10.240.206.182:8008/patroni" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="Postgres server successfuly restarted in master pod galaxy/galaxy-my-galaxy-galaxy-postgres-0" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="syncing pod disruption budgets" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=info msg="could not find the cluster's pod disruption budget" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=warning msg="error while syncing cluster state: could not sync pod disruption budget: could not create pod disruption budget: the server could not find the requested resource" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=error msg="could not sync cluster: could not sync pod disruption budget: could not create pod disruption budget: the server could not find the requested resource" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T15:00:11Z" level=info msg="recieved add event for already existing Postgres cluster" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0

Any help would be much appreciated :)
@nuwang
Copy link
Member

nuwang commented Oct 15, 2022

Are you using k8s 1.25 by chance? Looks like there's an issue with it: #385
ping @afgane

@pckroon
Copy link
Author

pckroon commented Oct 18, 2022

Thanks for the quick reply. I'm indeed running k8s 1.25.1:

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:49:27Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:42:30Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}

And in case it matters, the helm version:

version.BuildInfo{Version:"v3.10.0", GitCommit:"ce66412a723e4d89555dc67217607c6579ffcb21", GitTreeState:"clean", GoVersion:"go1.18.6"}

It looks like it's exactly the issue described in #385.
Do you have any idea about the release cadence of the pgsql operator? Does it make sense to report this upstream (again)?

@nuwang
Copy link
Member

nuwang commented Oct 18, 2022

There doesn't appear to be a fixed release cadence, but support for 1.25 appears to have been added: zalando/postgres-operator#1999

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nuwang @pckroon