Slowness when creating / updating objects

[MrTrustor]

This is a follow up to #552.

I'm having an issue where Gramps web is particularly slow for every create / update action (e.g creating an individual, updating a date, etc.). Those actions typically take 10-15s, and I'm not sure where to start to debug this. Any guidance would be appreciated.

Here is a recording of what I'm talking about. Notice that I click the "Add" button at the 19s mark, and the page for the new individual is not fully loaded before 34s.

My setup

Now, I realize that my setup is probably fairly unusual, so I don't necessarily expect a direct answer. But is there a way I could enable some debug logs, or add some logs myself to figure out where the slowness is coming from? I'm not a great developer, but I can figure out Python, and open a PR to add some log statements if someone gives me a point to start with.

Gramps info

Gramps 5.2.2
Gramps Web API 2.4.3 => Following #552, I'm currently running the latest image ghcr.io/gramps-project/grampsweb:latest-devel
Gramps Web Frontend 24.8.0
Gramps QL 0.3.0
Sifts 1.0.0
locale: en
multi-tree: false
task queue: true
OCR: true
chat:

Number of objects

Number of people 57
Number of families 30
Number of events 93
Number of places 39
Number of sources 38
Number of citations 38
Number of repositories 10
Number of notes 25
Number of media 53

Database setup

• PostgreSQL 16.4 for the main database and the user database, running behind pgbouncer
• SQLite for the search index

My PostgreSQL instance runs on my DS418play NAS, with only HDD as storage (no SSD). I use for databases for other apps, none very demanding. It has 2GB of RAM allocated.

Docker setup

I run my own Kubernetes cluster, using k3s, on an Intel NUC 10. Below is an anonymized copy of my Kubernetes deployment for Gramps.

gramps.yaml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app: gramps
  name: gramps
---
apiVersion: v1
data:
  config.cfg: |-
    POSTGRES_USER="gramps"
    POSTGRES_PASSWORD="XXXXXXXXXXXXXXXXXXXXX"
    POSTGRES_HOST="pgbouncer.infra.svc.cluster.local"
    POSTGRES_PORT="5432"
kind: ConfigMap
metadata:
  labels:
    app: gramps
  name: gramps-config-file-t4k97thh86
  namespace: gramps
---
apiVersion: v1
data:
  GRAMPSWEB_CELERY_CONFIG__broker_url: redis://redis.infra.svc.cluster.local:6379/0
  GRAMPSWEB_CELERY_CONFIG__result_backend: redis://redis.infra.svc.cluster.local:6379/0
  GRAMPSWEB_POSTGRES_HOST: pgbouncer.infra.svc.cluster.local
  GRAMPSWEB_POSTGRES_PORT: "5432"
  GRAMPSWEB_POSTGRES_USER: gramps
  GRAMPSWEB_RATELIMIT_STORAGE_URI: redis://redis.infra.svc.cluster.local:6379/1
  GRAMPSWEB_TREE: gramps
kind: ConfigMap
metadata:
  labels:
    app: gramps
  name: gramps-config-t94tg7dgm7
  namespace: gramps
---
apiVersion: v1
data:
  GRAMPSWEB_POSTGRES_PASSWORD:  XXXXXXXXXXXXXXXXXXXXX
  GRAMPSWEB_SEARCH_INDEX_DB_URI: XXXXXXXXXXXXXXXXXXXXX
  GRAMPSWEB_SECRET_KEY: XXXXXXXXXXXXXXXXXXXXX
  GRAMPSWEB_USER_DB_URI: XXXXXXXXXXXXXXXXXXXXX
kind: Secret
metadata:
  labels:
    app: gramps
  name: gramps-secret-fd44dgdgdh
  namespace: gramps
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: gramps
  name: gramps
  namespace: gramps
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 5000
  selector:
    app: gramps
    service: "true"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: gramps
  name: gramps
  namespace: gramps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gramps
      service: "true"
  serviceName: gramps
  template:
    metadata:
      labels:
        app: gramps
        service: "true"
    spec:
      containers:
      - env:
        - name: GRAMPSWEB_TREE
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_TREE
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_POSTGRES_HOST
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_POSTGRES_HOST
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_POSTGRES_PORT
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_POSTGRES_PORT
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_POSTGRES_USER
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_POSTGRES_USER
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: GRAMPSWEB_POSTGRES_PASSWORD
              name: gramps-secret-fd44dgdgdh
        - name: GRAMPSWEB_USER_DB_URI
          valueFrom:
            secretKeyRef:
              key: GRAMPSWEB_USER_DB_URI
              name: gramps-secret-fd44dgdgdh
        - name: GRAMPSWEB_CELERY_CONFIG__broker_url
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_CELERY_CONFIG__broker_url
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_CELERY_CONFIG__result_backend
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_CELERY_CONFIG__result_backend
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_RATELIMIT_STORAGE_URI
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_RATELIMIT_STORAGE_URI
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_SECRET_KEY
          valueFrom:
            secretKeyRef:
              key: GRAMPSWEB_SECRET_KEY
              name: gramps-secret-fd44dgdgdh
        image: ghcr.io/gramps-project/grampsweb:latest-devel
        name: gramps
        ports:
        - containerPort: 5000
          protocol: TCP
        resources:
          limits:
            cpu: 2000m
            memory: 4096Mi
          requests:
            cpu: 250m
            memory: 2048Mi
        volumeMounts:
        - mountPath: /app/config
          name: gramps-config-file
        - mountPath: /tmp
          name: tmp
        - mountPath: /app/cache
          name: cache
        - mountPath: /root/.gramps
          name: gramps-home
        - mountPath: /app/media
          name: media
        - mountPath: /app/secret
          name: gramps-secret
      - args:
        - -A
        - gramps_webapi.celery
        - worker
        - --loglevel=INFO
        command:
        - celery
        env:
        - name: GRAMPSWEB_TREE
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_TREE
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_CELERY_CONFIG__broker_url
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_CELERY_CONFIG__broker_url
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_CELERY_CONFIG__result_backend
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_CELERY_CONFIG__result_backend
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_RATELIMIT_STORAGE_URI
          valueFrom:
            configMapKeyRef:
              key: GRAMPSWEB_RATELIMIT_STORAGE_URI
              name: gramps-config-t94tg7dgm7
        - name: GRAMPSWEB_SECRET_KEY
          valueFrom:
            secretKeyRef:
              key: GRAMPSWEB_SECRET_KEY
              name: gramps-secret-fd44dgdgdh
        - name: GRAMPSWEB_CORS_ORIGINS
          value: https://accounts.google.com
        image: ghcr.io/gramps-project/grampsweb:latest-devel
        name: celery
        resources:
          limits:
            cpu: 2000m
            memory: 4096Mi
          requests:
            cpu: 250m
            memory: 2048Mi
        volumeMounts:
        - mountPath: /app/config
          name: gramps-config-file
        - mountPath: /tmp
          name: tmp
        - mountPath: /app/cache
          name: cache
        - mountPath: /root/.gramps
          name: gramps-home
        - mountPath: /app/media
          name: media
        - mountPath: /app/secret
          name: gramps-secret
      volumes:
      - configMap:
          name: gramps-config-file-t4k97thh86
        name: gramps-config-file
      - name: media
        nfs:
          path: /volume1/docker/gramps/media
          server: 192.168.1.40
      - emptyDir:
          sizeLimit: 500Mi
        name: tmp
      - emptyDir:
          sizeLimit: 500Mi
        name: cache
  volumeClaimTemplates:
  - metadata:
      labels:
        app: gramps
      name: gramps-home
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: longhorn
  - metadata:
      labels:
        app: gramps
      name: gramps-users
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: longhorn
  - metadata:
      labels:
        app: gramps
      name: gramps-index
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: longhorn
  - metadata:
      labels:
        app: gramps
      name: gramps-secret
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Mi
      storageClassName: longhorn
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingress.pomerium.io/policy: |
      allow:
        or:
          - email:
              is: [email protected]
          - email:
              is: [email protected]
  labels:
    app: gramps
  name: gramps
  namespace: gramps
spec:
  ingressClassName: pomerium
  rules:
  - host: gramps.xxxxxxx.net
    http:
      paths:
      - backend:
          service:
            name: gramps
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - gramps.xxxxxxx.net
    secretName: gramps.xxxxxx.net-tls

[DavidMStraub]

Thanks for sharing. I don't see anything wrong at first glance.

Two things to try:

• Reduce the number of gunicorn works https://www.grampsweb.org/install_setup/cpu-limited/
• Increase k8s memory limits and see if it has an effect

To pinpoint what exactly is taking so long, please

• share a screenshot of the network tab in the browser dev console showing the timings for the API calls
• share the log of the Gramps Web container

[MrTrustor]

I reduced the number of gunicorn workers to 2, but that didn't have much effect. I noticed that neither gramps nor celery looked like they used much CPU at all during those 15s, so I investigating my database setup.

After a few oopsies on my part that have nothing to do with gramps (involving data loss and restoration...), I changed a couple of settings of my PostgreSQL database. Specifically:

• wal_level from replica to minimal
• synchronous_commit from on to off

That sped up significantly my setup, and now adding entities to my tree takes less than half the time (5 to 7s). While still not ideal, this is much more usable than previously so for the time being I'm not going to investigate more. I'll note that I have half a dozen other applications using the same PostgreSQL instance as a database, some with much more data than Gramps, and they don't feel particularly slow.

Anyway, thanks a lot for your help! I'm looking forward to the future development of Grampsweb (I don't plan to use the desktop version). I'll see if I can tackle some of the beginner issues in the repo.