etcd slowly increase in size, sessions cleanup missing

[elfranne]

Sensu store sessions inside etcd but does not clean up. This will slowly increase the size of etcd, potentially stopping etcd as there is a hard limit at 2GB (default).

Expected Behavior

Clean up old/expired sessions

Current Behavior

Old sessions are kept under /sensu.io/user_sessions/admin/[uuid] inside etcd.

You can see the number of sessions kept in etcd:
etcdctl [insert authentication and endpoint info] --command-timeout=200s --dial-timeout=200s get "/sensu.io/user_sessions/admin" --prefix=true --keys-only | grep -c -v -e '^\s*$' . After a few month we were getting close to 1.7GB, with nearly 5.000.000 keys in the etcd, and over 4.9m just being sessions.

Possible Solution

Quick fix is to delete all sessions:
etcdctl [insert authentication and endpoint info] --command-timeout=200s --dial-timeout=200s del "/sensu.io/user_sessions/admin" --prefix=true
This reduced etcd from 1.7GB to 392MB.

Proper fix would be to clean expired sessions.

Steps to Reproduce (for bugs)

The scale of the issue would probably hard to reproduce but we can see that the sessions are kept in etcd and keeps increasing.

Your Environment

Sensu 6.10 OSS
Ubuntu 22.04 LTS

[jhenderson-pro]

Hey @elfranne. 👋

Thank you for the well-written issue and testing. I'll have our Engineering team take a look at this.

[elfranne]

not sure it's related but it am now getting new errors on the backend, every 1 to 2 minute:

{"component":"apid.graphql","error":"no claims found in the request context","level":"warning","msg":"couldn't access resource","time":"2024-08-22T10:56:03Z"}

[valsharess]

This is a major bug, is there any update from Engineering? We had 10.5 million user session keys that we had to cleanup, this has brought memory usage down by 9GB and reduced the database from 1.8GB to 20MB.

I don't want to have to check on this every few months as we use the api quite a lot so the sessions start piling up.

[ManishaKumari295]

This is a major bug, is there any update from Engineering? We had 10.5 million user session keys that we had to cleanup, this has brought memory usage down by 9GB and reduced the database from 1.8GB to 20MB.

I don't want to have to check on this every few months as we use the api quite a lot so the sessions start piling up.

Hi , the issue is being worked upon with code fixes . We will update you about the fix once it is integrated and available. Trying our best to make it soon.

[sensu-discourse]

This issue has been mentioned on Sensu Community. There might be relevant details there:

https://discourse.sensu.io/t/official-install-not-working-for-ubuntu-24-04-lts/3246/9

[ManishaKumari295]

PR raised - #5065