Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sensu checks are not executing on scheduled time #5051

Open
Shivani3351 opened this issue Feb 8, 2024 · 1 comment
Open

Sensu checks are not executing on scheduled time #5051

Shivani3351 opened this issue Feb 8, 2024 · 1 comment

Comments

@Shivani3351
Copy link

Shivani3351 commented Feb 8, 2024

Sensu checks are not executing occasionally on scheduled time

Expected Behavior

Checks should never stop being scheduled unless they are no longer published.

Current Behavior

Some sensu checks are not executing occasionally on scheduled time .

Sensu logs :
influxdb handler asset configured
{"component":"schedulerd","cron":"0 */2 * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-elkes-data-backup-validation","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"
/35 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-netty-thread-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-basic-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-connection-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-mdm-ui-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-rdp-api-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-rdp-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-system-user-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 0 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"topology-apm-stats-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}

Possible Solution

Steps to Reproduce (for bugs)

Context

Some sensu checks are not executing occasionally on scheduled time .

influxdb handler asset configured
{"component":"schedulerd","cron":"0 */2 * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-elkes-data-backup-validation","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"
/35 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-netty-thread-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-basic-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-connection-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-mdm-ui-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-rdp-api-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"
/4 * * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-rdp-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-system-user-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 0 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"topology-apm-stats-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}

Your Environment

  • Sensu version used (sensuctl, sensu-backend, and/or sensu-agent):
    sensu-go : 6.10.0
    sensu-server :
    sensu-etcd : v3.5.0
  • Installation method (packages, binaries, docker etc.):
    packages
  • Operating System and version (e.g. Ubuntu 14.04):
    Debian
@Shivani3351 Shivani3351 changed the title Sensu checks are not executing occasionally on scheduled time Sensu checks are not executing on scheduled time Feb 8, 2024
@elfranne
Copy link

elfranne commented Jul 5, 2024

I had a similar issue and I solved it by building a cluster. We have around 19.000 checks spread across 600 hosts. Made a cluster a 3 node backends and a 3 node etcd cluster. No issue since.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants