Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confluent kafka downgrade + worker reregistration deduplication #461

Merged
merged 1 commit into from
Nov 6, 2024

Conversation

Theodlz
Copy link
Collaborator

@Theodlz Theodlz commented Nov 6, 2024

This PR:

  • downgrades the "dependabot updated" confluent-kafka (from 2.6.0 back to 2.5.3), as the new version creates authentication issues.
  • Registering/restarting new topic listener now also avoids registering workers if they are already registered on the cluster/scheduler, which created memory leaks when we kept timing out from the kafka topics and registering the workers again and again.
  • I also remove some unnecessary logs from the ZTF workers, that have been polluting the logs for some time now and make it hard to find relevant lines when this kind of error occurs.

…reates authentication issues that cascade into memory leaks. Registering/restarting new topic listener now also avoids registering workers which creates said memory leak
@Theodlz
Copy link
Collaborator Author

Theodlz commented Nov 6, 2024

self merging as this is preventing us from running Kowalski in production!

@Theodlz Theodlz removed the request for review from mcoughlin November 6, 2024 19:31
@Theodlz
Copy link
Collaborator Author

Theodlz commented Nov 6, 2024

will merge once green

@Theodlz Theodlz merged commit 2c52618 into skyportal:main Nov 6, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant