Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

broken state after redis reset (restart) #562

Open
enapupe opened this issue Dec 12, 2022 · 0 comments
Open

broken state after redis reset (restart) #562

enapupe opened this issue Dec 12, 2022 · 0 comments

Comments

@enapupe
Copy link

enapupe commented Dec 12, 2022

hi there, apologies in advance if this is not appropriate or not related to this project itself.

today we had a minor outage caused by a redis restart: all workers kept running after this disconnect (redis went down for a patch upgrade) but seemingly workers did not registered and stopped processing any entries from the queue ever since the restart. killing the it all and starting workers again got things back on track

is that by design? I'm not very familiar with this project TBH, I found very strange that this reconnection wasn't handled gracefully.
I'm currently on django-rq = "==2.3.1" rq = "==1.2.2"

should I be specifically handling this situation with a custom exception handler? I don't have any descent logs but I assume the worker has successfully reconnected to redis, it just did not process the queue anymore.
thanks in advance

edit: before posting it, I found this rq/rq#1387 would that fix the issue reported above? I'm keeping this issue because I think it might help others facing the same problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant