Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Job Manager is not brought back up #709

Closed
live-wire opened this issue Jul 25, 2023 · 1 comment · Fixed by #710
Closed

Job Manager is not brought back up #709

live-wire opened this issue Jul 25, 2023 · 1 comment · Fixed by #710

Comments

@live-wire
Copy link
Contributor

live-wire commented Jul 25, 2023 •
edited
Loading

Noticed this issue with some streaming clusters using mode:Application and restartPolicy: FromSavepointOnFailure

  • To allow for a job restart(recreate), job status is set to Restarting by the operator.

  • In cases when the job's backoff limit is reached the operator can get stuck in this step: 2. Update cluster status because the job state keeps changing in this loop: Restarting -> Deploying -> Failed -> Restarting
    So, the reconciliation of the FlinkCluster never reaches the next step: 3. Compute the desired state

  • It seems like a race condition between the Job and this operator's reconcilers.

Potential fix could be to not have the operator not edit status.job.state manually and only read the actual job state?
@live-wire
Copy link
Contributor Author

This ☝️ could be the reason for: #683 as well

@live-wire live-wire mentioned this issue Jul 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make sure application mode clusters aren't stuck in status updating spotify/flink-on-k8s-operator
1 participant
@live-wire