Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sporadic "failed to reconcile pravega cluster" messages #193

Open
deenav opened this issue May 21, 2019 · 5 comments
Open

Sporadic "failed to reconcile pravega cluster" messages #193

deenav opened this issue May 21, 2019 · 5 comments
Assignees
@SrishT
Labels
kind/bug Something isn't working priority/P3 No direct impact to applications, e.g., typos in documentation or log messages status/ready The issue is ready to be worked on; or the PR is ready to review

Comments

@deenav
Copy link

deenav commented May 21, 2019

Operator logs shows failed to reconcile pravega cluster (pravega): failed to sync cluster version: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"pravega\": the object has been modified; please apply your changes to the latest version and try again message immediate after upgrade completed.

With Pravega operator : 0.4.0-rc1 , I have tried upgrading Pravega version from 0.5.0-2236.5228e2d to 0.5.0-2236.5228e2d-upgrade and the upgrade completed successfully.
But after upgrade completion, observing below log messages in pravega-operator logs.

time="2019-05-21T11:29:59Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="failed to reconcile pravega cluster (pravega): failed to sync cluster version: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"pravega\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-05-21T11:30:00Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:30:01Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"

Note: This message is observed only once out of 4 retries but filing this issue for tracking.

Complete Pravega-operator logs for the upgrade:

time="2019-05-21T11:04:11Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:30Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:04:30Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:31Z" level=info msg="syncing cluster version from 0.5.0-2236.5228e2d to 0.5.0-2236.5228e2d-upgrade"
time="2019-05-21T11:04:31Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:31Z" level=info msg="updating statefulset (pravega-bookie) template image to 'deenadock/bookkeeper:0.5.0-2236.5228e2d-upgrade'"
time="2019-05-21T11:04:42Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:42Z" level=info msg="statefulset (pravega-bookie) status: 0 updated, 4 ready, 4 target"
time="2019-05-21T11:04:42Z" level=info msg="upgrading pod: pravega-bookie-0"
time="2019-05-21T11:05:12Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:05:12Z" level=info msg="statefulset (pravega-bookie) status: 0 updated, 4 ready, 4 target"
time="2019-05-21T11:05:12Z" level=info msg="upgrading pod: pravega-bookie-0"
time="2019-05-21T11:05:42Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:05:43Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:05:43Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:05:43Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:06:13Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:06:13Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:06:43Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:06:43Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:07:13Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:07:14Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:07:44Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:07:44Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:08:14Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:08:14Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:08:44Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:08:45Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:09:15Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:09:15Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:09:45Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:09:45Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:10:15Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:10:16Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:10:46Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:10:46Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:11:16Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:11:16Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:11:46Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:11:47Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:12:17Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:12:17Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:12:47Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:12:47Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:13:17Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:13:18Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:13:48Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:13:48Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:14:18Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:14:18Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:14:48Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:14:49Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:15:19Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:15:19Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:15:49Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:15:49Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:16:19Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:16:20Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:16:50Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:16:50Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:17:20Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:17:20Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:17:50Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:17:51Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:18:21Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:18:21Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:18:51Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:18:51Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:19:21Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:19:22Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:19:52Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:19:52Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:20:22Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:20:22Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:20:52Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:20:53Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:21:23Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:21:23Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:21:53Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:21:53Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:22:23Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:22:24Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 4 ready, 4 target"
time="2019-05-21T11:22:24Z" level=info msg="upgrading pod: pravega-bookie-1"
time="2019-05-21T11:22:24Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:22:24Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 4 ready, 4 target"
time="2019-05-21T11:22:24Z" level=info msg="upgrading pod: pravega-bookie-1"
time="2019-05-21T11:22:54Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:22:54Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 4 ready, 4 target"
time="2019-05-21T11:22:54Z" level=info msg="upgrading pod: pravega-bookie-1"
time="2019-05-21T11:23:24Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:24Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 3 ready, 4 target"
time="2019-05-21T11:23:24Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:25Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 3 ready, 4 target"
time="2019-05-21T11:23:54Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:55Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 4 ready, 4 target"
time="2019-05-21T11:23:55Z" level=info msg="upgrading pod: pravega-bookie-2"
time="2019-05-21T11:23:55Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:55Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 4 ready, 4 target"
time="2019-05-21T11:23:55Z" level=info msg="upgrading pod: pravega-bookie-2"
time="2019-05-21T11:24:25Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:24:25Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 4 ready, 4 target"
time="2019-05-21T11:24:25Z" level=info msg="upgrading pod: pravega-bookie-2"
time="2019-05-21T11:24:55Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:24:55Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 3 ready, 4 target"
time="2019-05-21T11:24:56Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:24:56Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 3 ready, 4 target"
time="2019-05-21T11:25:26Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:25:26Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 4 ready, 4 target"
time="2019-05-21T11:25:26Z" level=info msg="upgrading pod: pravega-bookie-3"
time="2019-05-21T11:25:26Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:25:26Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 4 ready, 4 target"
time="2019-05-21T11:25:26Z" level=info msg="upgrading pod: pravega-bookie-3"
time="2019-05-21T11:25:56Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:25:56Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 4 ready, 4 target"
time="2019-05-21T11:25:56Z" level=info msg="upgrading pod: pravega-bookie-3"
time="2019-05-21T11:26:26Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:27Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 3 ready, 4 target"
time="2019-05-21T11:26:27Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:27Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 3 ready, 4 target"
time="2019-05-21T11:26:57Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:57Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:26:57Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:26:57Z" level=info msg="updating statefulset (pravega-pravega-segmentstore) template image to 'deenadock/pravega:0.5.0-2236.5228e2d-upgrade'"
time="2019-05-21T11:26:57Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:57Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:26:57Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:26:57Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 0 updated, 3 ready, 3 target"
time="2019-05-21T11:27:27Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:27Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:27Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:27Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:27:27Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:28Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:28Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:28Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:27:57Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:58Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:27:58Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:58Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:28:28Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:28:28Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:28:28Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:28:28Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 2 updated, 2 ready, 3 target"
time="2019-05-21T11:28:58Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:28:58Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:28:58Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:28:58Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 2 ready, 3 target"
time="2019-05-21T11:28:58Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:28:59Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:28:59Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:28:59Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 2 ready, 3 target"
time="2019-05-21T11:29:28Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:29Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:29Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="updating deployment (pravega-pravega-controller) pod template image to 'deenadock/pravega:0.5.0-2236.5228e2d-upgrade'"
time="2019-05-21T11:29:29Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:29Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:29Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="deployment (pravega-pravega-controller) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:29:29Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:30Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:30Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:30Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:30Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:30Z" level=info msg="deployment (pravega-pravega-controller) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:29:59Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:59Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:59Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:59Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:59Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:59Z" level=info msg="deployment (pravega-pravega-controller) status: 2 updated, 2 ready, 2 target"
time="2019-05-21T11:29:59Z" level=info msg="controller version sync has been completed"
time="2019-05-21T11:29:59Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:59Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="failed to reconcile pravega cluster (pravega): failed to sync cluster version: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"pravega\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-05-21T11:30:00Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:30:01Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"
time="2019-05-21T11:30:01Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:30:01Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:30:31Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:31:01Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
@Tristan1900 Tristan1900 added kind/bug Something isn't working priority/P2 Slight inconvenience or annoyance to applications, system continues to function labels May 21, 2019
@Tristan1900
Copy link
Member

This is because an update happens before another update in one reconciling loop so the later update is using an outdated resourceVersion. This is usually a benign error and it can be fixed by entering the next reconciling loop.
I'm tracking down what exactly happened.

@Tristan1900
Copy link
Member

Tristan1900 commented May 23, 2019 •
edited
Loading

After some investigations and experiments, I suppose the following lead to this error.
At the beginning of reconciling loop, the operator sdk uses a GET method to fetch the latest Pravega object and use it for the following updates. According to the doc, the default behavior of GET is to get from a local cache and that cache has only eventual consistency with the apiserver.
So what happens is that there are two UPDATE here and here that are executed almost at the same time. The apiserver gets two updates but the local cache somehow only get synced with the first update at the start of the next reconciling loop, which means the next loop is using a stale Pravega object and leads to this error. I have noticed that the return of GET is updated after a short period of time.
We can reproduce this inconsistency between the result of GET and UPDATE easily by a pseudo code like this:

for i := 0; i < 10; i++{
  UPDATE(i)
  n := GET()
  print(n)
}

As in my last comment, this is a benign error and GET will eventually fetch the latest Pravega object. Do you think we should close it or somehow fix it? @adrianmo Thank you!

@adrianmo adrianmo added area/controller Issue related to the operator controller status/ready The issue is ready to be worked on; or the PR is ready to review labels May 30, 2019
@adrianmo adrianmo changed the title Observed "failed to reconcile pravega cluster (pravega): failed to sync cluster version" message in operator logs after upgrade completed Sporadic "failed to reconcile pravega cluster" messages May 30, 2019
@pbelgundi pbelgundi added the version 0.4.0 Issue with Operator 0.4.0 label Jul 2, 2019
@Tristan1900
Copy link
Member

Tristan1900 commented Jul 17, 2019 •
edited
Loading

This is a benign error, it can be fixed without using cache but that will reduce the performance of the operator and it is suggested to use this cache by the operator framework, see the last line of this. Can we close it or otherwise? Thanks @deenav

@pbelgundi pbelgundi added the area/upgrade Impacts upgrade feature in operator label Jul 25, 2019
@pbelgundi pbelgundi assigned SrishT and unassigned Tristan1900 Jul 25, 2019
@pbelgundi pbelgundi added priority/P3 No direct impact to applications, e.g., typos in documentation or log messages and removed priority/P2 Slight inconvenience or annoyance to applications, system continues to function labels Sep 3, 2019
@pbelgundi
Copy link
Contributor

Downgrading to P3 since there is no functionality loss here.

@pbelgundi
Copy link
Contributor

pbelgundi commented Oct 11, 2019 •
edited
Loading

@SrishT, from the error message and analysis presented by @Tristan1900, this does not seem to be an "upgrade" issue but rather an issue with the k8s client we use and related caching.
Could you please try to see if this error is reproducible without doing an upgrade, by changing the number of replicas for segment store after the cluster is deployed?

@pbelgundi pbelgundi removed version 0.4.0 Issue with Operator 0.4.0 area/controller Issue related to the operator controller area/upgrade Impacts upgrade feature in operator labels May 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SrishT SrishT

Labels
kind/bug Something isn't working priority/P3 No direct impact to applications, e.g., typos in documentation or log messages status/ready The issue is ready to be worked on; or the PR is ready to review
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@adrianmo @SrishT @Tristan1900 @deenav @pbelgundi