Refresh button for source rows in Status page

[lovincyrus]

address #6154

• invalidate resources data when refresh source is clicked
• unable to invalidate individual source resource

[ericpgreen2]

Note: we should also include the refresh button for Models because 1) a Model can be a root node of the DAG, 2) an intermediate model could still have logic that needs a refresh independent from an upstream node, and 3) we're soon deprecating the Source concept anyways.

And actually, if it's not too much a scope expansion, it'd be nice to be able to refresh Alerts & Reports too. See:

[lovincyrus]

Note: we should also include the refresh button for Models because 1) a Model can be a root node of the DAG, 2) an intermediate model could still have logic that needs a refresh independent from an upstream node, and 3) we're soon deprecating the Source concept anyways.

And actually, if it's not too much a scope expansion, it'd be nice to be able to refresh Alerts & Reports too. See:

Added row click refresh button to Source and Model resource kind. I also added a tooltip to indicate that a resource trigger will update all dependent resources.

---

Latest demo:

CleanShot.2024-12-03.at.17.27.54.mp4

[ericpgreen2]

There's a lot of action when clicking the refresh button: flickering, jumpiness, table resizing. Some things I noticed:

1. The Resources table unmounts temporarily. See Jam and screenshot:
   
2. The page gets auto-scrolled to the top (Jam), probably due to point 1.
3. A RefreshTrigger resource is shown temporarily. Can we hide it from the list? Here's a screenshot from your video:
   

[lovincyrus]

Latest demo:

CleanShot.2024-12-04.at.12.26.40.mp4

resolved

[ericpgreen2]

Mostly looks good, but I ran into a weird state: ~50% of the time when refreshing a Source with an error, the source is marked "Idle", yet for about 30s the "Refresh all sources and models" button is disabled and I see polling ListResources network requests. Here's a Jam.

I also hit this 500 error, but I think it's been fixed on main, right? So it'd help if you could merge main into this branch.

[lovincyrus]

Mostly looks good, but I ran into a weird state: ~50% of the time when refreshing a Source with an error, the source is marked "Idle", yet for about 30s the "Refresh all sources and models" button is disabled and I see polling ListResources network requests. Here's a Jam.

I also hit this 500 error, but I think it's been fixed on main, right? So it'd help if you could merge main into this branch.

I can see from the Jam that the specific resource didn't get refreshed properly due to the "failed to ingest source" error. Here's a video of a happy path:

CleanShot.2024-12-10.at.10.32.21.mp4

Is it expected to disable the refresh icon when a resource errors out? I can see in the Jam, the resource continues to error out even after a refresh..

Update:

Gracefully stop polling after detecting reconcile error

CleanShot.2024-12-10.at.14.36.28.mp4

[lovincyrus]

I also hit this 500 error, but I think it's been fixed on main, right? So it'd help if you could merge main into this branch.

After a rebase, I still run into the 500 error occasionally when I fire a $createTrigger call from the individual rows. This doesn't need to block this PR. Would you have more context? @begelundmuller

Error message — "grpc: error while marshaling: marshaling rill.runtime.v1.ListResourcesResponse: size mismatch (see golang/protobuf#1609): calculated=229, measured=177"

[begelundmuller]

After a rebase, I still run into the 500 error occasionally when I fire a $createTrigger call from the individual rows. This doesn't need to block this PR. Would you have more context? @begelundmuller

Yeah that should get fixed by this PR: #6233

[ericpgreen2]

UXQA: I've triggered a Source refresh, yet I don't see any polling of the ListResource API.

Kapture.2024-12-16.at.14.40.05.mp4

[ericpgreen2]

1. onError will be deprecated in TanStack Query v5 (see here)
2. Rather than the imperative startPolling(), stopPolling(), have you considered using the built-in declarative refetchInterval: () => number | false?

[lovincyrus]

UXQA: I've triggered a Source refresh, yet I don't see any polling of the ListResource API.

Kapture.2024-12-16.at.14.40.05.mp4

In this PR, I've used the ListResources API to refresh a resource.

Taking another look at this,

Specifically, the Refresh a model command.

I see runtimev1.RefreshModelTrigger and runtimev1.CreateTriggerRequest from the refresh.go. Any reason why we don't have RefreshModelTrigger runtime service? @ericpgreen2

[begelundmuller]

@lovincyrus @ericpgreen2 FYI RefreshModelTrigger is used in CreateTriggerRequest.models (link) to enable fine-grained refreshes of models (incremental/full refreshes, partition refreshes, retries of errored partitions, etc.). You can still use specify a model in CreateTriggerRequest.resources (link), but then it will always be a "normal" model refresh (which is a full refresh for non-incremental models and an incremental refresh for incremental models).