You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we hide both MV and their state tables during backfilling. However, I think there's no need to hide the state tables:
Hiding state tables don't have large benefits, compared with hiding the MV (e.g., for consistency), since state table is an internal thing.
For backfill executors' state tables, they are only meaningful during the backfill stage. Their content can also be viewed as an observability tool for backfilling. So it's beneficial to expose them.
Since #17503, we've already broadcast all table catalogs (including both internal tables and the MV) to the frontend immediately when meta starts the creating procedure. You can verify this by SHOW INTERNAL TABLES.
Selecting from internal tables is intentionally disabled, by only resolving the "created" table during binding.
However, as described in #18944, the catalogs received by the frontend during creation are incomplete, with fields like fragment_id or vnode_count not correctly filled. Performing batch scan during this period may lead to problem.
Perhaps this is the real motivation for the refactor of only notifying the complete catalogs once to the frontends, which is the original idea of #18944:
However, as described in #18944, the catalogs received by the frontend during creation are incomplete, with fields like fragment_id or vnode_count not correctly filled. Performing batch scan during this period may lead to problem.
Perhaps this is the real motivation for the refactor of only notifying the complete catalogs once to the frontends, which is the original idea of #18944:
I suppose your proposal is to notify the catalogs to frontend, once the TableFragments are built, since only at that time the fragment_id and vnode_count will be correct. Sounds reasonable to me. Wdyt @yezizp2012.
This can allow us to expose the internal state table of backfill for querying.
Currently we hide both MV and their state tables during backfilling. However, I think there's no need to hide the state tables:
related:
SHOW JOBS
(another observability tool) feat: improve progress msg in SHOW JOBS for source backfill #18925 (comment)The text was updated successfully, but these errors were encountered: