Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ui] Always launch using pure asset backfills except on asset job pages #23439

Closed
wants to merge 1 commit into from

Conversation

bengotow
Copy link
Collaborator

@bengotow bengotow commented Aug 6, 2024

Related: https://linear.app/dagster-labs/issue/FE-499/launch-all-non-asset-job-runs-using-pure-asset-backfill-path

Summary & Motivation

We are switching from many hidden asset jobs to just one which does not have a predefined partition set (this has already merged). We are currently launching assets incorrectly because the new singleton-hidden-asset-job does not have a pipeline partition set definition, which prevents the UI from launching single runs of partitioned assets.

This PR explores launching ALL non-asset-job runs for assets as pure-asset backfills endpoint (the launchBackfill mutation with an assetSelection.

This is much simpler, and it does work (though you always get a backfill currently), BUT:

  • There is no way to provide config (previously shift-clicking would give you the launchpad)
  • There is no way to indicate whether to include checks (previously a checkbox inside the launchpad)
  • There is no way to launch a backfill using a single run and a partition range specified in tags
  • There is always a backfill created (though this seems fixable)

Curious to get thoughts on this direction @sryza

It seems like we could use the new singleton-hidden-asset-job for unpartitioned assets with config, but I can't think of a way to allow execution of a single partition of a partitioned asset with config.

Copy link

github-actions bot commented Aug 6, 2024

Deploy preview for dagit-core-storybook ready!

✅ Preview
https://dagit-core-storybook-n7zgs5v7n-elementl.vercel.app
https://bengotow-2024-08-FE-499.core-storybook.dagster-docs.io

Built with commit fb9c835.
This pull request is being automatically deployed with vercel-action

@sryza
Copy link
Contributor

sryza commented Aug 6, 2024

There is no way to provide config (previously shift-clicking would give you the launchpad)
There is no way to indicate whether to include checks (previously a checkbox inside the launchpad)

Would it make sense to still honor the launchpad if they force it?

There is no way to launch a backfill using a single run and a partition range specified in tags

This seems fine. We should be deciding whether to launch a single run or multiple runs based on the BackfillPolicys, not based on anything set in the frontend.

There is always a backfill created (though this seems fixable)

Yeah we can fix this on the backend if we want to.

@bengotow bengotow closed this Aug 14, 2024
@bengotow bengotow deleted the bengotow-2024-08/FE-499 branch November 21, 2024 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants