Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend Airflow operator by implementing CeleryKubernetesExecutor #313

Open
adwk67 opened this issue Aug 7, 2023 · 0 comments
Open

Extend Airflow operator by implementing CeleryKubernetesExecutor #313

adwk67 opened this issue Aug 7, 2023 · 0 comments

Comments

@adwk67
Copy link
Member

adwk67 commented Aug 7, 2023

As a user I want to have the option of running my airflow DAGs with the CeleryKubernetesExecutor, so that I have greater control over resource configuration across static workers (jobs queued via Redis) combined with dynamic works.

CeleryKubernetesExecutor

this combines a fixed number of celery workers plus the ability to spin up extra pods if the celery queue is full
this requires the KubernetesExecutor to be configured
The full list is here: airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types

Related to #2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant