Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: eliminate bakeries.yaml - split into database entry and bakery-specific config #47

Open
rabernat opened this issue Jan 31, 2022 · 0 comments

Comments

@rabernat
Copy link
Contributor

rabernat commented Jan 31, 2022

Now that we have the API, we may no longer need bakeries.yaml as our "database" of bakeries.

However, we need to make the distinction between two distinct needs:

  1. Public information about the bakeries that many clients need to know. For example, a recipe meta.yaml needs to route itself to a bakery. It needs to identify the bakery by a unique name. This also includes things like bakery contacts, description, etc.
  2. Private, internal configuration used in registering flows and within the bakery flow runs. For example:
        cluster_options:
       vpc: vpc-01160815e8310bbe0
       cluster_arn: arn:aws:ecs:us-west-2:552819999234:cluster/pangeo-forge-aws-bakery-dask-test-bakeryclusterdasktest4E1A8264-k6Cnpc8EuUpH
       task_role_arn: arn:aws:iam::552819999234:role/pangeo-forge-aws-bakery-d-prefectecstaskroledaskte-XUA2HGW10IJV
       execution_role_arn: arn:aws:iam::552819999234:role/pangeo-forge-aws-bakery-d-prefectecstaskexecutionr-KLL5UEHNBF8Z
       security_groups:
         - sg-0ca6b9d46294e5623
    
    This really doesn't feel like something that the world outside the bakery needs to care about. In addition, bakeries evidently also have environment variables which determine their config, e.g Add CLUSTER_REGION to .env pangeo-forge-gcs-bakery#27. So the config is spread around in multiple places. This seems fragile / hard to debug.

The challenge is that bakeries.yaml is currently storing both types of information. So I propose we separate these two categories. Stuff in category 1 needs to go into the API. Stuff in category 2 should live in a config file that is maintained by the bakery operator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant