interplay between executor and environment

[shaunc]

I'm trying to get my head around how the environment and the executor interact. Are they orthogonal somehow?

I would like to execute in dask_gateway in Kubernetes on EKS -- I see the snippet in the advanced tutorials for setting that up. Also, if I'm running the same flow often, and/or have it running on a schedule, I should use the dask kubernetes environment? How do the executor and the environment interact? Are the credentials I use for the dask gateway encrypted by the environment and stored in the storage (if so, using s3 storage I probably should encrypt it and use setup service accounts to access the keys ... another small snarl). If I want one flow to be able to create another flow, can the parent flow task simply create a flow and have security context inherited somehow, or should I pass and store credentials somewhere, and/or setup service accounts?

What is where and a picture and description of how they interact/are copied from place to place, and what service accounts are created and used by what (etc) would help me figure all of this out. Thanks!

Update it would seem that neither DaskKubernetesEnvironment nor DaskCloudProviderEnvironment are appropriate for interacting with dask_gateway as they setup a dask cluster themselves. Hmm... so should I use a local environment? But what about the store and the schedule? I don't want my flows to have any dependency on the developer workstation from which they are submitted.

Update 2 hmm... and what about the agent? Should I be creating a deployment with an agent in it ... using a LocalEnvironment with s3 storage ... say? hmm.. and where/how does the agent learn about new or scheduled flows from?

[joshmeek]

Hi @shaunc we're in the process of smoothing over some of this WRT the environments/executors/agents but for the meantime here are some examples of how the executor is used on correlation with the environment https://docs.prefect.io/orchestration/execution/local_environment.html#examples. In the environment's executor kwarg you can initialize a client for interacting with your dask_gateway.

[shaunc]

Thanks Josh! I think you and jcrist already straightened me out on slack -- thanks for the follow up though. Jeremiah was kind enough to chat with me. I understand that the concepts themselves are in flux to some degree; the documentation perhaps needs more clarity when they come to rest.... :)