Ability to easily share common logic between flows

[d80tb7]

Once you have more than a single prefect flow, it's likely that you'll end up with some common flow management code that you would want to share between flows. Examples here would include:

• Custom task types (e.g. run AWS Batch job)
• Parameter validation logic
• Common state handlers

At present, one must make sure that this code is present on the docker container running the prefect job, most likely by either copying the code into the image directly, or by extracting the common code into a pip package and pip installing it into the image. Essentially this means that the common code must be treated as a separate library and therefore separate from the flows themselves.

Unfortunately we have found that the above distinction gives the following issues:

• In many cases the flows and the common code are relatively tightly coupled (i.e the common code is really only designed to be used with the flows and vice versa). In this case we want to ensure that the flows and the common code are versioned together. Unfortunately, the of different deployment strategies here means that this is non trivial.
• Actually getting a prefect flow to even use common code is non trivial. You need to ensure that the common code is available for flow development and then also ensure that this is somehow copied into the base container. Once this is done, you then have to reason about the possibility of the common code in your dev env getting out of sync with the common code in the base container (essentially issue 1). This creates quite a high barrier to entry for extracting out common code and therefore encourages people to copy and paste.

The solution here is not straightforward as Prefect allows all flows to be registered independently. Because of this, I raise this ticket more to foster discussion of how this could be made easier, rather than presenting a fully formed solution. One thing I would note, however, is that Prefect appears to be moving more towards a airflow-style deployment model whereby flows live in git and are synced across automatically to the prefect backend. In this case, flows are not deployed independently and so one could imagine that any modules that are defined in the same git repo and that are referenced from the flows could be serialised alongside them and synced to the flow storage location.

[joshmeek]

Thanks for opening the discussion @d80tb7! Let me touch on a couple things you said and others could weigh in as well:

In many cases the flows and the common code are relatively tightly coupled (i.e the common code is really only designed to be used with the flows and vice versa). In this case we want to ensure that the flows and the common code are versioned together.

In the initial supported flow "packaging" method using pickles it was also intended that you would version the code your flow used by pinning dependencies on their environments (containers, local envs, etc.). However we have started to see that this could be a friction so (like you say below this) we are aiming to also adopt the file-like syncing mechanism. Not as a wholesale replacement to the pickle-based storage (because it definitely has its strengths) but as another option to users because Prefect is set up to handle a wide range of options here!

In this case, flows are not deployed independently and so one could imagine that any modules that are defined in the same git repo and that are referenced from the flows could be serialised alongside them and synced to the flow storage location.

Yeah this is definitely a route we are exploring when dealing with flow dependencies, open to suggestions because it isn't fully fleshed out yet. The gist of what we think would be helpful here is if you could "sync" your repo with something like your agent so it could automatically deploy flow code without registration.

We're really looking for some community buy-in and shared decisions to the sync design (as people were with the recent release of file-based storage) 🙂

[d80tb7]

Hi @joshmeek,

Thanks for replying, here are my thoughts:

I think one key thing here is what workflow(s) you want to support for Prefect going forward. It seems to me that you've started out supporting a very dynamic workflow whereby people are free to register any flows they want, any time they want. I can see the advantages here, but I guess that there's a whole set of use cases where this isn't a desirable property. Specifically for any flow that is relied upon by others, you probably want the deployment of this flow to be tightly controlled which means deployment via a CI pipeline and some sort of tight linking between the deployed flow and the source code of the flow in version control. At the moment, you can approximate this by implementing some sort of CI process that re registers all flows on a PR merge, but the experience is less than ideal which is why I assume you are looking towards adding an Airflow-style File syncer as a deployment option.

The reason that this is important is that I think you'll end up with two different deployment mechanisms, which makes designing features like the one I'm proposing something of a challenge. In this example, if you choose to go down the route of having your file sync deployment consider all flows in a given git repo as coupled together (which the git revision naturally gives you) then shared code becomes somewhat simpler. Unfortunately this won't then work for the original pickle and register style deployment as that is always going to have to treat flows as independent.

I guess what I'm saying here is that I guess you'll have to decide how much you are willing to let the file syncing code/functionality diverge from the pickle and load style code/functionality.