Implementing custom "graph tapes" for capturing lineage

[nicholasjng]

Hey all,

I am working on a small project collecting lineage information (that is, task/workflow metadata like names, docstrings, inputs/outputs etc.) for different types of orchestrators and workflow engines.

To make a first test, I wanted to create a PoC using the prefect quickstart, only to realize that it does not have graph building capabilities builtin (fair). But, there is a GraphVizTracker machinery to capture graph relationships between tasks for a flow.

I would like to extend this idea (a graph tape, essentially) to not only capture task names, but also metadata (inputs/outputs, docstrings, etc.) - how could I go about this? I would prefer not to override/subclass prefect's facilities for now, as I consider this only a last resort - do you think this Tracker API could be extended and opened up to users?

Best regards,
Nico