file-based workflows

[SultanOrazbayev]

My workflow is built around files that are passed between various scripts/notebooks. The examples on the website and in the coiled stream mentioned file-based caching, but it would be great to see an explicit example of a file-based workflow. Here's an example workflow in snakemake format:

# using dictionary expansion

mydictionary={
'apple': 'crunchy fruit',
'banana': 'mushy and yellow'
}

rule all:
    input:
        expand('{key}.txt', key=mydictionary.keys())

rule test:
    output: temp('{f}.txt')
    params:
        keyval=lambda wildcards: mydictionary[wildcards.f]
    shell:
        """
        echo {params.keyval} > {output}
        cat {output}
        """

[joshmeek]

Hi @SultanOrazbayev could you clarify more what you are after? Prefect has a notion of Results and Targets which you could take advantage of to write/read files in between tasks https://docs.prefect.io/core/concepts/results.html

[SultanOrazbayev]

Thank you for the response and the pointer to Results/Targets! With snakemake it's possible to have the workflow defined in a separate file that acts as a "glue" holding other code, and this is what I'm trying to achieve with prefect (to avoid having one large file/notebook for a complex analysis).