Feature request: option for running complex workflow without fireworks

[JaGeo]

I very much like that one can start jobs without setting up a MongoDB database with the help of jobflow. However, for more complex workflows (e.g., the phonon workflow in atomate2), the submission with 1 job script can be very inefficient (e.g., if one computation out of 10 of the workflow does not finish, one has the repeat the overall computation). It would be great to have an obvious and easy to use option in jobflow (e.g., based on a file database) where one can easily restart a part of the computations and/or submit the jobs of an workflow in several job scripts.

I think, @utf , we have talked about this before.

[Andrew-S-Rosen]

@JaGeo --- While perhaps not the perfect solution, it sounds like the MontyStore in maggma is very close in spirit to what you are looking for. See here.

[JaGeo]

@arosen93 thanks! I now have to find time to test it. Any other suggestions for approaches without db are also highly appreciated. HPCs in Europe and especially their comoute nodes are usually not easy to connect with database servers.

[utf]

@JaGeo have you spoken to @davidwaroquiers or @gpetretto about Jobflow-remote?

[JaGeo]

Yes, thank you. We have interacted in the meantime. I think we can close this!

[JaGeo]

Hi @utf @davidwaroquiers ,

I wanted to raise this point again. It might be worth considering a middle ground solution for smaller workflows, potentially with another type of store. jf-remote is awesome but still requires some setup.

[Andrew-S-Rosen]

This feature request is likely partially dependent on materialsproject/maggma#832.

[JaGeo]

@Andrew-S-Rosen thanks for linking this!