Organization of experimental results when code evolves over time

[bachi55]

Hello,

thank you for your great work. I would like to suggest a topic for a future RSH session.

Central question:
How to version control / organize experimental results (perhaps also figures and tables)?

Description:
When we run experiments, we get results. Than we change code, and we get new results. This either ends up being a mess or one can find some kind of organization for it.

I personally work with dictionary structures encoding the experimental parameters. Furthermore, there is a base directory for each major change of the algorithm / implementation. So I can have parallel result "branches".

However, I always felt that this isn't the "best" way to do it. It is, for example, currently not linked to the changes of the code base in git. I remember the code-refinery workshop touched that topic using makefiles. I tried it (where rudimentary I admit), but it felt I little bit clumsy.

Maybe, you can in a future session look at tools / workflows that can organize experiments, e.g. using a DB or even just more examples on this makefile idea.

Best regards,
Eric

EDIT: Here is an example, if how I handled this problem in one of my previous research projects.

[bast]

Thanks Eric for the suggestion and the example! Great idea.

[bast]

How long do typical such experiments/computations take?

[bast]

I might talk about it tomorrow but I am now doing some research first on what are good solutions.

[bast]

I am just reading and learning about https://dvc.org/ which looks like a great fit for this problem. I need to read more but might discuss this.

[bachi55]

How long do typical such experiments/computations take?

I would say the example runs fast. In the debug-mode even faster. If you want to try to run it, I recommend to make the requirements hard (= instead of >=). Recently someone has pointed out difficulties install all the deps. As you talked about in your podcast.

[bast]

Eric, is it OK if I mention your name on stream when discussing this contribution?

[bachi55]

Eric, is it OK if I mention your name on stream when discussing this contribution?

Yes, you can do that.