Multiple steps in workflow branch

[aetherfaerber]

After many days of trying we still can't figure out how non-trivial workflows are supposed to work or be modeled in Kitodo.

„Das Problem: Eine Aufgabe in paralleler Verzweigung kann keine zweite Aufgabe haben.“
„Das Problem: Auf die Parallel-Verzweigung folgt nur eine Aufgabe!“

As a result of these two problems, every branch is limited to one task before a joining is needed. But in case of parallel tasks like archiving the source images and publishing the derivative images several tasks are required. It also does not make any sense at all to serialize these tasks because the publication branch does not have to wait for the archiving to finish and vice versa.

How do you all work around these limitations?

The only thing we came up with so far is to create another fork in the branch which holds the second task and a bogus “continue“ task which does nothing but is finished right away:

I hope we all can agree that this solution is far from desirable.

Another possibility would be to always group two tasks, one from each branch, in one pair of gateways. But these two have no relation to each other at all and even worse, as a result, the following tasks will be held back by the completion of completely unrelated tasks:

So again: how do you get an usable and at the same time reasonable workflow out of this?

[solth]

@aetherfaerber yes, you are correct, this is an annoying limitation of the current implementation of parallel tasks in Kitodo.Production workflows. Unfortunately this is not a trivial change to implement, because parallelism of tasks in Production is determined via their ordering property. This property in turn is also used to calculated the progress of a process, so there are quite a few places that need adjustment in order to implement a more fully-fledged concept of parallel tasks in Kitodo.Production.

As far as I know the only institution that currently uses parallel tasks is the Swiss Federal Archive, where parallel tasks are used to allow scanning on multiple scanners at the same time. Since no additional steps are requered in the indvidual parallel branches/tasks after the scanning tasks, the current implementation of Kitodo.Production is sufficient for this use case.

We are currently in the process of modeling a similar workflow with multiple tasks in parallel workflow branches for another project. Here, we circumvent the described limitation by putting all parallel task sequences into one linear sequence and simulate the parallelism by adding mutually exclusive "workflow conditions" to each of the individual task sequences.

So the following "real" parallelism

                 [Task 2 A] --- [Task 2 B]
                /                         \
[Task 1] --- [X]                          [X] --- [Task 4] 
                \                         /
                 [Task 3 A] --- [Task 3 B]

could be configured in the following way:

[Task 1] --- [Task 2 A]* --- [Task 2 B]* --- [Task 3 A]** --- [Task 3 B]** --- [Task 4]

where (*) and (**) denote mutually exclusive workflow conditions.

This will only work for mutually exclusive parallel task branches, though. This means, if you want branch a and b to be potentially performed at the same time, this solution will not be of any help to you.

[aetherfaerber]

@aetherfaerber yes, you are correct, this is an annoying limitation of the current implementation of parallel tasks in Kitodo.Production workflows. Unfortunately this is not a trivial change to implement, because parallelism of tasks in Production is determined via their ordering property. This property in turn is also used to calculated the progress of a process, so there are quite a few places that need adjustment in order to implement a more fully-fledged concept of parallel tasks in Kitodo.Production.

😰

As far as I know the only institution that currently uses parallel tasks is the Swiss Federal Archive, where parallel tasks are used to allow scanning on multiple scanners at the same time. Since no additional steps are requered in the indvidual parallel branches/tasks after the scanning tasks, the current implementation of Kitodo.Production is sufficient for this use case.

I'm honestly surprised. And stunned.

We are currently in the process of modeling a similar workflow with multiple tasks in parallel workflow branches for another project. Here, we circumvent the described limitation by putting all parallel task sequences into one linear sequence and simulate the parallelism by adding mutually exclusive "workflow conditions" to each of the individual task sequences.

So the following "real" parallelism

                 [Task 2 A] --- [Task 2 B]
                /                         \
[Task 1] --- [X]                          [X] --- [Task 4] 
                \                         /
                 [Task 3 A] --- [Task 3 B]

could be configured in the following way:

[Task 1] --- [Task 2 A]* --- [Task 2 B]* --- [Task 3 A]** --- [Task 3 B]** --- [Task 4]

where (*) and (**) denote mutually exclusive workflow conditions.

This will only work for mutually exclusive parallel task branches, though. This means, if you want branch a and b to be potentially performed at the same time, this solution will not be of any help to you.

Thank you for pointing towards this solution. As you say, this will not help with a parallel workflow branches.
We had considered conditions as a possible solution but could not see how to easily create a script that checks if a given task in the current workflow has been finished and changes the status of another task in accordance (e.g. set Task 4 to ”open“ if Task 2 B has been finished or in my example set „Festplatte löschen“ to open as soon as „Prüfsummenabgleich durchführen“ has finished but not if only „Nutzungsderivate anhängen“ has finished). Is there any documentation on this? Is this even possible in a a way that does not feel like reimplementing a workflow engine besides the already existing workflow engine?