Cypher Questions

[rstyd]

This discussion is for asking questions about BEE's cypher code.

[kabir-vats]

@Boogie3D I have some clarification questions from the cypher code:

What does a 'BEGINS' edge mean? It appears to connect all tasks to the central workflow node, but there seems to be a complicated query to create the edge.

How does the workflow know a task is ready to be run with set_runnable_tasks_to_ready? The query checks if the input.values are not NULL, but wouldn’t the values always not be NULL? In this case, why have a separate method for set_init_tasks_to_ready?

Thank you for your help, I can tell you have put in a lot of work to implement neo4j on this project.

[Boogie3D]

Hi,

So the BEGINS edge connects the Workflow node specifically to the tasks which take Workflow Inputs as specified in the Workflow CWL file as their task inputs. If a task depends entirely on task outputs as their inputs then it will not have a BEGINS edge.

In general, all BEGINS edges are determined by matching task input sources to workflow inputs and DEPENDS_ON edges are determined by matching task input sources to task outputs.

set_init_tasks_to_ready specifically sets to ready the tasks whose inputs depend entirely on workflow inputs and not any other tasks, i.e. those tasks which have no DEPENDS_ON edges. set_runnable_tasks_to_ready checks if a task's inputs are entirely satisfied, either by workflow inputs or task outputs. I believe tasks which depend on other tasks for their input will have a NULL input.value field until set_task_input is invoked to set the task input.value. It may be possible for set_runnable_tasks_to_ready to subsume the role of set_init_tasks_to_ready, I just didn't really think of it at the time.

Let me know if you have any other questions!

Thanks,
Steven

[rstyd]

Hey @Boogie3D,

I was wondering if it's possible for the parser to parse inline inputs instead of them being included with a separate YAML file. The same way that the run section for a workflow doesn't necessarily need to be a link to another file.

As an example,

The lulesh workflow is

inputs:
  size: int
  iterations: int

outputs:
  lulesh_stdout:
    type: File
    outputSource: lulesh/lulesh_stdout

steps:
  lulesh:
    run:
      class: CommandLineTool
      baseCommand: [/lulesh2.0]
      stdout: lulesh_stdout.txt
      inputs:
        size:
          type: int
          inputBinding:
            prefix: -s
        iterations:
          type: int
          inputBinding:
            prefix: -i
      outputs:
        lulesh_stdout:
          type: stdout
    in:
      size: size
      iterations: iterations
    out: [lulesh_stdout]
    hints:
      DockerRequirement:
        beeflow:useContainer: /usr/projects/beedev/mpi/lulesh-x86_64.tgz
      beeflow:MPIRequirement:
        ntasks: 27

And the accompanying YAML file looks like

size: 20
iterations: 10

I'd like to change the cwl file so it looks like:

inputs:
  size: int
  iterations: int

outputs:
  lulesh_stdout:
    type: File
    outputSource: lulesh/lulesh_stdout

steps:
  lulesh:
    run:
      class: CommandLineTool
      baseCommand: [/lulesh2.0]
      stdout: lulesh_stdout.txt
      inputs:
        size:
          type: int
          inputBinding:
            prefix: -s
        iterations:
          type: int
          inputBinding:
            prefix: -i
      outputs:
        lulesh_stdout:
          type: stdout
    in:
      size: 20
      iterations: 10
    out: [lulesh_stdout]
    hints:
      DockerRequirement:
        beeflow:useContainer: /usr/projects/beedev/mpi/lulesh-x86_64.tgz
      beeflow:MPIRequirement:
        ntasks: 27

Is that actually possible/reasonable? We have a customer that would like to avoid using input YAML files if at all possible.

[Boogie3D]

Apologies for the late response. It looks like the CWL way to do it is to add a default field to the workflow input(s) (https://cwl.discourse.group/t/constructing-a-default-value-for-a-tool-input/161/4).

I don't think it would be hard to modify the CWL parser to handle this.

[Boogie3D]

However, I think the more conventional way to do this type of "quick-and-dirty" workflow parameter is to use the arguments field, which would need to be implemented in the parser. https://www.commonwl.org/user_guide/topics/additional-arguments-and-parameters.html