added family grouping plus family and cycle collapsing #1810
Conversation
|
Still to do (in order of priority).
|
|
Im struggling with grouping by families due to the fact that they can be nested ... The contextCollapsing and Expanding nodes is easier as the .dot file just needs the node/s added/removed .... and the edge (relationship between the node and other nodes) defined... This means I dont have to directly deal with a hierarchy (nested structure) as things are just being added/removed. For grouping the syntax is a little different, using subgraphs ... For grouping by cycle point there are no nested cycle points (doesnt make sense) so its just a case of making subgraph for each cycle. The subgraphs do need to account for the fact that the nodes may have been expanded or collapsed but that can be managed by calculating what nodes need to be included from the The problemThe problem is with nested grouping which is relevant for
The way this is represented in the .dot code is by having subgraphs within subgraphs...
The above is an simple example of some graphviz code for a simple nested family situation. Below is an example for a more complicated one... The subgraphs can be I have thought about giving each node a ranking in terms of how 'deep' it is in the hierarchy then ranking from most deep to least deep then looping through ... but this wont work because (as in the example above) you would miss out a lot of the graph |
|
I think this is a problem that warrants recursion as it's tricky to unroll as an iterative loop. Here's an idea of what that could look like (Python syntax):
from random import random
TASKS = {
'foo': {
'name': 'foo',
'parent': 'FOO',
},
'FOO': {
'name': 'FOO',
'parent': 'root'
},
'bar': {
'name': 'bar',
'parent': 'BAR1',
},
'baz': {
'name': 'baz',
'parent': 'BAR2',
},
'BAR1': {
'name': 'BAR1',
'parent': 'BAR',
},
'BAR2': {
'name': 'BAR2',
'parent': 'BAR',
},
'root': {
'name': 'root',
'parent': None,
},
}
TREE = {
'root': {
'FOO': None,
'BAR': {
'BAR1': None,
'BAR2': None,
},
},
}
def add_subgraph(dotcode, pointer, graph_sections):
for key, value in pointer.items():
dotcode.append(
f'subgraph cluster_{str(random())[2:]} {{'
f'\nlabel = "{key}"'
)
if value:
add_subgraph(dotcode, value, graph_sections)
if key in graph_sections:
dotcode.extend(graph_sections[key])
dotcode.append('}')
return dotcode
def get_dotcode(tasks):
graph_sections = {}
for task in tasks.values():
parent = task['parent']
if not parent:
continue
section = graph_sections.setdefault(parent, [])
section.append(f'{task["name"]} [title="{task["name"]}"]')
dotcode = ['digraph {']
add_subgraph(dotcode, TREE['root'], graph_sections)
return dotcode
for item in get_dotcode(TASKS):
print(item)digraph {
subgraph cluster_23300787190407446 {
label = "FOO"
foo [title="foo"]
}
subgraph cluster_5025488657295563 {
label = "BAR"
subgraph cluster_2135762450670372 {
label = "BAR1"
bar [title="bar"]
}
subgraph cluster_4413670667138756 {
label = "BAR2"
baz [title="baz"]
}
BAR1 [title="BAR1"]
BAR2 [title="BAR2"]
}I haven't taken cycles into account in this solution, you'll need to add a This solution will also add entries for families which have no tasks, so, you'll need some fancy logic for removing empty families, and any families that contain only empty families. |
markgrahamdawson
force-pushed
the
family-subgraphs
branch
from
21472f9 to
e45ccec
Compare
markgrahamdawson
force-pushed
the
family-subgraphs
branch
from
f1540f6 to
676466f
Compare
|
I'm still getting missing dependency arrows, though it was harder to reproduce: [scheduler]
allow implicit tasks = True
cycle point format = %Y
[scheduling]
initial cycle point = 1971
[[graph]]
P1Y = """
X[-P1Y]:succeed-all => start_cycle
start_cycle => X:succeed-all => _dummy_ => Y
"""
[runtime]
[[root]]
script = sleep 1000
[[X, Y]]
[[x]]
inherit = X
[[y]]
inherit = YI had collapset 1972 and X
|
|
I can't break it. 🚀 |
On this one we, we do use that margin attribute - line 855 in the src/views/Graph.vue |
src/views/Graph.vue
Outdated
| return nodeFirstFamily | ||
| } else if (ancestor) { | ||
| return this.allParentLookUp[nodeFirstFamily][0] | ||
| // this is almost certainly an over simplification -> better logic needed here |
There was a problem hiding this comment.
TODO
I'm not 100% on what this function does, but I suspect it's missing a bit.
E.g, for this inheritance:
- A
- A1
- A11
- A111
- A11
- A1
And this state:
| Fam | iscollapsed |
|---|---|
| A | ❌ |
| A1 | ✔️ |
| A11 | ✔️ |
| A111 | ❌ |
What family should be returned?
src/views/Graph.vue
Outdated
| const nodeFormattedArray = children.filter((a) => { | ||
| // if its not in the list of families (unless its been collapsed) | ||
| const isFamily = !this.familyArrayStore.includes(a.name) || this.collapseFamily.includes(a.name) | ||
| // its the node has been removed/collapsed | ||
| const isRemoved = !removedNodes.includes(a.name) | ||
| // is not numeric | ||
| const isNumeric = !parseFloat(a.name) | ||
| return isFamily && isRemoved && isNumeric | ||
| }).map(a => `"${a.id}"`) |
There was a problem hiding this comment.
There is an optimisation we can make here which will help to speed the code up.
E.G:
function one () {
console.log('one')
return false
}
function two () {
console.log('two')
return true
}
function three () {
console.log('three')
return true
}
if (one() && two() && three()) {
console.log('true')
} else {
console.log('false')
}This code contains an if statement that only evaluates true if all three of the tests one(), two() and three() return true.
However, take a look at what happens when you run the code:
$ node test.js
one
falseIt only actually performs the first test (one()), this returns false, so it skips the others.
This is "lazy evaluation", several scripting languages including JS and Python do it. This is a useful tool for optimisation as it helps us to avoid performing unnecessary comparisons.
To apply that to this example:
const nodeFormattedArray = children.filter((a) => {
return (
// if its not in the list of families (unless its been collapsed)
!this.familyArrayStore.includes(a.name) || this.collapseFamily.includes(a.name)
// its the node has been removed/collapsed
&& !removedNodes.includes(a.name)
// is not numeric
&& !parseFloat(a.name)
)
}).map(a => `"${a.id}"`)If the first condition (!this.familyArrayStore.includes(a.name) || this.collapseFamily.includes(a.name)) evaluates false, then the following conditions are not evaluated at all, saving CPU.
We can game this by putting the conditions that are most likely to resolve as false at the top of the list.
src/views/Graph.vue
Outdated
| return nodes.reduce((x, y) => { | ||
| (x[y.tokens.cycle] ||= []).push(y) | ||
| return x | ||
| }, {}) | ||
| }, | ||
| addSubgraph (dotcode, pointer, graphSections) { | ||
| pointer.children.forEach((key, i) => { | ||
| const value = key |
There was a problem hiding this comment.
Why reassign key to value?
There was a problem hiding this comment.
I'm curious about this too - tried getting rid of this either way and both seem to work.
src/views/Graph.vue
Outdated
| // its the node has been removed/collapsed | ||
| const isRemoved = !removedNodes.includes(a.name) | ||
| // is not numeric | ||
| const isNumeric = !parseFloat(a.name) |
There was a problem hiding this comment.
What is the isNumeric check for?
Is this here to filter out jobs?
src/views/Graph.vue
Outdated
| }`) | ||
| const graphSections = {} | ||
| Object.keys(cycles).forEach((cycle, iCycle) => { | ||
| const indexSearch = Object.values(this.cylcTree.$index).filter((node) => { |
There was a problem hiding this comment.
This is searching over the full cylcTree.$index, this is the complete data store which also contains nodes for other workflows.
We only need to iterate over the cycles in this workflow, i.e. Object.keys(cycles).forEach((key, i) => {.
| }) | ||
| return store | ||
| }, | ||
| allParentLookUp () { |
There was a problem hiding this comment.
As someone who isn't super fluent in JS, could I request more docstring?
| }, | ||
| allChildrenLookUp () { | ||
| const lookup = {} | ||
| // Calculate some values for familes that we need for the toolbar |
src/views/Graph.vue
Outdated
| pointer.children.forEach((key, i) => { | ||
| const value = key | ||
| const children = this.allChildrenLookUp[value.id] | ||
| if (!children) { return } | ||
| const removedNodes = new Set() | ||
| children.forEach((a) => { | ||
| if (this.collapseFamily.includes(a.name)) { | ||
| a.children.forEach((child) => { | ||
| removedNodes.add(child.name) | ||
| }) | ||
| } | ||
| }) | ||
| // filter parent | ||
| let openedBrackets = false | ||
| if ( | ||
| children.length && | ||
| this.groupFamily.includes(key.node.name) && | ||
| !this.collapseFamily.includes(key.node.name) && | ||
| !this.collapseFamily.includes(key.node.firstParent.name)) { | ||
| // filter child | ||
| const nodeFormattedArray = children.filter((a) => { | ||
| const isNumeric = !parseFloat(a.name) | ||
| let isAncestor = true | ||
| if (isNumeric) { | ||
| const nodeFirstParent = this.cylcTree.$index[a.id].node.firstParent.name | ||
| isAncestor = !this.isNodeCollapsedByFamily(nodeFirstParent) | ||
| } | ||
| return ( | ||
| // the node is not a numeric value | ||
| isNumeric && | ||
| // if its not in the list of families (unless its been collapsed) | ||
| (!this.familyArrayStore.includes(a.name) || this.collapseFamily.includes(a.name)) && | ||
| // the node has been removed/collapsed | ||
| !removedNodes.has(a.name) && | ||
| // the node doesnt have a collapsed ancestor | ||
| isAncestor | ||
| ) | ||
| }).map(a => `"${a.id}"`) | ||
| if (nodeFormattedArray.length) { | ||
| openedBrackets = true | ||
| dotcode.push(` | ||
| subgraph cluster_margin_family_${key.name}${key.tokens.cycle} | ||
| { | ||
| margin=100.0 | ||
| label="margin" | ||
| subgraph cluster_${key.name}${key.tokens.cycle} | ||
| {${nodeFormattedArray}${nodeFormattedArray.length ? ';' : ''} | ||
| label = "${key.name}" | ||
| fontsize = "70px" | ||
| style=dashed | ||
| margin=60.0 | ||
| `) | ||
| } | ||
| } | ||
|
|
||
| if (value) { | ||
| this.addSubgraph(dotcode, value, graphSections) | ||
| } | ||
| if (Object.keys(graphSections).includes(key)) { | ||
| dotcode.push(graphSections[key.id]) | ||
| } | ||
| if (openedBrackets) { | ||
| dotcode.push('}}') | ||
| } | ||
| }) | ||
| }, |
There was a problem hiding this comment.
This seemed fine:
| pointer.children.forEach((key, i) => { | |
| const value = key | |
| const children = this.allChildrenLookUp[value.id] | |
| if (!children) { return } | |
| const removedNodes = new Set() | |
| children.forEach((a) => { | |
| if (this.collapseFamily.includes(a.name)) { | |
| a.children.forEach((child) => { | |
| removedNodes.add(child.name) | |
| }) | |
| } | |
| }) | |
| // filter parent | |
| let openedBrackets = false | |
| if ( | |
| children.length && | |
| this.groupFamily.includes(key.node.name) && | |
| !this.collapseFamily.includes(key.node.name) && | |
| !this.collapseFamily.includes(key.node.firstParent.name)) { | |
| // filter child | |
| const nodeFormattedArray = children.filter((a) => { | |
| const isNumeric = !parseFloat(a.name) | |
| let isAncestor = true | |
| if (isNumeric) { | |
| const nodeFirstParent = this.cylcTree.$index[a.id].node.firstParent.name | |
| isAncestor = !this.isNodeCollapsedByFamily(nodeFirstParent) | |
| } | |
| return ( | |
| // the node is not a numeric value | |
| isNumeric && | |
| // if its not in the list of families (unless its been collapsed) | |
| (!this.familyArrayStore.includes(a.name) || this.collapseFamily.includes(a.name)) && | |
| // the node has been removed/collapsed | |
| !removedNodes.has(a.name) && | |
| // the node doesnt have a collapsed ancestor | |
| isAncestor | |
| ) | |
| }).map(a => `"${a.id}"`) | |
| if (nodeFormattedArray.length) { | |
| openedBrackets = true | |
| dotcode.push(` | |
| subgraph cluster_margin_family_${key.name}${key.tokens.cycle} | |
| { | |
| margin=100.0 | |
| label="margin" | |
| subgraph cluster_${key.name}${key.tokens.cycle} | |
| {${nodeFormattedArray}${nodeFormattedArray.length ? ';' : ''} | |
| label = "${key.name}" | |
| fontsize = "70px" | |
| style=dashed | |
| margin=60.0 | |
| `) | |
| } | |
| } | |
| if (value) { | |
| this.addSubgraph(dotcode, value, graphSections) | |
| } | |
| if (Object.keys(graphSections).includes(key)) { | |
| dotcode.push(graphSections[key.id]) | |
| } | |
| if (openedBrackets) { | |
| dotcode.push('}}') | |
| } | |
| }) | |
| }, | |
| pointer.children.forEach((value, i) => { | |
| const children = this.allChildrenLookUp[value.id] | |
| if (!children) { return } | |
| const removedNodes = new Set() | |
| children.forEach((a) => { | |
| if (this.collapseFamily.includes(a.name)) { | |
| a.children.forEach((child) => { | |
| removedNodes.add(child.name) | |
| }) | |
| } | |
| }) | |
| // filter parent | |
| let openedBrackets = false | |
| if ( | |
| children.length && | |
| this.groupFamily.includes(value.node.name) && | |
| !this.collapseFamily.includes(value.node.name) && | |
| !this.collapseFamily.includes(value.node.firstParent.name)) { | |
| // filter child | |
| const nodeFormattedArray = children.filter((a) => { | |
| const isNumeric = !parseFloat(a.name) | |
| let isAncestor = true | |
| if (isNumeric) { | |
| const nodeFirstParent = this.cylcTree.$index[a.id].node.firstParent.name | |
| isAncestor = !this.isNodeCollapsedByFamily(nodeFirstParent) | |
| } | |
| return ( | |
| // the node is not a numeric value | |
| isNumeric && | |
| // if its not in the list of families (unless its been collapsed) | |
| (!this.familyArrayStore.includes(a.name) || this.collapseFamily.includes(a.name)) && | |
| // the node has been removed/collapsed | |
| !removedNodes.has(a.name) && | |
| // the node doesnt have a collapsed ancestor | |
| isAncestor | |
| ) | |
| }).map(a => `"${a.id}"`) | |
| if (nodeFormattedArray.length) { | |
| openedBrackets = true | |
| dotcode.push(` | |
| subgraph cluster_margin_family_${value.name}${value.tokens.cycle} | |
| { | |
| margin=100.0 | |
| label="margin" | |
| subgraph cluster_${value.name}${value.tokens.cycle} | |
| {${nodeFormattedArray}${nodeFormattedArray.length ? ';' : ''} | |
| label = "${value.name}" | |
| fontsize = "70px" | |
| style=dashed | |
| margin=60.0 | |
| `) | |
| } | |
| } | |
| if (value) { | |
| this.addSubgraph(dotcode, value, graphSections) | |
| } | |
| if (Object.keys(graphSections).includes(value)) { | |
| dotcode.push(graphSections[value.id]) | |
| } | |
| if (openedBrackets) { | |
| dotcode.push('}}') | |
| } | |
| }) | |
| }, |
Partly addresses issue #1130
The grouping of nodes by cycle point is completed in this pr #1763
----Notes on work----
Some ideas for a unified approach to grouping/collapsing cycles/families. I'm suggesting unifying the handling of cycles and families (note, cycles represent the "root" family so they are essentially the same thing).
Grouping/Ungrouping - Drawing dashed boxes around a cycle/family.
Collapsing/Expanding - Reducing a family down to a single node.
Limitations of the Cylc 7 approach:
Note, for simplicity, this approach groups/collapses all instances of selected families rather than managing this at a per-cycle level. I think this is probably more aligned with expectations, but does represent a minor limitation, e.g. there's no ability to collapse all but one cycle. The ability to expand/collapse specific cycle instances would be a reasonable enhancement.
Design Sketch
Had a quick discussion on this (more to come):
Check List
CONTRIBUTING.mdand added my name as a Code Contributor.setup.cfg(andconda-environment.ymlif present).CHANGES.mdentry included if this is a change that can affect users?.?.xbranch.