Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update validate model-config readme #4632

Merged
merged 7 commits into from
Oct 21, 2024
Merged

update validate model-config readme #4632

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
aaeffd5
update readme
mwdchang Aug 30, 2024
58fb52a
fix wrongly committed file
mwdchang Aug 30, 2024
6868f46
Merge remote-tracking branch 'origin/main' into update-validation-test
mwdchang Aug 30, 2024
06e9998
Merge branch 'main' into update-validation-test
mwdchang Sep 4, 2024
b633d85
Merge branch 'main' into update-validation-test
shawnyama Oct 18, 2024
b88be4e
new instructions
shawnyama Oct 21, 2024
a8b344d
Merge branch 'main' into update-validation-test
shawnyama Oct 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/client/hmi-client/src/components/workflow/ops/funman/tera-funman-drilldown.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@
class="mt-2"
text
icon="pi pi-plus"
label="Add new check"
label="Add constraint"
size="small"
@click="addConstraintForm"
/>
Expand Down Expand Up @@ -107,7 +107,7 @@
<tera-input-number class="mt-1" v-model="knobs.currentTimespan.end" />
</div>
<div class="timespan-input">
<label>Number of timepoints</label>
<label>Number of timesteps</label>
<tera-input-number class="mt-1" v-model="knobs.numberOfSteps" />
</div>
</div>
Expand Down
100 changes: 41 additions & 59 deletions testing/manual/validate-model-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
## Validate Model Configuration
Please go through __every__ step of the test scenario.\
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### Preamble
Validation checks whether a given model-configuration satisfies specific constraints when it is simulated. Then if so, what are the feasible parameter value ranges one can use.

For example, one might have an `SEIRD` mode-configuration, where the infection rate is between `[0.1, 0.3]`, we want to know what is the maximum infection rate in order to keep the infected population under 1200 people across all times. Based on the rest of the model and its initial conditions, validation result may simply say this isn't feasible, or maybe it can be achieved if infection rate can be kept below `0.225`.

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
```
Expand All @@ -11,62 +15,40 @@ Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/te
2. Create, or open, project named `QA [Your Name]`

### 2. Setup the workflow
1. Create a new workflow named `Validate Model Configuration`
2. Upload the [Configured SIR.json](https://github.com/DARPA-ASKEM/terarium/tree/main/testing/data) model
3. Drag and drop the model from the Resources panel onto the workflow
4. Create a _Configure Model_ operator in the workflow by right-clicking anywhere in workflow canvas and selecting _Work with model/Validate configuration_
5. Link the output port of the _Model_ operator to the model port of the _Configure Model_ operator
6. Open the _Configure Model_ operator
7. Verify that the model is reasonably configured by inspecting the value of the initial conditions and parameters, e.g.\
`S = 1000.0, I = 2.0, R = 0.0, β = 0.0006, γ = 0.2`
8. Add uncertainty to the parameters e.g +/- 25%
9. Save as a new configuration
10. Note that this configuration is now available in the _Suggested configurations_ section and appears as the output port of the operator

### 3. Setup the Validate operator
1. Create a _Validate configuration_ operator by right-clicking anywhere in the workflow canvas and selecting _Work with model/validate configuration_
2. Link the output port of the _Configure model_ operator to the _Model configuration_ input port of the _Validate configuration_ operator
3. Open the _Validate configuration_ operator by clicking _Review checks_
4. A _parameters of interest_ dropdown should be available, populated with the parameters and state variables of the model
- select `β` and `γ` (this determines for which inputs the operator should run the validation)
5. In the Settings section, use the _Tolerance_ input and slider \
1. Upload the [SIR to validate.json](https://drive.google.com/drive/folders/1j46RpsEflGzBz2yLnSZbmhs9gexpBnGk) model
2. Create a new workflow named `Validate Model Configuration`
3. Drag and drop the model from the Resources panel onto the workflow.
4. Attach the model to a _Configure model_ operator.
5. Open the _Configure Model_ operator to verify that the model is reasonably configured by inspecting the value of the initial conditions and parameters:
- `S = 2000.0, I = 1.0, R = 0.0, beta = [0.03, 0.05], gamma = [0.01, 0.03]`
6. Attach the _Configure Model_ operator to a _Validate configuration_ operator.

### 3. Setup and run
1. Open the _Validate configuration_ operator.
2. Under settings in the _Parameters of interest_ dropdown select `beta` and `gamma` (this determines for which inputs the operator should run the validation)
3. In the Settings section, use the _Tolerance_ input and slider \
- set the value to `0.05` (this determines how granular the operator segment the input space of the model for validation)


### 4. Add sanity checks
1. Ensure that the _Compartmental constraint_ is present (it requires the operator to check that the sum of the state variables at all times is equal to their sum at time = 0 and that each state variable is positive) \
`I + R + S = 1002.0, I >= 0, R >= 0, S >= 0`
2. Create a new sanity check that ensures the number of infected persons is less than `750.0` in the first `100` days by first clicking _Add new check_
3. Name it `Infected cap`
4. Enter the values: The `State variables` `I` should be less than or equal to `750` persons from timepoint `0` to `100`

### 5. Successful Validation
1. Click _Run_
2. Select `I` in _Trajectory State_ dropdown
3. Note that the time-series chart show the evolution of the model variables as a function of time, with one trajectory per sampled points in parameter space (coloured green when they satisfy all sanity checks, yellow otherwise)
4. Verify that the x-axis of this chart include the given _Start time_ and _End time_ (`0` and `100`) and that the trajectories have as many points as specified by _Number of steps_ (`10`)
5. Verify that there is a semi-transparent green rectangle (`width = [0, 100] days, height = [0, 750]`), representing the region in which a trajectory needs to be for satisfiability
6. Select the parameter space plot associated with the `γ` parameter under _Configuration parameters_
7. Verify that a _β:γ pairwise drilldown_ chart appears and selecting each of the rectangular regions (labelled `box<N>`) highlights a trajectory in the _Trajectory State_ chart (both the selected rectangle and highlighted trajectory should shared the same colour)
8. Verify that the x-axis is labelled `β` with the same range as configured (`[0.4, 0.8]`) and ditto for the y-axis (`γ` and `[0.1, 0.3]`)
9. Verify that every green rectangular region maps to a green trajectory which is always between `I = 0.0` and `I = 750.0`
10. Click _X_ on the _drilldown_ chart should close it

### 6. Tighten sanity check
1. Lower the upper bound of the `Infected cap` constraint from `750.0` to `100.0` (this should restrict the regions of parameter space that can satisfy this sanity check)
2. Click `Run` again
3. Verify that orange trajectories appear in the _Trajectory State_ chart and they violate the newly tightened sanity check (i.e. `I(t) > 100.0` at some time `t`)
4. Verify that orange rectangular regions appear in the parameter space plots and they map to the orange trajectories
5. Delete this constraint by clicking the _Bin_ icon

### 7. Create a monotonicity constraint
1. Create a new constraint named `Sus` with type `Monotonicity constraint` and target `S`
2. Set it to _Decreasing_ to check whether the state variable `S` always decrease at every time point
3. Click _Run_
4. Select `I` in the _Trajectory State_ dropdown
5. Verify that all parameter space regions are coloured green and map to trajectories that always decrease in value over time, within the simulated time range (i.e. `dS/dt <= 0`)

### 8. Create an impossible monotonicity constraint
1. Change the direction of the `Sus` constraint from `Decreasing` to `Increasing`
2. Click _Run_ again
3. Verify that all parameter space regions are coloured orange and all trajectories do not satisfy the new constraint, i.e. no combination of parameter values as input into the model produces outputs with `S` increasing over time
4. Ensure that the _Compartmental constraint_ is present (it requires the operator to check that the sum of the state variables at all times is equal to their sum at time = 0 and that each state variable is positive): `S >= 0, I >= 0, R >= 0, S + I + R = 2001.0`
5. Choose one of the following steps:
- a. Add a constraint that ensures that state variables S and R should be greater than or equal to `500` persons from `20` to `40` days.
- b. Add a constraint that ensures that state variables S and R should be less than `500` persons from `20` to `60` days.
6. Click _Run_

### 5. Inspect output
#### State charts
1. Once an output has been generated look at the `S` state chart
2. Note that the time-series chart shows the evolution of the model variables as a function of time, with one trajectory per sampled points in parameter space (coloured green when they satisfy all sanity checks, yellow otherwise)
3. Verify that:
- The points of the trajectories are plotted as specified by _Number of timesteps_ (`10`)
- The light blue model check box matches the constraint configuration (you won't see it yet if you chose step 3.5b)
4. In the Output settings slider check the checkbox that says _Only show furthest results_. See if the furthest true, false and ambiguous boxes are the only ones plotted (step 3.5a's constraint will just show you one true box).
5. Check the _Focus on model checks_ checkbox. See if the chart changes to show the large range that the model check covers.
6. Uncheck both checkboxes.

#### Parameter charts
1. Click on some ticks on the parameter charts.
2. Notice that the tick's value corresponds to ticks on other parameter charts and a line in the state chart as they are highlighted.
3. To show everything again click a space on the parameter bar that doesn't have a tick.

### 6. Test the other constraint
1. Delete your constraint and create the other constraint in Step 3.5, run it and redo Step 5.