Skip to content

Commit

Permalink
Describe how to check if test passed
Browse files Browse the repository at this point in the history
  • Loading branch information
@AnHeuermann
AnHeuermann committed Jan 11, 2024
1 parent e8c6184 commit 52fdf9b
Show file tree
Hide file tree
Showing 2 changed files with 87 additions and 20 deletions.
80 changes: 78 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,27 @@ Character to separate model names in reference files.
E.g. for `Modelica.Blocks.Examples.PID_Controller.mat` it would be `'.'`\
Default: `'.'`

### `publish-gh-pages`

When `publish-gh-pages` is true (default) the results are pushed to branch specified by
`gh-pages-ref`.
If [GitHub Pages](https://pages.github.com/) is setup to
[publish from the specified branch](https://docs.github.com/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#publishing-from-a-branch)
the results of PRs and branches are getting published automatically.

> [!NOTE]
> For the action to be allowed to push to your repository you'll need to give write
> permissions.
> ```yml
> permissions:
> contents: write
> ```

### `gh-pages-ref`

Set branch to push HTML results to, so they can be published with GitHub Pages.
Default: `'gh-pages'`

## Example usage

```yaml
Expand All @@ -67,6 +88,7 @@ jobs:
pull-requests: write
steps:
- uses: openmodelica-library-testing
id: library-testing
with:
package-name: MyLibrary
package-version: 2.2
Expand All @@ -77,22 +99,41 @@ jobs:
reference-files-dir: ReferenceFiles
reference-files-format: mat
reference-files-delimiter: .
publish-gh-pages: true
gh-pages-ref: gh-pages
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

## Outputs

The action will not fail if test fail but will return variables that can be checked in an
additional step in your workflow.

```yml
- name: Check test results
shell: bash
run: |
echo "simulation-tests-passing: ${{ steps.library-testing.outputs.simulation-tests-passing }}"
echo "verification-tests-passing: ${{ steps.library-testing.outputs.verification-tests-passing }}"
if [ "${{ steps.library-testing.outputs.verification-tests-passing }}" == "True" ]; then
exit 0;
else
exit 1;
fi
```

## `simulation-tests-passing`

True if all simulation tests are passing.
`'True'` if all simulation tests are passing, `'False'` otherwise.

## ` n-simulation-passing`

Number of successful simulation tests.

## `verification-tests-passing`

True if all verification tests are passing.
`'True'` if all verification tests are passing, `'False'` otherwise.

## `n-verification-passing`

Expand All @@ -102,6 +143,13 @@ Number of successful verification tests.

### HTML Results

#### GitHub Pages

When `publish-gh-pages` is true (default) the results are pushed to branch specified by
`gh-pages-ref`.

#### Self hosted

Download the `MyLibrary.html.zip` artifact, unzip it and start a HTML server to display
the results. This can be used to host results on a server or GitHub pages.

Expand All @@ -113,3 +161,31 @@ python3 -m http.server -d html
### SQlite

For future test the SQlite data base `sqlite3.db` is achieved.

## Example

This action tests Modelica library [MyLibrary](examples/MyLibrary/package.mo) consisting
of two models from the Modelica Standard Library and compares them to reference results
in [examples/ReferenceFiles](examples/ReferenceFiles) taken from
[https://github.com/modelica/MAP-LIB_ReferenceResults](https://github.com/modelica/MAP-LIB_ReferenceResults/blob/v4.0.0).
The reference results for MyLibrary.Blocks.Examples.PID_Controller are altered to check
that verification will fail for variables `spring.w_rel`, `spring.phi_rel`, `inertia1.w`
and `inertia1.phi`.

The expected output is:

> # GitHub Actions Test summary
> ## Summary
>
> | | Total | Frontend | Backend | SimCode | Templates | Compilation | Simulation | Verification |
> |---:|--------:|-----------:|----------:|----------:|------------:|--------------:|-------------:|---------------:|
> | 0 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 1 |
>
> ## Results
> | | Model | Verified | Simulate | Total buildModel | Parsing | Frontend | Backend | SimCode | Templates | Compile |
> |---:|:------------------------------------------------------|:------------------|-----------:|-------------------:|----------:|-----------:|----------:|----------:|------------:|----------:|
> | 0 | MyLibrary.Blocks.Examples.PID_Controller (sim) | 0.06 (4/7 failed) | 0.03 | 2.46 | 1.86 | 0.23 | 0.03 | 0.01 | 0.03 | 2.16 |
> | 1 | MyLibrary.Mechanics.MultiBody.Examples.Pendulum (sim) | 0.01 (3 verified) | 0.26 | 3.37 | 1.86 | 0.25 | 0.37 | 0.02 | 0.05 | 2.67 |

The HTML results can be hosted with GitHub Pages, for this example they can be found at
[https://anheuermann.github.io/openmodelica-library-testing-action](https://anheuermann.github.io/openmodelica-library-testing-action/).
27 changes: 9 additions & 18 deletions action.yml
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
name: 'openmodelica-library-testing'
description: 'Setup OpenModelicaLibraryTesting script and run them on Modelica package and publishes results.'
description: 'Setup OpenModelicaLibraryTesting scripts and run them on Modelica package and publishes results.'
author: 'Andreas Heuermann'

# Define your inputs here.
# Action inputs
inputs:
package-name:
description: 'Name of Modelica package to test.'
Expand Down Expand Up @@ -46,6 +46,7 @@ inputs:
required: true
default: 'gh-pages'

# Action outputs
outputs:
simulation-tests-passing:
description: 'True if all simulation tests are passing'
Expand All @@ -66,7 +67,7 @@ runs:
- name: Setup OpenModelica
uses: AnHeuermann/[email protected]
with:
version: stable
version: ${{ inputs.omc-version }}
packages: |
'omc'
libraries: ${{ inputs.dependencies }}
Expand Down Expand Up @@ -99,12 +100,12 @@ runs:
- name: Run library test
working-directory: OpenModelicaLibraryTesting
shell: bash
run: python test.py --branch="omc-stable" --noclean configs/myConf.json
run: python test.py --branch="${{ inputs.omc-version }}" --noclean configs/myConf.json

- name: Generate HTML results
working-directory: OpenModelicaLibraryTesting
shell: bash
run: python report.py --branches="omc-stable" configs/myConf.json
run: python report.py --branches="${{ inputs.omc-version }}" configs/myConf.json

- name: Summary
id: summary
Expand All @@ -115,7 +116,7 @@ runs:
- name: Zip HTML results
shell: bash
run: |
python scripts/archieveResults.py "${{ inputs.package-name }}" "${{ github.ref_name }}" "omc-stable" "OpenModelicaLibraryTesting" "html/"
python scripts/archieveResults.py "${{ inputs.package-name }}" "${{ github.ref_name }}" "${{ inputs.omc-version }}" "OpenModelicaLibraryTesting" "html/"
- name: Archive sqlite3.db
uses: actions/upload-artifact@v4
Expand All @@ -142,7 +143,7 @@ runs:
- name: Remove old files/ directory
if: ${{ inputs.publish-gh-pages }}
shell: bash
run: rm -rf omc-stable/files
run: rm -rf ${{ inputs.omc-version }}/files

- name: Get HTML artifact
if: ${{ inputs.publish-gh-pages }}
Expand All @@ -154,19 +155,9 @@ runs:
if: ${{ inputs.publish-gh-pages }}
uses: EndBug/add-and-commit@v9 # You can change this to use a specific version.
with:
add: 'omc-stable index.html'
add: '${{ inputs.omc-version }} index.html'
default_author: github_actions
fetch: false

# The message for the commit.
# Default: 'Commit from GitHub Actions (name of the workflow)'
#message: 'Coverage for ${{ github.ref_name }}'

# The way the action should handle pathspec errors from the add and remove commands. Three options are available:
# - ignore -> errors will be logged but the step won't fail
# - exitImmediately -> the action will stop right away, and the step will fail
# - exitAtEnd -> the action will go on, every pathspec error will be logged at the end, the step will fail.
# Default: ignore
pathspec_error_handling: ignore

- name: Checkout
Expand Down

0 comments on commit 52fdf9b

Please sign in to comment.