The openeo-validation-engine uses the OpenEO-sentinel-reference-jobs as a submodule and fetches the results of these jobs from every specified back-end provider. The results are then validated and a report is generated for each job. These reports are then pushed to the gh-pages branch of this repository.
Due to filesize limitations, comparison images are currently not stored in this process.
We use Python3 in a venv environment. Using a venv is not necessary but recommended. The following commands have to be run sequentially to install the application and its dependencies.
git clone --recursive https://github.com/Open-EO/openeo-result-validation-engine
cd openeo-result-validation-engine
python3 -m venv ./venv
source venv/bin/activate
pip install -r requirements.txt
cd openeo-python-client
pip install -r requirements.txt
pip install -e .OpenEO back-end providers can be added in the backendProvider.json. This JSON contains an array of JSON objects that contain the information for a given provider.
name: The name of the backend-provider.baseURL: The address of the openEO API endpoint from the backend-provider.credentials: The account information required to authenticate openEO requests.user: The username, this is optional and can be left emptypassword: The password to authenticate, this is optional and can be left empty
{
"name": "WWU_GEE",
"baseURL": "https://earthengine.openeo.org/v0.3",
"credentials": {
"user": "Example",
"password": "Example123"
}
}Reference jobs (jobs that are validated) can be added in the folder openeo-sentinel-reference-jobs. They are currently
separated by arbitrary regions to make it a bit easier to distinguish them. Each Job has a folder that should be called
Job-N (N is the counter). This folder then has to contain a folder that matches the name specified in the backendProvider.json,
e.g. openEO-GEE in our current example. This folder then contains a single process graph.
Validation rules can be configured on a per job basis, in a file called validation-rules.json. There are three possible options:
-
Specifying no rules;
defaultvalidation rules get selected (stored in the folderrule-configurations) -
Specifying a
nameof a ruleset; the validation engine then tries to find a json file with the namename-validation-rules.json
{
"ruleset-name": "quick"
}- Specifying validation rules directly inside the json.
{
"validation": {
"rules": {
"input-data-checks": {
},
"output-data-checks": {
"matching-file-extensions": "png",
"file-size-check": 0.2
},
"color-checks": {
"threshold-correlation": 0.5,
"threshold-intersection": 345,
"threshold-chi-squared": 1500,
"threshold-hellinger": 0.5,
"threshold-chi-square-alt": 5000,
"threshold-kullback-leibler": 5000
},
"pixel-checks": {
"resolution-allow-divergence": 0.5,
"image-similarity-measures": 0.95,
"resize-factor": 0.5
},
"classification-checks": {
"matching-boundaries": 0.20,
"resize-factor": 0.5
},
"nan-value-check": {
"allow-nan": false
}
}
}
}Activate the Python venv and start the ValidationEngine.
source venv/bin/activate
python ValidationEngine.pyRun python ValidationEngine.py --help to get information on all available CLI commands.
If you want to prevent the validation engine from downloading the same OpenEO jobs again you can pass the argument --offline True.
Example:
python ValidationEngine.py --offline True
To speed up the validation, a resize factor can be used. This factor reduces the size of the image and thus all computations are quicker. A value of 0.10 results in an image that is 10% of the original size. This of course reduces the quality of the validation.
Example:
python ValidationEngine.py --resize 0.10
Mocked examples (jobs stored in the folder mock-examples) can be run with:
python ValidationEngine.py --mock True To run only a specific reference job, the job-identifier can be passed as an argument. The structure of the id is as follows: -Job- Examples: Island-Job-2, Netherlands-Job-2
python ValidationEngine.py --refJob Switzerland-Job-1 Add a similar json-object as the one below to the backendProvider.json.
{
"name": "Local-Python",
"local": true
}In the folder openeo-sentinel-reference-jobs, add a folder with the name of the backend provider, e.g. Local-Python.
This folder then should contain a json that stores the path to the locally stored file instead of the process-graph.
{
"file": "/path/to/file/file.tiff"
}This allows to validate local results against openEO cloud-processed results and it also enables validation of just local results.
The ValidationEngine runs every 24 hours to compute the validation results for two "studies".
Validation reports for the comparison between back-end providers EURAC and WWU/GEE can be found in the branch gh-pages of this repository. It only contains the JSON reports as the raw data from EURAC is to large.
Validation reports for the comparison between the COPERNICUS/S2 and COPERNICUS/S2_SR data set of the WWU/GEE back end can be found in the branch reports-GEE_S2-vs-GEE_SR of this repository. These branch also contains imagery, as we are not able to retrieve native resolution from the WWU/GEE back end and thus are able to store the images on GitHub.
To publish the validation reports to Github this repository needs a valid GitHub Token.
- install the Travis CI Command Line Client by running
gem install travisand login withtravis login --org - Generate SSH-KEY
ssh-keygen -t rsa -b 4096 -C "[email protected]" - Add deploy_key.pub to the repository https://github.com/Open-EO/openeo-result-validation-engine/settings/keys
- Encrypt the deploy_key.pub file with the Travis CLI
travis encrypt-file deploy_key --add (This will generate an encryption label) - Adjust encryption label in .travis.yml line 21