openQA is an automated test tool that makes it possible to test the whole installation process of an operating system. It uses virtual machines to reproduce the process, check the output (both serial console and screen) in every step and send the necessary keystrokes and commands to proceed to the next. openQA can check whether the system can be installed, whether it works properly in 'live' mode, whether applications work or whether the system responds as expected to different installation options and commands.
Even more importantly, openQA can run several combinations of tests for every revision of the operating system, reporting the errors detected for each combination of hardware configuration, installation options and variant of the operating system.
openQA is free software released under the GPLv2 license. The source code and documentation are hosted in the os-autoinst organization on GitHub.
This document describes the general operation and usage of openQA. The main goal is to provide a general overview of the tool, with all the information needed to become a happy user.
For a quick start, if you already have an openQA instance available you can refer to the section Cloning existing jobs - openqa-clone-job directly to trigger a new test based on already existing job. For a quick installation refer directly to Quick bootstrapping under openSUSE or Container based setup.
For the installation of openQA in general see the Installation Guide, as a user of an existing instance see the Users Guide. More advanced topics can be found in other documents. All documents are also available in the official repository.
Although the project as a whole is referred to as openQA, there are in fact several components that are hosted in separate repositories as shown in the following figure.
The heart of the test engine is a standalone application called 'os-autoinst' (blue). In each execution, this application creates a virtual machine and uses it to run a set of test scripts (red). 'os-autoinst' generates a video, screenshots and a JSON file with detailed results.
'openQA' (green) on the other hand provides a web based user interface and infrastructure to run 'os-autoinst' in a distributed way. The web interface also provides a JSON based REST-like API for external scripting and for use by the worker program. Workers fetch data and input files from openQA for os-autoinst to run the tests. A host system can run several workers. The openQA web application takes care of distributing test jobs among workers. Web application and workers can run on the same machine as well as connected via network on multiple machines within the same network or distributed. Running the web application as well as the workers in the cloud is perfectly possible.
Note that the diagram shown above is simplified. There exists a more sophisticated version which is more complete and detailed. (The diagram can be edited via its underlying GraphML file.)
| test modules |
an individual test case in a single perl module file, e.g. "sshxterm". If not further specified a test module is denoted with its "short name" equivalent to the filename including the test definition. The "full name" is composed of the test group (TBC), which itself is formed by the top-folder of the test module file, and the short name, e.g. "x11-sshxterm" (for x11/sshxterm.pm) |
| test suite |
a collection of test modules, e.g. "textmode". All test modules within one test suite are run serially |
| job |
one run of individual test cases in a row denoted by a unique number for one instance of openQA, e.g. one installation with subsequent testing of applications within gnome |
| test run |
equivalent to job |
| test result |
the result of one job, e.g. "passed" with the details of each individual test module |
| test step |
the execution of one test module within a job |
| distri |
a test distribution but also sometimes referring to a product (CAUTION: ambiguous, historically a "GNU/Linux distribution"), composed of multiple test modules in a folder structure that compose test suites, e.g. "opensuse" (test distribution, short for "os-autoinst-distri-opensuse") |
| product |
the main "system under test" (SUT), e.g. "openSUSE", also called "Medium Types" in the web interface of openQA |
| job group |
equivalent to product, used in context of the webUI |
| version |
one version of a product, don’t confuse with builds, e.g. "Tumbleweed" |
| flavor |
a specific variant of a product to distinguish differing variants, e.g. "DVD" |
| arch |
an architecture variant of a product, e.g. "x86_64" |
| machine |
additional variant of machine, e.g. used for "64bit", "uefi", etc. |
| scenario |
A composition of
|
| build |
Different versions of a product as tested, can be considered a "sub-version" of version, e.g. "Build1234"; CAUTION: ambiguity: either with the prefix "Build" included or not |
One of the most important features of openQA is that it can be used to test several combinations of actions and configurations. For every one of those combinations, the system creates a virtual machine, performs certain steps and returns an overall result. Every one of those executions is called a 'job'. Every job is labeled with a numeric identifier and has several associated 'settings' that will drive its behavior.
A job goes through several states. Here is (an incomplete list) of these states:
-
scheduled Initial state for newly created jobs. Queued for future execution.
-
setup/running/uploading In progress.
-
cancelled The job was explicitly cancelled by the user or was replaced by a clone (see below) and the worker has not acknowledged the cancellation yet.
-
done The worker acknowledged that the execution finished or the web UI considers the job as abandoned by the worker.
Jobs in the final states 'cancelled' and 'done' have typically gone through a whole sequence of steps (called 'testmodules') each one with its own result. But in addition to those partial results, a finished job also provides an overall result from the following list.
-
none For jobs that have not reached one of the final states.
-
passed No critical check failed during the process. It does not necessarily mean that all testmodules were successful or that no single assertion failed.
-
failed At least one assertion considered to be critical was not satisfied at some point.
-
softfailed At least one known, non-critical issue has been found. That could be that workaround needles are in place, a softfailure has been recorded explicitly via
record_soft_failure(from os-autoinst) or a job failure has been ignored explicitly via a job label. -
timeout_exceeded The job was aborted because
MAX_JOB_TIMEorMAX_SETUP_TIMEhas been exceeded, see Changing timeout for details. -
skipped Dependencies failed so the job was not started.
-
obsoleted The job was superseded by scheduling a new product.
-
parallel_failed/parallel_restarted The job could not continue because a job which is supposed to run in parallel failed or was restarted.
-
user_cancelled/user_restarted The job was cancelled/restarted by the user.
-
incomplete The test execution failed due to an unexpected error, e.g. the network connection to the worker was lost.
Sometimes, the reason of a failure is not an error in the tested operating system itself, but an outdated test or a problem in the execution of the job for some external reason. In those situations, it makes sense to re-run a given job from the beginning once the problem is fixed or the tests have been updated. This is done by means of 'cloning'. Every job can be superseded by a clone which is scheduled to run with exactly the same settings as the original job. If the original job is still not in 'done' state, it’s cancelled immediately. From that point in time, the clone becomes the current version and the original job is considered outdated (and can be filtered in the listing) but its information and results (if any) are kept for future reference.
One of the main mechanisms for openQA to know the state of the virtual machine is checking the presence of some elements in the machine’s 'screen'. This is performed using fuzzy image matching between the screen and the so called 'needles'. A needle specifies both the elements to search for and a list of tags used to decide which needles should be used at any moment.
A needle consists of a full screenshot in PNG format and a json file with the same name (e.g. foo.png and foo.json) containing the associated data, like which areas inside the full screenshot are relevant or the mentioned list of tags.
{
"area" : [
{
"xpos" : INTEGER,
"ypos" : INTEGER,
"width" : INTEGER,
"height" : INTEGER,
"type" : ( "match" | "ocr" | "exclude" ),
"match" : INTEGER, // 0-100. similarity percentage
"click_point" : CLICK_POINT, // Optional click point
},
...
],
"tags" : [
STRING, ...
]
}There are three kinds of areas:
-
Regular areas define relevant parts of the screenshot. Those must match with at least the specified similarity percentage. Regular areas are displayed as green boxes in the needle editor and as green or red frames in the needle view (green for matching areas, red for non-matching ones).
-
OCR areas also define relevant parts of the screenshot. However, an OCR algorithm is used for matching. In the needle editor OCR areas are displayed as orange boxes. To turn a regular area into an OCR area within the needle editor, double click the concerning area twice. Note that such needles are only rarely used.
-
Exclude areas can be used to ignore parts of the reference picture. In the needle editor exclude areas are displayed as red boxes. To turn a regular area into an exclude area within the needle editor, double click the concerning area. In the needle view exclude areas are displayed as gray boxes.
Each regular match area in a needle can optionally contain a click point.
This is used with the testapi::assert_and_click function to match GUI
elements such as buttons and then click inside the matched area.
{
"xpos" : INTEGER, // Relative coordinates inside the match area
"ypos" : INTEGER,
"id" : STRING, // Optional
}Each click point can have an id, and if a needle contains multiple click points
you must pass it to testapi::assert_and_click to select which click point
to use.
Some actions in openQA require special privileges. openQA provides authentication through openID. By default, openQA is configured to use the openSUSE openID provider, but it can very easily be configured to use any other valid provider. Every time a new user logs into an instance, a new user profile is created. That profile only contains the openID identity and two flags used for access control:
-
operator Means that the user is able to manage jobs, performing actions like creating new jobs, cancelling them, etc.
-
admin Means that the user is able to manage users (granting or revoking operator and admin rights) as well as job templates and other related information (see the the corresponding section).
Many of the operations in an openQA instance are not performed through the web interface but using the REST-like API. The most obvious examples are the workers and the scripts that fetch new versions of the operating system and schedule the corresponding tests. Those clients must be authorized by an operator using an API key with an associated shared secret.
For that purpose, users with the operator flag have access in the web interface to a page that allows them to manage as many API keys as they may need. For every key, a secret is automatically generated. The user can then configure the workers or any other client application to use whatever pair of API key and secret owned by him. Any client to the REST-like API using one of those API keys will be considered to be acting on behalf of the associated user. So the API key not only has to be correct and valid (not expired), it also has to belong to a user with operator rights.
For more insights about authentication, authorization and the technical details of the openQA security model, refer to the detailed blog post about the subject by the openQA development team.
A job can belong to a job group. Those job groups are displayed on the index
page when there are recent test results in these job groups and in the Job
Groups menu on the navigation bar. From there the job group overview pages
can be accessed. Besides the test results the job group overview pages provide
a description about the job group and allow commenting.
Job groups have properties. These properties are mostly cleanup related. The configuration can be done in the operators menu for job groups.
It is also possible to put job groups into categories. The nested groups will then inherit properties from the category. The categories are meant to combine job groups with common builds so test results for the same build can be shown together on the index page.
|
Important
|
openQA automatically deletes data that it considers "old" based on different settings. For example old jobs and assets are deleted at some point. |
The following cleanup settings can be done on job-group-level:
| size limit |
Limits size of assets |
| keep logs for |
Specifies how long logs of a non-important job are retained after it finished |
| keep important logs for |
How long logs of an important job are retained after it finished |
| keep results for |
specifies How long results of a non-important job are retained after it finished |
| keep important results for |
How long results of an important job are retained after it finished |
NOTE Deletion of job results includes deletion of logs and will cause the job to be completely removed from the database.
NOTE Checkout the Cleanup section for more details and the Build tagging section for how to mark a job as important.
NOTE New groups use the limits configured in the [default_group_limits] section
of /etc/openqa/openqa.ini. Jobs outside of any group use the limits configured in
the [no_group_limits] section of /etc/openqa/openqa.ini.
NOTE Archiving of important jobs can be enabled. Checkout the related settings
within the [archiving] section of the config file for details.
Just as the worker uses an API key+secret every user of the client script
must do the same. The same API key+secret as previously created can be used or
a new one created over the webUI.
The personal configuration should be stored in a file
~/.config/openqa/client.conf in the same format as previously described for
the client.conf, i.e. sections for each machine, e.g. localhost.
An easy way to start using openQA is to start testing openSUSE or Fedora as they have everything setup and prepared to ease the initial deployment. If you want to play deeper, you can configure the whole openQA manually from scratch, but this document should help you to get started faster.
You can point CASEDIR and NEEDLES_DIR to Git repositories. openQA will
checkout those repositories automatically and no manual setup is needed.
Otherwise you will need to clone tests and needles manually. First you need to
get actual tests. You can get openSUSE tests and needles (the expected results)
from GitHub. It
belongs into the /var/lib/openqa/tests/opensuse directory. To make it easier,
you can just run:
/usr/share/openqa/script/fetchneedlesWhich will download the tests to the correct location and will set the correct rights as well.
Fedora’s tests are also in git. To use them, you may do:
cd /var/lib/openqa/share/tests
mkdir fedora
cd fedora
git clone https://pagure.io/fedora-qa/os-autoinst-distri-fedora.git
./templates --clean
cd ..
chown -R geekotest fedora/To get everything configured to actually run the tests, there are plenty of options to set in the admin interface. If you plan to test openSUSE Factory, using tests mentioned in the previous section, the easiest way to get started is the following command:
/var/lib/openqa/share/tests/opensuse/products/opensuse/templates [--apikey API_KEY] [--apisecret API_SECRET]This will load some default settings that were used at some point of time in
openSUSE production openQA. Therefore those should work reasonably well with
openSUSE tests and needles. This script uses /usr/share/openqa/script/openqa-load-templates,
consider reading its help page (--help) for documentation on possible extra arguments.
For Fedora, similarly, you can call:
/var/lib/openqa/share/tests/fedora/templates [--apikey API_KEY] [--apisecret API_SECRET]Some Fedora tests require special hard disk images to be present in
/var/lib/openqa/share/factory/hdd/fixed. The createhdds.py script in the
createhdds
repository can be used to create these. See the documentation in that repo
for more information.
To start testing a new ISO put it in /var/lib/openqa/share/factory/iso and call
the following commands:
# Run the first test
openqa-cli api -X POST isos \
ISO=openSUSE-Factory-NET-x86_64-Build0053-Media.iso \
DISTRI=opensuse \
VERSION=Factory \
FLAVOR=NET \
ARCH=x86_64 \
BUILD=0053If your openQA is not running on port 80 on 'localhost', you can add option
--host=http://otherhost:9526 to specify a different port or host.
|
Warning
|
Use only the ISO filename in the 'client' command. You must place the
file in /var/lib/openqa/share/factory/iso. You cannot place the file elsewhere and
specify its path in the command. However, openQA also supports a
remote-download feature of assets from trusted domains.
|
For Fedora, a sample run might be:
# Run the first test
openqa-cli api -X POST isos \
ISO=Fedora-Everything-boot-x86_64-Rawhide-20160308.n.0.iso \
DISTRI=fedora \
VERSION=Rawhide \
FLAVOR=Everything-boot-iso \
ARCH=x86_64 \
BUILD=Rawhide-20160308.n.0More details on triggering tests can also be found in the Users Guide.
Take a look at Documented Pitfalls.