sktm is an orchestrator for skt keeping track of known baselines and tested patches.
NOTE: The instructions below are intended for Fedora Server 28 and will work with minimal install.
Sktm uses Jenkins to queue and execute jobs. You can install and start latest Jenkins like this:
sudo curl -o /etc/yum.repos.d/jenkins.repo \
https://pkg.jenkins.io/redhat/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo dnf install -y java jenkins
sudo systemctl enable jenkins
Latest versions of Fedora have Java versions with tightened-down encryption algorithm requirements, which Jenkins code and infrastructure do not yet meet. Until those are updated, you will need to relax the requirements for Jenkins:
sudo tee /var/lib/jenkins/java.security >/dev/null <<"EOF"
jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
RSA keySize < 1023, DSA keySize < 1024, EC keySize < 224
EOF
sudo sed -i /etc/sysconfig/jenkins \
-e '/^JENKINS_JAVA_OPTIONS=/ s%"$% -Djava.security.properties=/var/lib/jenkins/java.security"%'
Start Jenkins:
sudo systemctl start jenkins
From now on the documentation assumes Jenkins is serving HTTP requests on the default URL of http://localhost:8080/.
Make sure you have Python 2 and git installed to be able to run sktm:
sudo dnf install -y python2 git
Install sktm requirements:
sudo pip2 install dateutils enum34 jenkinsapi requests
Install Jenkins Job Builder to be able to automate Jenkins setup:
sudo pip2 install jenkins-job-builder
Install Python's virtualenv to be able to run skt within Jenkins jobs:
sudo pip2 install virtualenv
Install and start sendmail to allow skt to send report messages:
sudo dnf install -y sendmail
sudo systemctl enable --now sendmail
Install Kerberos client to enable skt authentication to Beaker:
sudo dnf install -y krb5-workstation
Install skt dependencies following its
README.md.
To run all tests execute:
python -m pytest -v -s tests
To run some specific tests, you can execute a specific test like this:
python -m pytest -v -s tests/test_patchwork.py
pip install git+https://github.com/RH-FMK/sktm
You will need access to a Beaker instance configured for the jenkins
user. First, create the Beaker client's configuration directory:
sudo -u jenkins mkdir -p ~jenkins/.beaker_client
Then you can setup Kerberos authentication to Beaker like this:
sudo -u jenkins tee ~jenkins/.beaker_client/config >/dev/null <<"EOF"
HUB_URL = "<BEAKER_URL>"
AUTH_METHOD = "krbv"
KRB_REALM = "<KERBEROS_REALM>"
EOF
Where <BEAKER_URL> would be the Beaker service URL, and <KERBEROS_REALM>
would be your Kerberos realm. E.g.:
sudo -u jenkins tee ~jenkins/.beaker_client/config >/dev/null <<"EOF"
HUB_URL = "https://beaker.example.com"
AUTH_METHOD = "krbv"
KRB_REALM = "EXAMPLE.COM"
EOF
If your Beaker instance is accessed via HTTPS you might need to supply the CA certificate as well, by adding a line like this to the above configuration:
CA_CERT = "<CA_CERT>"
Where <CA_CERT> would be a path to a CA certificate file in PEM format. E.g.
sudo -u jenkins tee ~jenkins/.beaker_client/config >/dev/null <<"EOF"
HUB_URL = "https://beaker.example.com"
AUTH_METHOD = "krbv"
KRB_REALM = "EXAMPLE.COM"
CA_CERT = "/etc/beaker/ca.pem"
EOF
If you have a keytab for a dedicated principal to use for accessing Beaker, you can supply it by adding these two lines to the configuration file:
KRB_KEYTAB = "<KEYTAB>"
KRB_PRINCIPAL = "<JENKINS_PRINCIPAL>"
Here <KEYTAB> would be a path to the keytab file, and <JENKINS_PRINCIPAL>
would be the Kerberos principal for Jenkins to use. For example (together with
the CA certificate):
sudo -u jenkins tee ~jenkins/.beaker_client/config >/dev/null <<"EOF"
HUB_URL = "https://beaker.example.com"
AUTH_METHOD = "krbv"
KRB_REALM = "EXAMPLE.COM"
CA_CERT = "/etc/beaker/ca.pem"
KRB_KEYTAB = "/etc/beaker/keytab"
KRB_PRINCIPAL = "jenkins/special-principals.example.com"
EOF
Make sure to take steps to protect the keytab file in production environments.
If you don't have a dedicated principal and a keytab, you can omit
KRB_KEYTAB and KRB_PRINCIPAL settings, and let Jenkins use your Kerberos
credentials like this:
sudo -u jenkins kinit <YOUR_PRINCIPAL>
Where <YOUR_PRINCIPAL> would be the principal you usually login with. E.g.:
sudo -u jenkins kinit [email protected]
Note that in this case you will need to periodically refresh the credentials Jenkins has as they expire, by re-running the command above.
Make sure you get a dedicated principal and a keytab in production environment, instead of letting Jenkins use your credentials.
To configure Jenkins, open http://localhost:8080 in your browser and follow the wizard. Installing the suggested plugins should be sufficient. The examples below assume that the admin user, created during Jenkins configuration, was named "sktm", and assigned password "sesame".
If the Jenkins web-interface fails to open and browser reports connection was rejected, check if Jenkins is running. On Fedora 28 it fails to start due to crashing Java interpreter. In that case restart the service with
sudo systemctl restart jenkins
and try again.
At the moment, sktm doesn't support CSRF protection when authenticating to Jenkins. So, after the Jenkins setup is complete, navigate to Manage Jenkins -> Configure Global Security and uncheck the "Prevent Cross Site Request Forgery exploits" checkbox. Note that you shouldn't do this on production, or exposed systems.
Create a configuration file for Jenkins Job Builder, describing how to access Jenkins:
mkdir -p ~/.config/jenkins_jobs
cat >~/.config/jenkins_jobs/jenkins_jobs.ini <<"EOF"
[jenkins]
url=http://localhost:8080
user=sktm
password=
EOF
Then go to "sktm" user preferences in
Jenkins, click "Show API
token...", copy the text from the displayed "API Token" field, and paste it
into the password field value of the configuration file created above. Note:
secure your configuration file appropriately when configuring production
systems.
Test that Jenkins Job Builder can reach and authenticate to Jenkins by running
jenkins-jobs list. It should output something like INFO:root:Matching jobs: 0 and complete succesfully.
The Jenkins project description (example-project.yaml) used in this documentation will run an HTTP server on port 4040 to let Beaker download the tested kernel images. If your Jenkins host is behind NAT (e.g. in a VM), make sure to setup port redirection, to have it accessible both from the Beaker instance and from inside the host, at the same address, and change the URL skt will be publishing with:
sed -i -e 's/`hostname -f`:4040/<HOSTNAME>:<PORT>/' example-project.yaml
Here, <HOSTNAME> would be the name of the NAT host Beaker would need to
access, and <PORT> would be the redirected port number. E.g.:
sed -i -e 's/`hostname -f`:4040/pc.example.com:4040/' example-project.yaml
You will need to update the file again, if your NAT host name changes.
The example-project.yaml file will be instructing
skt to send test results to Root <[email protected]>. Change the
file if you'd like to send the reports somewhere else:
sed -i -e 's/Root <root@localhost\.localdomain>/<EMAIL_ADDRESS>/' \
example-project.yaml
Here, <EMAIL_ADDRESS> would be the address to send reports to. E.g.:
sed -i -e 's/Root <root@localhost\.localdomain>/User <[email protected]>/' \
example-project.yaml
Create the Jenkins project for sktm to trigger, from the
example-project.yaml file, using Jenkins Job
Builder:
jenkins-jobs update example-project.yaml
Then go to the created project in Jenkins UI and click "Build Now" to check that it works, and to have the pipeline parameters (which sktm requires) registered by Jenkins.
Afterwards, every time you would change your YAML project description you will
need to re-run jenkins-jobs update example-project.yaml and build the
project manually once, as described above.
Create the ~/.sktmrc file telling sktm how to access Jenkins:
cat >~/.sktmrc <<"EOF"
[config]
jurl = http://localhost:8080
jlogin = sktm
jpass = sesame
jretry = 30
EOF
On the first execution sktm creates a database file (~/.sktm.db by default),
which is then used to track status of tested kernel branches, patches, and
the execution of Jenkins jobs sktm submits.
Before beginning testing a kernel branch, you need to establish a working "baseline" commit, the patches would be applied on:
sktm -v --jjname <JENKINS_PROJECT> baseline <GIT_REPO_URL> <GIT_REF>
Here, <JENKINS_PROJECT> would be the name of the Jenkins project running
skt, <GIT_REPO_URL> would be a kernel Git repository URL, and <GIT_REF>
would be a published Git reference in that repository from which to start
locating a stable "baseline" commit, and usually will be a branch name.
E.g., following our setup example above, this command would find a baseline commit in the current "scsi" tree:
sktm -v --jjname sktm baseline \
git://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git \
for-next
And this would do the same for the "net-next" tree:
sktm -v --jjname sktm baseline \
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git \
master
You will need to run the "baseline" command periodically to have your baseline commits up-to-date, to allow newer patches to apply.
Once your baseline command has finished, you will be able to start checking
patches from a Patchwork instance. Here are very similar examples of commands
to be used with the Patchwork 1 and Patchwork 2 instances:
For Patchwork v1 run:
sktm -v --jjname <JENKINS_PROJECT> patchwork \
<GIT_REPO_URL> \
<PATCHWORK_BASE_URL> <PATCHWORK_PROJECT> \
--lastpatch <PATCHWORK_PATCH_ID> \
--skip <PATTERN> [... <PATTERN>]
and for Patchwork v2 run:
sktm -v --jjname <JENKINS_PROJECT> patchwork \
<GIT_REPO_URL> \
--restapi <PATCHWORK_BASE_URL> <PATCHWORK_PROJECT> \
--lastpatch <PATCHWORK_PATCH_ID> \
--skip <PATTERN> [... <PATTERN>]
Here, <PATCHWORK_BASE_URL> would be the base URL of the Patchwork instance,
<PATCHWORK_PROJECT> - the name of the Patchwork project to check for new
patches. The <PATCHWORK_PATCH_ID> would be the ID of the newest patch (as
seen in Patchwork URLs) to ignore. All newer patches (with greater IDs) will be
considered for testing. Finally, a list of regex <PATTERN>s (case
insensitive) can be provided to skip testing of patches which match the
patterns. This last option is useful for example if the Patchwork project
contains patches for additional tools besides kernel ones. By default, git pull
requests and tools from netdev list (iproute, ethtool etc.) are skipped.
E.g. this command would test all patches after the one with ID 10363835, from the "scsi" tree's Patchwork v1 instance, applying them onto the latest baseline commit from the corresponding Git repo:
sktm -v --jjname sktm patchwork \
git://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git \
https://patchwork.kernel.org linux-scsi \
--lastpatch 10363835
And this one would test all patches received by the "next-next" tree's
Patchwork v2 instance after Thu, 3 May 2018 14:35:00 +0100 timestamp:
sktm -v --jjname sktm patchwork \
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git \
--restapi https://patchwork.ozlabs.org netdev \
--lastpatch 823457
Note: do not run the commands above with the --lastpatch option value
intact, as that would likely result in a lot of Jenkins jobs submitted,
because development has moved on since the writing of this guide, and many
more patches have been recorded by these Patchwork instances. Look at the
patches in the instances and pick your own values.
After this initial run is complete, the sktm database will have the last
tested patch recorded, and further patches could be tested by executing the
"patchwork" commands above without the --lastpatch option.
However, it can be used again to push the last tested patch back, and retest already-tested patches, or to push it forward to skip testing some patches.
In case database schema changes, new migration scripts will be provided in
db_migrations directory. They aren't needed for new checkouts, but are
required for sktm to work correctly when upgrading. New scripts since the last
upgrade should be applied in the correct (numerical) order with commands:
sqlite3 <db_path> < <script_name>
For example, if the database path is .sktm.db and migration 01-pending.sql
is being applied, the command will be
sqlite3 ~/.sktm.db < 01-pending.sql
Developers can test changes to sktm by using "development mode" from python's
setuptools package. First, cd to the directory where sktm is cloned and
run:
pip install --user -e .
This installs sktm in a mode where any changes within the repo are
immediately available simply by running sktm. There is no need to repeatedly
run pip install . after each change.
Using a virtual environment is highly recommended. This keeps sktm and all
its dependencies in a separate Python environment. Developers can build a
virtual environment for sktm quickly:
virtualenv ~/sktm-venv/
source ~/sktm-venv/bin/activate
pip install -e .
To deactivate the virtual environment, simply run deactivate.
sktm is distributed under GPLv2 license.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.