103 LTS Testing with external projects and test suites #5766
Comments
|
Hi @richtja I would love to test but would prefer to do so with the remote spawner branch merged (meaning I have to adapt all points noted by @clebergnu but then in order to do so I need #5649 to be closed as explained in the comments there. Just letting you know about the state of my testing. |
|
Hi @pevogam, thank you very much that you plan to participate in testing again. About the #5649 AFAIK @clebergnu put a priority on that one and we still plan to have remote spawner branch merged before LTS. We still have some issues in milestone 103 and we will start testing when those are addressed, so there is still a time. I opened this issue now, so that we won't forget about testing those projects before release. |
|
In fact thank you @richtja for opening this issue so that I can also subscribe to it and track the overall progress on the LTS release from early on. |
clebergnu
moved this from Short Term (Current Q) Backlog
to In progress
in Default project
|
FYI, I've started the testing for QEMU. |
|
I am also already using most recent master (but perhaps 1-2 weeks old at the time of writing this, updated every once and a while) in all of my test runs in order to test the release on the side. |
|
Out of testing QEMU it was found: #5801 |
|
Hi @clebergnu I tested the most recent source over the weekend and the HTML report (which has some nice new colors for the test statuses) doesn't seem to filter out tests by status any longer (tests are only shown with the ALL status filter). Should I report an issue about this or are you already aware of this in an issue I can track? |
Hi @pevogam, please open an issue, we are not aware of that. Thank you. |
Alright, #5828. |
|
Marking QEMU as done, based on the testing such as https://gitlab.com/cleber.gnu/qemu/-/commit/bee33ecd33d1c6d03d29e2577009ea6d71808ed2 |
|
Taking on SOS testing. I'm covering two types of tests are run with Avocado:
|
|
Marking SOS as done, with the following notes. There's some API changes on diff --git a/tests/collect_tests/help_output_tests.py b/tests/collect_tests/help_output_tests.py
index 258bb8d7..2d74a78e 100644
--- a/tests/collect_tests/help_output_tests.py
+++ b/tests/collect_tests/help_output_tests.py
@@ -10,13 +10,10 @@ import glob
import os
import re
-from avocado.utils import software_manager
+from avocado.utils.software_manager.manager import SoftwareManager
from sos_tests import StageOneOutputTest, SOS_REPO_ROOT, skipIf
-installer = software_manager
-sm = installer.SoftwareManager()
-
-PEXPECT_PRESENT = sm.check_installed('python3-pexpect')
+PEXPECT_PRESENT = SoftwareManager().check_installed('python3-pexpect')
class CollectHelpOutputTest(StageOneOutputTest):
diff --git a/tests/sos_tests.py b/tests/sos_tests.py
index d15395f0..398da4bd 100644
--- a/tests/sos_tests.py
+++ b/tests/sos_tests.py
@@ -10,8 +10,9 @@
from avocado.core.exceptions import TestSkipError
from avocado.core.output import LOG_UI
from avocado import Test
-from avocado.utils import archive, process, distro, software_manager
+from avocado.utils import archive, process, distro
from avocado.utils.cpu import get_arch
+from avocado.utils.software_manager import manager, distro_packages
from fnmatch import fnmatch
import glob
@@ -771,8 +772,7 @@ class StageTwoReportTest(BaseSoSReportTest):
self.end_of_test_case = False
# seems awkward, but check_installed() and remove() are not exposed
# together with install_distro_packages()
- self.installer = software_manager
- self.sm = self.installer.SoftwareManager()
+ self.sm = manager.SoftwareManager()
for dist in self.packages:
if isinstance(self.packages[dist], str):
@@ -842,7 +842,7 @@ class StageTwoReportTest(BaseSoSReportTest):
self._strip_installed_packages()
if not self.packages[self.local_distro]:
return
- installed = self.installer.install_distro_packages(self.packages)
+ installed = distro_packages.install_distro_packages(self.packages)
if not installed:
raise Exception(
"Unable to install requested packages %s"The Other than that, everything seems to be working great. |
|
Taking on testing of Avocado-VT. |
|
@clebergnu Would you mind pinging me when merging the remote spawner branch so I can do one final test spin with everything included and send my final confirmation here? |
|
About Avocado-VT: CI builds have been running in Cirrus with the latest development Avocado. The latest run can be seen at: https://cirrus-ci.com/build/4549218879143936 . All the "avocado_devel" jobs use the latest Avocado development version. On this build, it can be seen: The QEMU based jobs run actual tp-qemu tests, and not just dry-runs. Example of the results: I'll follow up with some extended testing, but so far, overall compatibility seems to be covered. |
As our plugin is also using VT tests and most of the Avocado VT infrastructure under the hood, I can confirm this - so far all VT tests have been passing just fine for us too. Most importantly, our test suite contains thousands of VT tests with various needs when it comes to both Avocado and Avocado VT. |
|
I've picked a few extra Avocado-VT tests to run locally: The error is really caused by an outdated use of the I'm marking Avocado-VT compatibility check as complete, based on @pevogam, CI and my own testing. |
|
Alright, will do overnight testing with the lastest master branch but I anticipate things to go smoothly since there aren't that many changes since the last time we included a recent upstream source. |
|
Marking POK testing as done. Output of testing follows: |
|
Sorry for some delay, the overnight testing went fine. Here is a summary from a minimal run: Whoever is capable can cross of "Avocado I2N" from the checklist above. |
|
I've run a number of avocado-misc-tests, and found no issues with APIs or general behavior of Avocado. The avocado-misc-tests team do a pretty good job of keeping the code current with the latest avocado. |
|
For DAOS, I've tried my best, but the environment needed to get any of the tests running simply exceeded what I was capable of setting up. I've reached out to the DAOS team and asked them to get back to us when they feel like an Avocado version bump is due. |
|
Falco dropped all Python based tests, migrating to a different repo, see falcosecurity/falco#2716. Marking it off from the list. |
|
LibVirt(TCK) usage of Avocado is limited to running the tests with the TAP runner. Tested a few tests such as: |
github-project-automation
bot
moved this from In progress
to Done 103
in Default project
We know about a number of projects with public tests using Avocado. To validate the migration procedures from the Avocado version they are using to 103.0 LTS, we can try to run those with the (to be) 103.0 LTS.
Examples of projects are here: https://avocado-framework.readthedocs.io/en/91.0/misc/index.html
IIRC, for 92.0 LTS, we sent early announcements to those project's mailing lists, and also ran tests for QEMU, DAOS, Falco, Avocado-VT and Avocado-I2N (with @pevogam help) Libvirt, SoS, POK, Avocado-misc-tests.
The text was updated successfully, but these errors were encountered: