Using avocado for parallel linter tests

[pevogam]

Hi all, we have a rather complex bash script that would perform a number of linter steps in each file in a code base in serial order. Considering avocado's versatility in interpreting any kind of binaries as potential tests I was wondering if there is a way to achieve parallelism and proper debug logs for linting each separate file. My idea was to have a bash script like

pylint $1
black $1
mypy $1

which should receive a filename argument and would correspond to a separate test for each such possible argument. The question then remains could we easily generate the tests with resolving something like an expansion linted_files/*. Has someone tried something like this?

[pevogam]

I guess this is a good starter

$ cat lintme.sh 
#!/bin/sh
if [[ -z ${CHECK_FILE} ]]; then
  CHECK_FILE="aaa"
fi

echo "Linting $CHECK_FILE"
pylint $CHECK_FILE
$ cat variants.json 
[
{"paths": ["/run/*"], "variant_id": "run-file1", "variant": [["/run/f1", [["/run/f1", "check_file", "abc"]]]]},
{"paths": ["/run/*"], "variant_id": "run-file2", "variant": [["/run/f2", [["/run/f2",  "check_file", "def"]]]]}
]
[root@c2 intra2net-qa]# cat lintme.sh 
#!/bin/sh
if [[ -z ${CHECK_FILE} ]]; then
  CHECK_FILE="aaa"
fi

echo "Linting $CHECK_FILE"
pylint $CHECK_FILE

then running with

$ avocado run lintme.sh --json-variants-load variants.json
JOB ID     : 4cbeb8f1e2676c1a6e0c88400038f50c51ac0084
JOB LOG    : /mnt/local/results/job-2024-12-18T15.08-4cbeb8f/job.log
 (1/2) lintme.sh;run-file1: STARTED
 (2/2) lintme.sh;run-file2: STARTED
 (2/2) lintme.sh;run-file2: FAIL (0.33 s)
 (1/2) lintme.sh;run-file1: FAIL (0.34 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 2 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /mnt/local/results/job-2024-12-18T15.08-4cbeb8f/results.html
JOB TIME   : 1.77 s

Test summary:
1-lintme.sh;run-file1: FAIL
2-lintme.sh;run-file2: FAIL

would have almost worked but it seems avocado will not convert the parameters passed by the varianter to env variables. Does the varianter only work with python scripts? Considering tests should have been agnostic to programming languages I would have expected at least an implicit conversion of the varianter params into env variables to be used by the bash script.

[pevogam]

So to answer my own initial inquiry, I made it work using a class to inherit the avocado test:

import subprocess

from avocado import Test
from avocado.core import exceptions


class LintTest(Test):
    """
    This test invokes all needed linting for a file.

    :param check_file: file to check with all required linters
    """

    def test(self):
        """Perform the actual test."""
        self.log.debug([p for p in self.params.iteritems()])

        check_file = self.params.get("check_file")
        if check_file is None:
            raise ValueError("Missing file to check by all linters")

        self.log.debug("Linting file %s", check_file)
        result = subprocess.run(
            f"pylint {check_file}",
            stdout=subprocess.PIPE,
            #stdout=self.log.info,
            #stderr=self.log.error,
            shell=True,
        )
        self.log.debug(result.stdout.decode())
        if result.returncode != 0:
            self.log.error(f"Got exit status {result.returncode}")
            raise exceptions.TestFail(f"Lint of {check_file} didn't pass")

and a variant generator from file names like

import os
import json
import sys

json_data = []
dirname = sys.argv[1]
files = os.listdir(dirname)
for i, filename in enumerate(files):
    if not filename.endswith(".py"):
        continue
    variant = filename[:-3]
    json_data.append(
        {
            "paths": ["/run/*"],
            "variant_id": f"lint-{variant}",
            "variant": [[f"/run/f{i}", [
                [f"/run/f{i}", "check_file", os.path.join(dirname, filename)]
            ]]]},
    )

with open("lint.json", "w") as f:
    json.dump(json_data, f, indent=4)

Profit!