Merge pull request airspeed-velocity#1421 from HaoZeke/fixCompare

BUG: Fix compare and continuous

.github/workflows/ci.yml

@@ -14,7 +14,7 @@ jobs:
       fail-fast: false
       matrix:
         os: ["ubuntu-latest"]
-        python-version: ["3.7", "3.12", "pypy-3.8", "pypy-3.9"]
+        python-version: ["3.7", "3.12", "pypy-3.9"]
         r-version: ['release']
     timeout-minutes: 60
     steps:

CHANGES.rst

@@ -1,3 +1,12 @@
+0.6.4 (TBD)
+------------------
+
+Other Changes and Additions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- ``asv`` defaults to ``--force-reinstall`` now to facilitate ``compare`` and ``continuous`` for manually versioned projects (#1421)
+
+
 0.6.3 (2024-02-25)
 ------------------
 

asv/environment.py

@@ -891,7 +891,9 @@ def _install_project(self, repo, commit_hash, build_dir):
             # Run pip via python -m pip, avoids shebang length limit on Linux.
             # Don't run it in build directory, because it may contain Python packages
             # that pip believes to be already installed.
-            cmd = ["in-dir={env_dir} python -mpip install {wheel_file}"]
+            # --force-reinstall is needed since versions may not change between
+            # asv runs (esp. for compare), e.g. gh-1421
+            cmd = ["in-dir={env_dir} python -mpip install {wheel_file} --force-reinstall"]
 
         if cmd:
             commit_name = repo.get_decorated_hash(commit_hash, 8)

docs/source/using.rst

@@ -612,3 +612,49 @@ are color coded). The threshold can be set with the
 into ones that have improved, stayed the same, and worsened, using the
 same threshold using the ``--split`` option.
 See :ref:`cmd-asv-compare` for more.
+
+ASV also has a compare column which can be used to get a quick (and colorless)
+visual summary of benchmark results. This consists of a single ``mark`` where
+each of its symbolic states can be understood as:
+
+.. list-table:: ASV Change column states, ``before`` is the first commit ID, ``after`` is the second commit ID
+   :widths: 15 25 15 15 15
+   :header-rows: 1
+
+   * - Change
+     - Color
+     - Description
+     - After
+     - Before
+   * - ``x``
+     - Light Gray
+     - Not comparable
+     -
+     -
+   * - ``!``
+     - Red
+     - Introduced a failure
+     - Failed
+     - Succeeded
+   * -
+     - Green
+     - Fixed failure
+     - Succeeded
+     - Failed
+   * -
+     -
+     - Both failed or either was skipped or no significant change
+     -
+     -
+   * - ``-``
+     - Green
+     - Relative improvement
+     - Better
+     - Worse
+   * - ``+``
+     - Red
+     - Relatively worse
+     - Worse
+     - Better
+
+Additionally, statistically insignificant results have ``~`` in the ratio column as well.