useless wheel files in plugin !?

[matthieumarrast]

I recently migrated from metwork v2.1 to v2.2. So I rebuilt only the plugin with embedded python packages (local/lib/python3.X/site-packages) because of the python upgrade from v3.10 to v3.11. It seems useless to me to rebuild the others plugins.

But into my plugin built in v2.1, in directory python3_virtualenv_sources\src\ I found the wheel file:

• certifi-2022.12.7-py3-none-any.whl

I do not understand why the wheel is embedded in the plugin whereas:

• it is not mentionned in my requirements-to-freeze.txt
• certifi is provided by metwork
• it is not installed in the local/lib/python3.X/site-packages of the plugin

What's the purpose of this wheel file into the plugin ?

So my plugin was working on metwork v2.1 and still working on v2.2 without any modification.

But if I install the plugin on metwork v2.2, I face the error caused by the useless wheel (I suppose):

The conflict is caused by:
    The user requested certifi==2022.12.7
    The user requested (constraint) certifi==2023.7.22

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
===== BASH STDERR ========================
ERROR: Cannot install certifi==2022.12.7 because these package versions have conflicting dependencies.

[thebaptiste]

Give me the content of your requirements-to-freeze.txt and of your requirements3.txt (if any)
Which module ?

[matthieumarrast]

my requirements-to-freeze.txt:

coverage

I think because of this above line the wheel files (dependencies of coverage) are collected.
note: no flag "override" present in my python3_virtualenv_sources directory

[matthieumarrast]

I tried to build a plugin with "coverage" in the requirements-to-freeze.txt file, here is the log:

make develop
- Checking plugin...                                       [ OK ]              
[DEBUG]: loading tempolayer104698[/home/mfdata/foo/python3_virtualenv_sources/tempolayer104698]
Requirement already satisfied: coverage in /opt/metwork-mfext-2.2/opt/python3_devtools/lib/python3.11/site-packages (from -r requirements-to-freeze.txt.tmp (line 3)) (7.3.2)
[DEBUG]: loading tempolayer104747[/home/mfdata/foo/python3_virtualenv_sources/tempolayer104747]
Collecting calver==2022.6.26 (from -r requirements3.txt (line 1))
  Using cached calver-2022.6.26-py3-none-any.whl.metadata (2.0 kB)
Collecting certifi==2023.7.22 (from -r requirements3.txt (line 2))
  Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting distlib==0.3.7 (from -r requirements3.txt (line 3))
  Using cached distlib-0.3.7-py2.py3-none-any.whl.metadata (5.1 kB)
Collecting distro==1.8.0 (from -r requirements3.txt (line 4))
  Using cached distro-1.8.0-py3-none-any.whl.metadata (6.9 kB)
Collecting editables==0.5 (from -r requirements3.txt (line 5))
  Using cached editables-0.5-py3-none-any.whl.metadata (3.1 kB)
Collecting filelock==3.12.3 (from -r requirements3.txt (line 6))
  Using cached filelock-3.12.3-py3-none-any.whl.metadata (2.7 kB)
Collecting flit_core==3.9.0 (from -r requirements3.txt (line 7))
  Using cached flit_core-3.9.0-py3-none-any.whl.metadata (822 bytes)
Collecting hatch-vcs==0.3.0 (from -r requirements3.txt (line 8))
  Using cached hatch_vcs-0.3.0-py3-none-any.whl.metadata (7.3 kB)
Collecting hatchling==1.18.0 (from -r requirements3.txt (line 9))
  Using cached hatchling-1.18.0-py3-none-any.whl.metadata (3.8 kB)
Collecting importlib_metadata==6.8.0 (from -r requirements3.txt (line 10))
  Using cached importlib_metadata-6.8.0-py3-none-any.whl.metadata (5.1 kB)
Collecting pathspec==0.11.2 (from -r requirements3.txt (line 11))
  Using cached pathspec-0.11.2-py3-none-any.whl.metadata (19 kB)
Collecting platformdirs==3.10.0 (from -r requirements3.txt (line 12))
  Using cached platformdirs-3.10.0-py3-none-any.whl.metadata (11 kB)
Collecting pluggy==1.3.0 (from -r requirements3.txt (line 13))
  Using cached pluggy-1.3.0-py3-none-any.whl.metadata (4.3 kB)
Collecting setuptools_scm==7.1.0 (from -r requirements3.txt (line 14))
  Using cached setuptools_scm-7.1.0-py3-none-any.whl.metadata (24 kB)
Collecting toml==0.10.2 (from -r requirements3.txt (line 15))
  Using cached toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB)
Collecting tomli==2.0.1 (from -r requirements3.txt (line 16))
  Using cached tomli-2.0.1-py3-none-any.whl.metadata (8.9 kB)
Collecting trove-classifiers==2023.8.7 (from -r requirements3.txt (line 17))
  Using cached trove_classifiers-2023.8.7-py3-none-any.whl.metadata (2.2 kB)
Collecting typing_extensions==4.8.0 (from -r requirements3.txt (line 18))
  Using cached typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB)
Collecting zipp==3.16.2 (from -r requirements3.txt (line 19))
  Using cached zipp-3.16.2-py3-none-any.whl.metadata (3.7 kB)
Collecting packaging>=21.3 (from hatchling==1.18.0->-r requirements3.txt (line 9))
  Using cached packaging-23.1-py3-none-any.whl.metadata (3.1 kB)
Collecting setuptools (from setuptools_scm==7.1.0->-r requirements3.txt (line 14))
  Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB)
Using cached calver-2022.6.26-py3-none-any.whl (7.0 kB)
Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Using cached distlib-0.3.7-py2.py3-none-any.whl (468 kB)
Using cached distro-1.8.0-py3-none-any.whl (20 kB)
Using cached editables-0.5-py3-none-any.whl (5.1 kB)
Using cached filelock-3.12.3-py3-none-any.whl (11 kB)
Using cached flit_core-3.9.0-py3-none-any.whl (63 kB)
Using cached hatch_vcs-0.3.0-py3-none-any.whl (8.0 kB)
Using cached hatchling-1.18.0-py3-none-any.whl (75 kB)
Using cached importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Using cached pathspec-0.11.2-py3-none-any.whl (29 kB)
Using cached platformdirs-3.10.0-py3-none-any.whl (17 kB)
Using cached pluggy-1.3.0-py3-none-any.whl (18 kB)
Using cached setuptools_scm-7.1.0-py3-none-any.whl (43 kB)
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Using cached trove_classifiers-2023.8.7-py3-none-any.whl (13 kB)
Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Using cached zipp-3.16.2-py3-none-any.whl (7.2 kB)
Using cached packaging-23.1-py3-none-any.whl (48 kB)
Using cached setuptools-69.0.3-py3-none-any.whl (819 kB)
Saved ./src/calver-2022.6.26-py3-none-any.whl
Saved ./src/certifi-2023.7.22-py3-none-any.whl
Saved ./src/distlib-0.3.7-py2.py3-none-any.whl
Saved ./src/distro-1.8.0-py3-none-any.whl
Saved ./src/editables-0.5-py3-none-any.whl
Saved ./src/filelock-3.12.3-py3-none-any.whl
Saved ./src/flit_core-3.9.0-py3-none-any.whl
Saved ./src/hatch_vcs-0.3.0-py3-none-any.whl
Saved ./src/hatchling-1.18.0-py3-none-any.whl
Saved ./src/importlib_metadata-6.8.0-py3-none-any.whl
Saved ./src/pathspec-0.11.2-py3-none-any.whl
Saved ./src/platformdirs-3.10.0-py3-none-any.whl
Saved ./src/pluggy-1.3.0-py3-none-any.whl
Saved ./src/setuptools_scm-7.1.0-py3-none-any.whl
Saved ./src/toml-0.10.2-py2.py3-none-any.whl
Saved ./src/tomli-2.0.1-py3-none-any.whl
Saved ./src/trove_classifiers-2023.8.7-py3-none-any.whl
Saved ./src/typing_extensions-4.8.0-py3-none-any.whl
Saved ./src/zipp-3.16.2-py3-none-any.whl
Saved ./src/packaging-23.1-py3-none-any.whl
Saved ./src/setuptools-69.0.3-py3-none-any.whl
[DEBUG]: loading local@mfdata[/home/mfdata/foo/local]
Looking in links: /home/mfdata/foo/python3_virtualenv_sources/src
Requirement already satisfied: calver==2022.6.26 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 1)) (2022.6.26)
Requirement already satisfied: certifi==2023.7.22 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 2)) (2023.7.22)
Requirement already satisfied: distlib==0.3.7 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 3)) (0.3.7)
Requirement already satisfied: distro==1.8.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 4)) (1.8.0)
Requirement already satisfied: editables==0.5 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 5)) (0.5)
Requirement already satisfied: filelock==3.12.3 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 6)) (3.12.3)
Requirement already satisfied: flit_core==3.9.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 7)) (3.9.0)
Requirement already satisfied: hatch-vcs==0.3.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 8)) (0.3.0)
Requirement already satisfied: hatchling==1.18.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 9)) (1.18.0)
Requirement already satisfied: importlib_metadata==6.8.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 10)) (6.8.0)
Requirement already satisfied: pathspec==0.11.2 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 11)) (0.11.2)
Requirement already satisfied: platformdirs==3.10.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 12)) (3.10.0)
Requirement already satisfied: pluggy==1.3.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 13)) (1.3.0)
Requirement already satisfied: setuptools_scm==7.1.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 14)) (7.1.0)
Requirement already satisfied: toml==0.10.2 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 15)) (0.10.2)
Requirement already satisfied: tomli==2.0.1 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 16)) (2.0.1)
Requirement already satisfied: trove-classifiers==2023.8.7 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 17)) (2023.8.7)
Requirement already satisfied: typing_extensions==4.8.0 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 18)) (4.8.0)
Requirement already satisfied: zipp==3.16.2 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from -r requirements3.txt.tmp (line 19)) (3.16.2)
Requirement already satisfied: packaging>=21.3 in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from hatchling==1.18.0->-r requirements3.txt.tmp (line 9)) (23.1)
Requirement already satisfied: setuptools in /opt/metwork-mfext-2.2/opt/python3_core/lib/python3.11/site-packages (from setuptools_scm==7.1.0->-r requirements3.txt.tmp (line 14)) (69.0.3)
- Devlinking plugin foo...                                 [ OK ]

Even if coverage is a "Requirement already satisfied [...] in /opt/metwork-mfext-2.2/opt/python3_devtools", the coverage dependencies (all are also "already satisfied") are collected into the "python3_virtualenv_sources/src/" directory of the plugin as wheel files.... but "coverage" itself is not embedded !?

ll python3_virtualenv_sources/src/
total 1896
-rw-r--r--. 1 mfdata metwork   7049 18 sept. 11:56 calver-2022.6.26-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork 158334 18 sept. 11:56 certifi-2023.7.22-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork 468869 18 sept. 11:56 distlib-0.3.7-py2.py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  20315 18 sept. 11:56 distro-1.8.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork   5098 18 sept. 11:56 editables-0.5-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  11208 18 sept. 11:56 filelock-3.12.3-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  63141 18 sept. 11:56 flit_core-3.9.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  75746 18 sept. 11:56 hatchling-1.18.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork   7989 18 sept. 11:56 hatch_vcs-0.3.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  22933 18 sept. 11:56 importlib_metadata-6.8.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  48905 18 sept. 11:56 packaging-23.1-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  29603 18 sept. 11:56 pathspec-0.11.2-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  17189 18 sept. 11:56 platformdirs-3.10.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  18934 18 sept. 11:56 pluggy-1.3.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork 819530 18 sept. 11:56 setuptools-69.0.3-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  43829 18 sept. 11:56 setuptools_scm-7.1.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  16588 18 sept. 11:56 toml-0.10.2-py2.py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  12757 18 sept. 11:56 tomli-2.0.1-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  13231 18 sept. 11:56 trove_classifiers-2023.8.7-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  31584 18 sept. 11:56 typing_extensions-4.8.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork   7203 18 sept. 11:56 zipp-3.16.2-py3-none-any.whl

Now if I build the same plugin without the layer python3_devtools, the coverage's wheel appears:

ll python3_virtualenv_sources/src/
total 2132
-rw-r--r--. 1 mfdata metwork   7049 Sep 18 10:03 calver-2022.6.26-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork 158334 Sep 18 10:03 certifi-2023.7.22-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork 238549 Sep 18 10:03 coverage-7.6.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r--. 1 mfdata metwork 468869 Sep 18 10:03 distlib-0.3.7-py2.py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  20315 Sep 18 10:03 distro-1.8.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork   5098 Sep 18 10:03 editables-0.5-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  11208 Sep 18 10:03 filelock-3.12.3-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  63141 Sep 18 10:03 flit_core-3.9.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork   7989 Sep 18 10:03 hatch_vcs-0.3.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  75746 Sep 18 10:03 hatchling-1.18.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  22933 Sep 18 10:03 importlib_metadata-6.8.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  48905 Sep 18 10:03 packaging-23.1-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  29603 Sep 18 10:03 pathspec-0.11.2-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  17189 Sep 18 10:03 platformdirs-3.10.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  18934 Sep 18 10:03 pluggy-1.3.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork 819530 Sep 18 10:03 setuptools-69.0.3-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  43829 Sep 18 10:03 setuptools_scm-7.1.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  16588 Sep 18 10:03 toml-0.10.2-py2.py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  12757 Sep 18 10:03 tomli-2.0.1-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  13231 Sep 18 10:03 trove_classifiers-2023.8.7-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork  31584 Sep 18 10:03 typing_extensions-4.8.0-py3-none-any.whl
-rw-r--r--. 1 mfdata metwork   7203 Sep 18 10:03 zipp-3.16.2-py3-none-any.whl

[thebaptiste]

1. I don't think any of the "already satisfied" requirements is theoretically necessary under python3_virtualenv_sources/src directory. Until release 1.0 (included), they were not. It's a change in pip (always downloading dependencies, even if already satisfied) which explains their presence. I don't think I can easily prevent them to be there... This mechanism is only true for the dependencies, not for the required packages in requirements-to-freeze.txt, so it's "normal" that the coverage wheel is missing...
2. But this dependencies being there, I think the plugin will install properly on a machine where some of them are missing (in case of the missing of an optional rpm), so may be it would be better if the coverage wheel was included too (in case layer python3_devtools is not installed on the target machine)... I don't know if it would be easy to change...
3. If you want coverage to be present in python3_virtualenv_sources/src, just add it in requirements3.txt after first make. You can also remove the already satisfied dependencies from this file if you want. In order to handle exactly the content of your plugin, requirements3.txt is the important file, not requirements-to-freeze.txt which will possibly lead to a different requirements3.txt file at each build if a new release of the not already satisfied dependencies is out
4. You can also add the override file which I don't know why is missing by default for mfdata (unlike mfserv). Should I add it by default ?
5. If the coverage wheel is not in python3_virtualenv_sources/src, python3_devtools@mfext should be added in layerapi2_dependencies (in a not optionally way, without -)