Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update upper bound for galaxy packages to < 23.2 #1388

Merged
merged 4 commits into from
Dec 6, 2023

Conversation

bernt-matthias
Copy link
Contributor

@bernt-matthias bernt-matthias commented Sep 14, 2023

... waiting for the 23.1 release :)

@bernt-matthias
Copy link
Contributor Author

Test is failing reproducibly. But not sure if related.

@bgruening
Copy link
Member

The error I can see is:

E           PackagesNotFoundError: The following packages are not available from current channels:
E           
E             - fleeqtk
E           
E           Current channels:
E           
E             - https://conda.anaconda.org/conda-forge/linux-64
E             - https://conda.anaconda.org/conda-forge/noarch
E             - https://conda.anaconda.org/bioconda/linux-64
E             - https://conda.anaconda.org/bioconda/noarch
E             - https://repo.anaconda.com/pkgs/main/linux-64
E             - https://repo.anaconda.com/pkgs/main/noarch
E             - https://repo.anaconda.com/pkgs/r/linux-64
E             - https://repo.anaconda.com/pkgs/r/noarch

Which is true. This packages does not exist, its a fake package that is build locally during a test.

I think --conda_use_local is not working in this PR.

@bernt-matthias
Copy link
Contributor Author

bernt-matthias commented Nov 12, 2023

Tried to execute the commands manually and got

/tmp/tmpn84st44m/miniconda3/bin/conda build /home/berntm/projects/planemo/project_templates/conda_answers/exercise_2/fleeqtk   

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda/exceptions.py", line 1132, in __call__
        return func(*args, **kwargs)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda/cli/main.py", line 69, in main_subshell
        exit_code = do_call(args, p)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 115, in do_call
        return args.plugin_subcommand.action(sys.argv[2:])
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda_build/plugin.py", line 10, in build
        execute(*args, **kwargs)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda_build/cli/main_build.py", line 516, in execute
        _parser, args = parse_args(args)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda_build/cli/main_build.py", line 470, in parse_args
        check_recipe(args.recipe)
    AttributeError: 'Namespace' object has no attribute 'recipe'

`$ /tmp/tmpn84st44m/miniconda3/bin/conda build /home/berntm/projects/planemo/project_templates/conda_answers/exercise_2/fleeqtk`

  environment variables:
                 CIO_TEST=<not set>
    CONDA_ALLOW_SOFTLINKS=false
                CONDA_EXE=/home/berntm/miniconda3/bin/conda
         CONDA_PYTHON_EXE=/home/berntm/miniconda3/bin/python
               CONDA_ROOT=/tmp/tmpn84st44m/miniconda3
              CONDA_SHLVL=0
           CURL_CA_BUNDLE=<not set>
            DEFAULTS_PATH=/usr/share/gconf/gnome-xorg.default.path
               LD_PRELOAD=<not set>
           MANDATORY_PATH=/usr/share/gconf/gnome-xorg.mandatory.path
                     PATH=/tmp/venv-planemo/bin:/usr/local/go/bin:/home/berntm/perl5/bin:no:/hom
                          e/berntm/bin:/home/berntm/.local/bin:/home/berntm/miniconda3/condabin:
                          /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/game
                          s:/usr/local/games:/snap/bin:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=2

     active environment : None
            shell level : 0
       user config file : /home/berntm/.condarc
 populated config files : /tmp/tmpn84st44m/miniconda3/.condarc
                          /home/berntm/.condarc
          conda version : 23.3.1
    conda-build version : 3.27.0
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /tmp/tmpn84st44m/miniconda3  (writable)
      conda av data dir : /tmp/tmpn84st44m/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/anaconda/linux-64
                          https://conda.anaconda.org/anaconda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /tmp/tmpn84st44m/miniconda3/pkgs
                          /home/berntm/.conda/pkgs
       envs directories : /tmp/tmpn84st44m/miniconda3/envs
                          /home/berntm/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Linux/5.15.0-83-generic ubuntu/22.04.3 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers? [y/N]: ^C
Timeout reached. No report sent.

Might be solved by an update to conda conda/conda-build#4984 (comment)

bernt-matthias added a commit to bernt-matthias/galaxy that referenced this pull request Nov 12, 2023
potential solution for the build prblems in galaxyproject/planemo#1388
@bernt-matthias
Copy link
Contributor Author

Bumping conda helps with the failing test. Let's see if there are side effects galaxyproject/galaxy#17013

@nsoranzo
Copy link
Member

I am not sure how galaxyproject/galaxy#17013 can help, it bumps the minimum conda version but in tests it's most likely that the latest conda version is installed either way.

@bernt-matthias
Copy link
Contributor Author

All I know is that in the failing test on my local computer not the latest was installed, but:

           conda version : 23.3.1
    conda-build version : 3.27.0

When I bumped the minimum requirement the tests were successful. No idea how to test this here, without bumping the Galaxy packages first. Can we somehow patch tool-util during the test?

@nsoranzo
Copy link
Member

All I know is that in the failing test on my local computer not the latest was installed, but:

           conda version : 23.3.1
    conda-build version : 3.27.0

Is that what you have in ~/miniconda3 ?

When I bumped the minimum requirement the tests were successful. No idea how to test this here, without bumping the Galaxy packages first. Can we somehow patch tool-util during the test?

Did you update the conda package manually? Changing CONDA_PACKAGE_SPECS only affects new conda installations.

@nsoranzo
Copy link
Member

What's weird when looking at the CI logs is that test_local_conda_dependencies_version failed at the planemo test stage, when the [email protected] conda environment should have been created already by planemo conda_install right before.

@bernt-matthias
Copy link
Contributor Author

What's weird when looking at the CI logs is..

Yes. They had exit code 0, despite the failure.

@mvdbeek
Copy link
Member

mvdbeek commented Nov 13, 2023

Can you check in the github workflow if there is any interference from the conda or .condarc already installed on the VM ?

@bernt-matthias
Copy link
Contributor Author

So, now we see the error from the build step. Seems that this installs 23.3.1 .. my hypothesis is as follows:

  • the mambaforge installer comes with preinstalled conda 23.3.1 (just verified)
  • when installing conda use the two-step approach
    1. try to fulfill requirements such that installed packages don't need to be updated
    2. if this is not possible fulfill requirements and allow updating installed packages

so with our current conda pins, we are stuck at 23.3.1 (maybe also because we install in the base env?) and not the newest packages are installed.

Does this sound reasonable?

Note that also planemo conda_search seems not to use the exit code of the exec_command function. If desired I could try to fix it here.

@nsoranzo
Copy link
Member

Thanks @bernt-matthias , the latest commit to check the exit code really helped clarifying the issue, and your explanation makes sense. I didn't expect the latest Mambaforge to install conda 23.3.1, but any way it looks like that galaxyproject/galaxy#17013 is the correct solution here.

Note that also planemo conda_search seems not to use the exit code of the exec_command function. If desired I could try to fix it here.

Yes, that would be good, thanks!

@nsoranzo
Copy link
Member

planemo/commands/cmd_conda_install.py also has the same issue.

bernt-matthias added a commit to bernt-matthias/galaxy that referenced this pull request Nov 13, 2023
potential solution for the build prblems in galaxyproject/planemo#1388
@bernt-matthias
Copy link
Contributor Author

Does anyone have an idea what's going on with the 3.7 tests?

@mvdbeek
Copy link
Member

mvdbeek commented Dec 5, 2023

dev doesn't work on 3.7, that's why I rebased your PR

@mvdbeek mvdbeek changed the title update upper bound for galaxy packages Update upper bound for galaxy packages to < 23.2 Dec 6, 2023
Copy link
Member

@mvdbeek mvdbeek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool, thanks @bernt-matthias

@mvdbeek mvdbeek merged commit 5951e90 into galaxyproject:master Dec 6, 2023
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants