`include_directories` is ignored in conda environment #106

Midnighter · 2020-01-11T07:28:07Z

Issue:

I'm trying to build a Python package that uses cmake, pybind11, GLPK, and SUNDIALS. Within the linked repository you can see that I can build a ubuntu based Docker image just fine. However, when I try to do the same within a conda environment compilation fails because the include directories are not added correctly to the compilation commands.

Steps to reproduce:

I have created an isolated conda environment with:

FROM continuumio/miniconda3:4.7.12

ENV PYTHONUNBUFFERED=1

ARG UID=1000
ARG GID=1000
ARG USER_=dunamis

RUN groupadd --system --gid "${GID}" "${USER_}" \
    && useradd --system --create-home --shell /bin/bash --home-dir "/home/${USER_}" \
        --uid "${UID}" --gid "${USER_}" "${USER_}"

COPY .condarc /root/

COPY .condarc "/home/${USER_}/"

COPY environment.yml /opt/

RUN set -eux \
    && conda env create --file /opt/environment.yml --prefix "/home/${USER_}/.dfba" \
    && conda clean --all --force-pkgs-dirs --yes \
    && chown --recursive "${USER_}" "/home/${USER_}"

WORKDIR "/home/${USER_}"

USER dunamis

RUN set -eux \
    && conda init bash

which uses .condarc

auto_activate_base: false
channel_priority: strict
channels:
  - conda-forge
  - bioconda
  - defaults

and environment.yml

name: dfba
dependencies:
- binutils-meta
- cmake
- compilers
- cxx-compiler
- git
- glpk=4.65
- make
- pkg-config
- pybind11
- python=3.7
- sundials=5.0.0

Once you run the Docker image, activate the environment with conda activate /home/dunamis/.dfba, then download the package curl -O https://gitlab.com/davidtourigny/dynamic-fba/-/archive/master/dynamic-fba-master.tar.gz. Extract the source and check compiler arguments:

tar -xzf dynamic-fba-master.tar.gz
cmake dynamic-fba-master
make -n

You should see that only -I/home/dunamis/.dfba/include/python3.7m is in the compiler arguments. If you edit the CMakeLists.txt file and change the include_directories command to add_compile_options("-I${GLPK_INCLUDE_DIR}") this is added and the package compiles correctly.

Do you have any idea what could cause these different behaviours?

Environment (conda list):

$ conda list
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       0_gnu    conda-forge
binutils-meta             1.0.4                         0    conda-forge
binutils_impl_linux-64    2.33.1               he1b5a44_7    conda-forge
binutils_linux-64         2.33.1              h9595d00_16    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
c-compiler                1.0.4                h516909a_0    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
certifi                   2019.11.28               py37_0    conda-forge
cmake                     3.16.2               h28c56e5_0    conda-forge
compilers                 1.0.4                         0    conda-forge
curl                      7.65.3               hf8cf82a_0    conda-forge
cxx-compiler              1.0.4                hc9558a2_0    conda-forge
expat                     2.2.5             he1b5a44_1004    conda-forge
fortran-compiler          1.0.4                he991be0_0    conda-forge
gcc_impl_linux-64         7.3.0                hd420e75_4    conda-forge
gcc_linux-64              7.3.0               h553295d_16    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
gfortran_impl_linux-64    7.3.0                hdf63c60_4    conda-forge
gfortran_linux-64         7.3.0               h553295d_16    conda-forge
git                       2.24.0          pl526hce37bd2_1    conda-forge
glpk                      4.65              he80fd80_1002    conda-forge
gmp                       6.1.2             hf484d3e_1000    conda-forge
gxx_impl_linux-64         7.3.0                hdf63c60_4    conda-forge
gxx_linux-64              7.3.0               h553295d_16    conda-forge
krb5                      1.16.4               h2fd8d38_0    conda-forge
ld_impl_linux-64          2.33.1               h53a641e_7    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libcurl                   7.65.3               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.3.0                hdf63c60_4    conda-forge
libgomp                   9.2.0                h24d8f2e_2    conda-forge
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libopenblas               0.3.7                h5ec1e0e_6    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
libuv                     1.34.0               h516909a_0    conda-forge
make                      4.2.1             h14c3975_2004    conda-forge
metis                     5.1.0             he1b5a44_1005    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
openssl                   1.1.1d               h516909a_0    conda-forge
pcre                      8.43                 he1b5a44_0    conda-forge
perl                      5.26.2            h516909a_1006    conda-forge
pip                       19.3.1                   py37_0    conda-forge
pkg-config                0.29.2            h516909a_1006    conda-forge
pybind11                  2.4.3            py37hc9558a2_0    conda-forge
python                    3.7.6                h357f687_2    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
rhash                     1.3.6             h14c3975_1001    conda-forge
setuptools                44.0.0                   py37_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
suitesparse               5.6.0                h717dc36_0    conda-forge
sundials                  5.0.0                h0186668_3    conda-forge
tbb                       2019.9               hc9558a2_1    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

Details about conda and system ( conda info ):

$ conda info

     active environment : /home/dunamis/.dfba
    active env location : /home/dunamis/.dfba
            shell level : 1
       user config file : /home/dunamis/.condarc
 populated config files : /home/dunamis/.condarc
          conda version : 4.7.12
    conda-build version : not installed
         python version : 3.7.4.final.0
       virtual packages :
       base environment : /opt/conda  (read only)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /home/dunamis/.conda/pkgs
       envs directories : /home/dunamis/.conda/envs
                          /opt/conda/envs
               platform : linux-64
             user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Linux/5.3.0-7625-generic debian/10 glibc/2.28
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

The text was updated successfully, but these errors were encountered:

leej3 · 2020-06-19T14:25:57Z

I encountered this issue too. I think it is the cmake binary itself that is causing the problem. I can compile/install as I expect for a range of versions of cmake but for a conda distributed cmake binaries I can trigger this failure for header include directories. This error occurs inside and outside of the conda build environment. More specifically I observe this for cmake version 3.14.0 on the default channel and for the conda-forge channel 3.13.4 >= broken < 3.17.0.

Currently fixes include:

set the cmake version according to your version/channel needs

It looks like you can't set the channel within the yaml file but you can build with something along the lines of conda build -c conda-forge .

EDIT: I do not know the intended behavior so calling it "broken" may be a misnomer

Install fails with certain cmake versions: conda-forge/cmake-feedstock#106

traversaro · 2020-11-28T14:13:05Z

I think that this issue is similar to #129, but related to implicit include directories. If I print CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES in the same system described in that issue, I get:

-- CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES: /home/traversaro/miniconda3/envs/cmake-tests/include;/home/traversaro/miniconda3/envs/cmake-tests/x86_64-conda-linux-gnu/include/c++/9.3.0;/home/traversaro/miniconda3/envs/cmake-tests/x86_64-conda-linux-gnu/include/c++/9.3.0/x86_64-conda-linux-gnu;/home/traversaro/miniconda3/envs/cmake-tests/x86_64-conda-linux-gnu/include/c++/9.3.0/backward;/home/traversaro/miniconda3/envs/cmake-tests/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include;/home/traversaro/miniconda3/envs/cmake-tests/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include-fixed;/home/traversaro/miniconda3/envs/cmake-tests/x86_64-conda-linux-gnu/include;/home/traversaro/miniconda3/envs/cmake-tests/x86_64-conda-linux-gnu/sysroot/usr/include

But if I try to check the actual default include directories of the compiler, I get:

#include "..." search starts here:
#include <...> search starts here:
 /home/traversaro/miniconda3/envs/cmake-tests/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/include/c++/9.3.0
 /home/traversaro/miniconda3/envs/cmake-tests/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/include/c++/9.3.0/x86_64-conda-linux-gnu
 /home/traversaro/miniconda3/envs/cmake-tests/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/include/c++/9.3.0/backward
 /home/traversaro/miniconda3/envs/cmake-tests/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/include
 /home/traversaro/miniconda3/envs/cmake-tests/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/include-fixed
 /home/traversaro/miniconda3/envs/cmake-tests/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/include
 /home/traversaro/miniconda3/envs/cmake-tests/bin/../x86_64-conda-linux-gnu/sysroot/usr/include
End of search list.

This means that any call to include_directories("/home/traversaro/miniconda3/envs/cmake-tests/include") would be ignored, even if /home/traversaro/miniconda3/envs/cmake-tests/include is not considered by default by the compiler.

jakirkham · 2022-02-23T03:00:05Z

Is this still an issue?

ntorresalberto · 2022-03-31T20:00:26Z

Is this still an issue?

Yes, I think I've just ran into the same problem in a personal project and googling a bit (I didn't test the example though).

jakirkham · 2022-03-31T20:05:26Z

@ntorresalberto do you have a minimal reproducer that you can share? 🙂

ntorresalberto · 2022-04-07T13:32:56Z

@ntorresalberto do you have a minimal reproducer that you can share? slightly_smiling_face

I apologize for the delay and for the bad info.
You were very sharp to request a minimal reproducer because now I don't think I can.
I'm not sure what went wrong in my first tries before, but if I ever find out I'll report back.

willian-m · 2022-05-31T12:54:52Z

I encountered this issue too. I think it is the cmake binary itself that is causing the problem. I can compile/install as I expect for a range of versions of cmake but for a conda distributed cmake binaries I can trigger this failure for header include directories. This error occurs inside and outside of the conda build environment. More specifically I observe this for cmake version 3.14.0 on the default channel and for the conda-forge channel 3.13.4 >= broken < 3.17.0.

Currently fixes include:

set the cmake version according to your version/channel needs

It looks like you can't set the channel within the yaml file but you can build with something along the lines of conda build -c conda-forge .

EDIT: I do not know the intended behavior so calling it "broken" may be a misnomer

I found a similar behavior with cmake 3.20 in a conda environment. On my makefile I had

find_package(Boost REQUIRED COMPONENTS program_options)
include_directories(${Boost_INCLUDE_DIRS})

and I was getting a "boost/program_options.hpp" not found even after a conda install boost. After I downgraded to v3.12, my project compiled normally.

lionjo · 2022-09-05T16:47:49Z

I had the same error as @willian-m using cmake 2.24 in a conda environment: Boost wasn't found despite doing

include_directories(${Boost_INCLUDE_DIRS})
find_package(Boost REQUIRED COMPONENTS program_options)

A simple

unset CXX_FLAGS

solved it for me. Apparantly cmake checks if the include directory exists in $CXX_FLAGS and if yes, does not include it.

isuruf · 2022-09-05T21:03:20Z

Going to close this issue. Please open a new one with the required information and a minimal reproducer.

Midnighter · 2022-09-06T07:05:18Z

You have a reproducible example in my original post, no?

isuruf · 2022-09-06T07:09:47Z

@Midnighter, would you be able to give a minimal reproducer instead of the complex reproducer you have here?

Midnighter · 2022-09-06T07:13:25Z

I'll try. Plenty of other things on my plate now so it will take me a bit.

WilliamTambellini · 2023-01-31T16:49:33Z

Still same issue here with cmake 3.24.1.
The missing path is indeed in CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES :
08:45:52 -- CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES: libraries/python-3.8.0/include/python3.8
which is NOT used/added when calling g++ so compilation error.
tbc.

leej3 added a commit to leej3/staged-recipes that referenced this issue Jul 16, 2020

restrict cmake version used

ba4af23

Install fails with certain cmake versions: conda-forge/cmake-feedstock#106

isuruf closed this as completed Sep 5, 2022

isuruf reopened this Sep 6, 2022

mglisse mentioned this issue Nov 20, 2023

add GudhUI compilation flag GUDHI/gudhi-devel#1001

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`include_directories` is ignored in conda environment #106

`include_directories` is ignored in conda environment #106

Midnighter commented Jan 11, 2020

leej3 commented Jun 19, 2020 •

edited

Loading

traversaro commented Nov 28, 2020 •

edited

Loading

jakirkham commented Feb 23, 2022

ntorresalberto commented Mar 31, 2022

jakirkham commented Mar 31, 2022

ntorresalberto commented Apr 7, 2022

willian-m commented May 31, 2022 •

edited

Loading

lionjo commented Sep 5, 2022

isuruf commented Sep 5, 2022

Midnighter commented Sep 6, 2022

isuruf commented Sep 6, 2022

Midnighter commented Sep 6, 2022

WilliamTambellini commented Jan 31, 2023

include_directories is ignored in conda environment #106

include_directories is ignored in conda environment #106

Comments

Midnighter commented Jan 11, 2020

Issue:

Steps to reproduce:

leej3 commented Jun 19, 2020 • edited Loading

traversaro commented Nov 28, 2020 • edited Loading

jakirkham commented Feb 23, 2022

ntorresalberto commented Mar 31, 2022

jakirkham commented Mar 31, 2022

ntorresalberto commented Apr 7, 2022

willian-m commented May 31, 2022 • edited Loading

lionjo commented Sep 5, 2022

isuruf commented Sep 5, 2022

Midnighter commented Sep 6, 2022

isuruf commented Sep 6, 2022

Midnighter commented Sep 6, 2022

WilliamTambellini commented Jan 31, 2023

`include_directories` is ignored in conda environment #106

`include_directories` is ignored in conda environment #106

leej3 commented Jun 19, 2020 •

edited

Loading

traversaro commented Nov 28, 2020 •

edited

Loading

willian-m commented May 31, 2022 •

edited

Loading