jinja2 + envvars and aliases

[campos-ddc]

Add support for jinja2 rendering of environment.yml files (if jinja2 is available).

When using jinja2, on top of the usual template capabilities, you have access to the
following variables:

• root: The directory containing environment.yml
• os: Python's os module.

Add support for setting environment variables and aliases in environment.yml:

name: oracle
dependencies:
  - oracle_instantclient

# List type environment variables will be joined with os.pathsep (':' in unix, ';' in windows).
# These values will be inserted in front of any existing value in the current environment.
# e.g.:
#   current PATH: "/usr/local/bin:/usr/bin"
#   new     PATH: "{{ root }}/bin:/usr/local/bin:/usr/bin"
environment:
  ORACLE_HOME: /usr/local/oracle_instantclient
  PATH:
    - {{ root }}/bin

aliases:
  run_db: bash {{ root }}/bin/run_db.sh

Notes:

• This PR includes the fix in PR Fix CONDA_ENV_PATH #86 (I needed it to make this work)
• Might invalidate the need for PR Add support for activate/deactivate scripts #85

[malev]

I can't think of a use case for this. would you mind providing me one?

[campos-ddc]

I'm using this to create a development environment for my projects.
In my case, there are multiple projects that are just git repositories cloned in a directory (and also external dependencies handled through vanilla conda)

My environment.yml file currently looks like this:

name: coilib50

includes:
  - {{ root }}/../../aasimar/environment.yml

  - {{ root }}/../../ben10/environment.yml

dependencies:
  - gcc
  - boost==1.55.0
  - scons==2.3.0

environment:
  - PYTHONPATH:
    - {{ root }}/source/python
    - {{ root }}/artifacts/12.0-redhat64-libs
  - LD_LIBRARY_PATH:
    - {{ root }}/artifacts/12.0-redhat64-libs

You can see that environment can be useful to add stuff to my path/pythonpath. Although some of that could be done using pip develop, I also need to set LD_LIBRARY_PATH for binaries, and other PATH values in Windows for the same reason.

jinja2 support allows me to find the root directory containing the environment file, and create paths relative to it.

Even more useful than that, I can use jinja to create different sets of dependencies based on the current platform (correct me if I'm wrong, but i don't think selectors such as # win are available in conda env):

  {% if os.sys.platform != 'win32' %}
  - unrar==5.2.7
  {% else %}
  - gnuwin32=0.6.3
  - openssl=1.0.0e
  - pywin32==217
  {% endif %}

[malev]

@campos-ddc this is the problem I see from your PR. It was meant to fix your own problem and it's hard for me to see any other situation where you would need something like this.
On the other hand, I like that the environment.yaml file is yaml, so it can be read by any yaml interpreter without any preprocessing tool.
I do like that you can have an environment with dependencies for linux or windows and they are independent, but I wish there was a different way to do that without adding a preprocessing layer. Do you understand what I'm saying?

[campos-ddc]

I used Jinja2 because it's the same thing available to conda-build.

What do you think about using selectors instead? Like [win] or [unix]. Still requires pre-processing, but not as ugly as jinja.

Otherwise, this cases can only be solved by having two separate environment files.

It's a pity that you don't think that conda-env should configure more of the environment (vars and aliases), I'll just have to maintain a fork with my requirements :)

[nicoddemus]

Hi @malev,

I work with @campos-ddc at @ESSS and would like to join the discussion, if I may. 😄

We have been using internal tools that take care of configuring our environment in a cross-platform way, and are now switching to conda and conda-env for our "virtual env" needs and we love the tools so far.

Our setup consist of several small projects, all dependent on one another, where we work on them in "develop" mode. This poses a few challenges that are not solved (as far as we know) by conda or conda-env out of the box:

• environment variables: a lot of our projects have complex C++ code and bindings written in Boost:Python, so we have to configure PATH and LD_LIBRARY_PATH to point to the appropriate location where those binaries are compiled so they can be loaded correctly. Because we switch a lot between different projects, configuring those environment variables in a .bashrc file is not practical or reliable, so it felt natural to us for conda-env to take care of that during environment activation.
• aliases: some of our projects configure some aliases which are useful to all developers working on that project, and the requirements regarding when aliases should be set/unset are the same for environment variables, during environment activation/deactivation.

We felt those requirements could be met by some patches to conda-env, which @campos-ddc has been working on for the past few weeks. We have been using it since then and it is working great for us, but you have a point that perhaps those requirements are not that obvious when seen from outside.

it's hard for me to see any other situation where you would need something like this.

How do you configure the environment variables your projects need? Or you don't have this need at all? We would love to hear how you guys solve this, we might have tunnel vision here as this is just the way we are used to work.

Regarding the PR as it stands now, could you pinpoint what are the points about the implementation or about the motivation behind it that you don't like? We are eager to discuss and make any changes you guys feel appropriate.

We at @ESSS would love to contribute back to the project, and we feel that our use case is general enough, if perhaps not that common.

Cheers,
🍻

[malev]

@campos-ddc @nicoddemus I see what you guys are saying, let me think a little bit more about this. I also want to be able to have conditions and environmental variables and I'm not a fan of [win] or [linux]. They don't seem as standard yaml.
One comment, if we had post-activate scripts, could you guys use that to store aliases and env variables? Looks like a more proper location for that kind of configuration. I know we don't have post-activate scripts, but there is a discussion about it somewhere.

[nicoddemus]

I see what you guys are saying, let me think a little bit more about this.

No problem. 😄

One comment, if we had post-activate scripts, could you guys use that to store aliases and env variables?

Doesn't some support for that exist already? I mean, this PR works by creating a .sh/.bat file in the /bin/activate.d folder in the environment which activates/deactivates environment variables and aliases. Or do you have something else in mind?

[nicoddemus]

Any news on this @malev? Any concerns you would like to address regarding this PR?

[nicoddemus]

@malev do you think it would be better to separate the features in this PR into individual PRs? If you prefer that (both to review and perhaps to reject some of them) we can work on it.

I would drop the aliases support though. There are differences between on how Windows and Linux define aliases, plus Windows by default doesn't even support aliases without external tools. Besides the user can get around it by putting shell scripts on the PATH instead, or using setuptools entry points.

[malev]

@nicoddemus I have an idea for this.
how about we add a new Spec (similar to RequirementsSpec in https://github.com/conda/conda-env/pull/203/files#diff-e335ea9881cadfa3a20099c73add7875R5

something like JinjaSpec.
So the object jinja_spec will take a environment.yml.jinja file and will return an environment object. I think this would be a unobtrusive way to solve the problem? I'll start coding a proof of concept

[nicoddemus]

At least for our use cases it seems this would work for us. Thanks for looking into this.

I wonder though how much other systems look for an environment.yml file to see if a project uses conda? @Korijn for example mentioned that readthedocs now supports conda environments (this is welcome news to me), I wonder if they decide to use that based on the repository having an environment.yml file? Another approach to the problem is using pre-processed comments like in conda-build.

Also, another feature we use a lot is environment variable support. How do you feel about that?

cc @tadeu, @campos-ddc do you have any thoughts here?

[malev]

if we do what I want do to, conda env create --filename environment.yml.jinja will work transparently. Maybe in a future PR we can have it working with conda env create.
I'm not familiar with pre-processed comments in conda-build.

[nicoddemus]

Sorry, the correct term is Preprocessing selectors.

IMO they feel like they would be a good fit for conda-env too (replacing jinja2).

[malev]

mmh, I'm not a fan of them, they don't look like yaml standard. I rather go with jinja2. Also with jinja we will have super versatile environment definitions.

[nicoddemus]

No problem, let's go with jinja2 then. 😄

[Korijn]

See readthedocs/readthedocs.org#857 for the readthedocs conda support development, I work at @clinicalgraphics and we're sponsoring the development of that feature. So to clarify, the intention is to just call conda env create from the root directory, at this point.

Although I can understand @malev's objection that it's not standard yaml (comments are allowed though right?), I really dig the preprocessed comments. It also seems logical to me to make similar and/or related features behave similarly across conda, conda-build and conda-env.

[tadeu]

I support this view:

It also seems logical to me to make similar and/or related features behave similarly across conda, conda-build and conda-env.

It should be noted that conda-build's meta.yaml already supports jinja2 and line selectors, and it also supports separating build, test and run dependencies.

It seems conda-env is turning into a generic tool that supports multiple ways of defining environments/dependencies (with the specs), but I still don't know if that is good or bad: it surely is more versatile, but it is more work for external tools to look at all these different ways of defining dependencies and environment things.

I wish that conda-env could work like a "conda-build develop mode", i.e., the developer defines a single conda.recipe/meta.yaml file in the project, and it would be useful both to build a package and to create an environment (for developing, running, building or testing).

[Korijn]

Ooh, such a good idea! Reuse of meta.yaml 👍

[nicoddemus]

I wish that conda-env could work like a "conda-build develop mode", i.e., the developer defines a single conda.recipe/meta.yaml file in the project, and it would be useful both to build a package and to create an environment (for developing, running, building or testing).

Perhaps instead of implementing what has been discussed here so far, it would be better then to just support meta.yml as a different spec? Is this a direction conda-env's team thinks would be better for the project?

[tadeu]

@malev @kalefranz

We still have interest to have these features in conda-env, specially environment variables and jinja2 support. :)

I've seem that it is tagged for 2.5.0, and that the 2.5.0 release date seems to be close. Would it help if we resolve the conflicts, and also perhaps simplify it and remove the aliases and tcc.exe part?

Thanks!

[jerowe]

I was reading over this and got interested.

Why don't you just use environment modules? Utilities like appending to environmental variables, manipulating PATHS (PATH, LD_LIBRARY_PATH, etc), and defining aliases are exactly what modules are for.

EasyBuild should fit a lot of what you are looking for.

We are using a very similar workflow on our HPC envirment. Right now we build the environment with the software stack, and upload it to anaconda env. It is then build in its correct location using easybuild, and from the module it is very easy to add environmental variables.

ETA - I say HPC, but I also use this to manage software stacks and development environments on servers, and even on my own personal laptop.

easyblock = 'EB_CondaEnv'

name = "gencore_base"
version = "1.0"
variant = "Linux-x86_64"

builddependencies = [('gencore_anaconda', '2-4.0.0')]

#If using an environment.yml file
#sources = ["environment.yml"]
#environment_file = sources[0]

#if using a remote env
remote_environment="jerowe/gencore_1.0"

modextravars = {
    'MY_SPECIAL_VAR':  'thing',
}

... tests

LD libraries are already as the root/lib.

ETA: I read back over this, and unfortunately I do not think environment modules can be built for windows, but I'm not sure.