DAS without the anaconda channel

[postpop]

Many institutions are blocking the anaconda channel because of licensing issues, so we need builds that work without anaconda. This is mainly an issue for windows because we rely on anaconda to provide the tensorflow-gpu package.

The new release 0.38.4 was built without anaconda in all supported platforms (linux, macos (arm only), windows).

You can give it a try with:
conda create -n das python=3.10 das=0.32.4 -c conda-forge -c ncb

Linux and Mac builds should also work with python=3.11.

Please test and report any issues with the no-anaconda builds here.

May also address #77.

[avakiai]

Just commenting that this worked very well for me. Just a note: don't be surprised if the install takes much longer than previous installations of das...

[postpop]

Thanks! Glad it works.

And thanks for the note on the slowness: It didn't take longer on my machine. What version of conda are you on? The newer versions (23.10+) use the faster mamba solver by default (https://conda.org/blog/2023-11-06-conda-23-10-0-release/).

So either update conda (conda update conda -n base) or use mamba instead of conda for the installation. We will update the docs with this information once the new builds have been tested properly.

[hummuscience]

I am getting errors that the ncb channel does not include files for osx-arm64

conda install ncb::das

[postpop]

Hi, please use the installation instruction from the github readme or the documentation.
This is the correct command:

conda create python=3.11 das=0.32.4 -c conda-forge -c ncb -c nvidia -c apple -n das -y

Just tested it on my M1 mac and it works. Please re-open if you run into issues with this.

[hummuscience]

Our institute blocks the default conda repo.

Collecting package metadata (current_repodata.json): failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 161, in conda_http_errors
        yield
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 127, in repodata
        response.raise_for_status()
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://conda.anaconda.org/ncb/osx-arm64/current_repodata.json
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/requests/models.py", line 974, in json
        return complexjson.loads(self.text, **kwargs)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
        return _default_decoder.decode(s)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/simplejson/decoder.py", line 372, in decode
        obj, end = self.raw_decode(s)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/simplejson/decoder.py", line 402, in raw_decode
        return self.scan_once(s, idx=_w(s, idx).end())
    simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/exceptions.py", line 1132, in __call__
        return func(*args, **kwargs)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/main.py", line 69, in main_subshell
        exit_code = do_call(args, p)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 122, in do_call
        return getattr(module, func_name)(args, parser)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/notices/core.py", line 121, in wrapper
        return func(*args, **kwargs)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/main_create.py", line 41, in execute
        install(args, parser, 'create')
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/install.py", line 264, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 134, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 177, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 282, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/common/io.py", line 84, in decorated
        return f(*args, **kwds)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 449, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 1064, in _prepare
        reduced_index = get_reduced_index(self.prefix, self.channels,
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/index.py", line 267, in get_reduced_index
        new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 169, in query_all
        result = tuple(chain.from_iterable(executor.map(subdir_query, channel_urls)))
      File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
        yield fs.pop().result()
      File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
        return self.__get_result()
      File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
        raise self._exception
      File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 158, in subdir_query
        return tuple(
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 174, in query
        self.load()
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 264, in load
        _internal_state = self._load()
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 366, in _load
        raw_repodata_str = self._repo.repodata(cache.state)  # type: ignore
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 127, in repodata
        response.raise_for_status()
      File "/Users/abdelhaym/miniconda/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 208, in conda_http_errors
        raise RepodataIsEmpty(
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/exceptions.py", line 462, in __init__
        body = response.json()
      File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/requests/models.py", line 978, in json
        raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

`$ /Users/abdelhaym/miniconda/bin/conda create python=3.11 das=0.32.4 -c conda-forge -c ncb -c nvidia -c apple -n das -y`

  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=base
                CONDA_EXE=/Users/abdelhaym/miniconda/bin/conda
             CONDA_PREFIX=/Users/abdelhaym/miniconda
    CONDA_PROMPT_MODIFIER=(base)
         CONDA_PYTHON_EXE=/Users/abdelhaym/miniconda/bin/python
               CONDA_ROOT=/Users/abdelhaym/miniconda
              CONDA_SHLVL=1
           CURL_CA_BUNDLE=<not set>
               LD_PRELOAD=<not set>
                     PATH=/Users/abdelhaym/bin:/Users/abdelhaym/homebrew/bin:/Users/abdelhaym/.c
                          argo/bin:/Users/abdelhaym/homebrew/opt/qt@5/bin:/Users/abdelhaym/minic
                          onda/bin:/Users/abdelhaym/miniconda/condabin:/usr/local/bin:/System/Cr
                          yptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.s
                          ecurity.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.app
                          le.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple
                          .security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X
                          11/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/usr/local/munki
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : base
    active env location : /Users/abdelhaym/miniconda
            shell level : 1
       user config file : /Users/abdelhaym/.condarc
 populated config files : /Users/abdelhaym/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.9.16.final.0
       virtual packages : __archspec=1=arm64
                          __osx=14.4.1=0
                          __unix=0=0
       base environment : /Users/abdelhaym/miniconda  (writable)
      conda av data dir : /Users/abdelhaym/miniconda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/ncb/osx-arm64
                          https://conda.anaconda.org/ncb/noarch
                          https://conda.anaconda.org/nvidia/osx-arm64
                          https://conda.anaconda.org/nvidia/noarch
                          https://conda.anaconda.org/apple/osx-arm64
                          https://conda.anaconda.org/apple/noarch
          package cache : /Users/abdelhaym/.conda/pkgs
       envs directories : /Users/abdelhaym/.conda/envs
                          /Users/abdelhaym/miniconda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.32.3 CPython/3.9.16 Darwin/23.4.0 OSX/14.4.1
                UID:GID : 1214968452:140006770
             netrc file : None
           offline mode : False

[postpop]

Ah, I see. I don't think your institute needs to do that. To avoid licensing issues, all they need to do is block the anaconda channel (https://conda.anaconda.org/anaconda), not the user channels. The above command works with an MPI and our institute - so I guess they block channels more selectively.

Don't think there is much I can do about this atm. I could share an environment file, but even that requires access to the ncb channel.

In the future, we might try to get DAS onto the conda-forge channel, but this is a lengthy process and would make deployment more complicated.

[hummuscience]

Thanks for checking. I will talk with IT and see if I can find a solution.

[postpop]

Thanks! Would be good for me and other's to know!

In the meantime, I will explore how to get DAS onto conda-forge.

[avakiai]

@hummuscience Did you update your ~/.condarc file to remove "defaults" from the channels? (IT sent an email about this in September). That should allow it to get around the firewall and only query conda-forge, if I've understood things correctly.

[postpop]

Thanks for the note. Yes, "defaults" were removed from condarc.

The problem is that DAS needs to query the ncb channel, not just the conda-forge channel. For some reason, that one also seems to be blocked.

[hummuscience]

@avakiai yes. Defaults is removed from condarc. Did you manage to install DAS on your machine?
(@postpop we work in the same institute)

[avakiai]

Yes, I recently upgraded my local machine to have the latest version of das. Initially I ran into some issues, but after some fixes from Jan (above), it worked fine. I wasn't aware that they changed anything about the firewall since September, hence I'm a bit puzzled.

In case it solves your problem for now, there is a working das env in the cluster. It's a recent but not the latest version - I imagine that shouldn't make a large difference though.