(RFC) How to update existing installations

[jaimergp]

The update logic in the installers is a bit convoluted right now, and I don't think it's working at all.

The background is that previous iterations of constructor would consider the target environment (if not empty) as part of the fresh installation. However, at some point, the installer was instructed to error if the target directory was not empty (by default). Adding the -f or -u switches in the shell installer would somehow allow that, but there were no guarantees it would succeed. I am not aware of similar switches on Windows installers.

These days, the installer does not solve the target environment, it simply runs an explicit install, overwriting as needed if -f was passed.

I think that's still the right approach (happy to discuss that, though), but some users are confused by how to update the installation. There are two options:

• In installers providing a conda deployment, conda update -n base conda should suffice assuming the base environment is tractable.
• If conda is not involved, then it's a matter of uninstalling / removing and installing the new one.

I am thinking that maybe we could provide either a custom field to change the default message, or at least parameterize the message depending on whether conda is included in base or not. However, some apps might not want to mention conda at all to keep their delivery story simpler for their users (e.g. Spyder or napari).

[marcoesters]

The update logic in the installers is a bit convoluted right now, and I don't think it's working at all.

I would second that. There are a lot of intricacies involved. It looks like -u and -f are actually identical flags. So, updating really just installs whatever is in the installer into the conda environment.

That can, of course, cause all kinds of conflicts with existing packages, even in the best case scenario where the base environment has not been changed at all.

[jaimergp]

Yes, to add some more context to other contributors:

• constructor runs an explicit install on the target location. This means that the solver does not run at all. It's mostly extracting packages to the location. This is awesome when the location is empty, but it's a source of problems when we are "updating".
• What to expect out of an update? This depends on the lifetime of the application.
  • If the application is not expected to change during its lifetime, an update might just be a standardized way of running the uninstallation before running the new installation on the same location. In other words, the "update" code would just delete everything in the target location, and then run the installation on top, usually keeping configuration files.
  • If the application does change during its lifetime, then it's a bit more complex. In installations like Miniconda, what are we supposed to do? Run conda update conda? If we blindly extract things there, it will most likely result in a broken installation, as Marco said above.

So I think we should approach this very carefully, and only in some cases. Maybe the easiest from our maintenance point of view is to add a flag allow_updates, which is then exposed to the pre_install script (or a new pre_cache_setup).

[mrclary]

What we have done for Spyder so far is incorporate the update mechanism into our Spyder application. Assuming a fresh install from our constructor-based installer, our update plugin checks for available updates on Github and downloads a conda-lock file for minor/micro updates, or the full (constructor-based) package installer if a major update.

For micro updates, Spyder uses conda update with the runtime environment conda-lock file on the runtime environment since major changes are not expected. For minor updates, Spyder first removes the runtime environment, updates the base environment with a base environment conda-lock file and then recreates the runtime environment with the runtime conda-lock file. For major updates, the installer installs into a new directory (spyder-<major>) in order to avoid conflict with the existing previous version install directory.

I agree that it would be nice to have some mechanism to allow installing over an existing install. However, I think it would be best to achieve this by first uninstalling the old version before installing the new one.

[marcoesters]

The challenge here are conda installations that have additional environments and package changes between installer versions.

• If a package is added between installer versions, conda update may not be sufficient.
• If a package is removed between installer versions, what do we do with the removed package? Removing it may result in a bad user experience since this isn't immediately obvious. If leave it, we risk bloat and conflicts down the road. Conflicts can be resolved by re-installing the base environment with a user's consent. However:
• If we re-install the base environment, we must ensure that it works the same way as before, but with different packages. That does at least include preserving .condarc files and the pkgs directory (so that the other environments work). The latter is complicated because the new base environment also writes to it. What other files do we need to preserve? What about files that the user put in there for some reason?

Offering to re-install everything during the installation if the directory isn't empty may be a good first step (even though we have to be clear that this will destroy all environments), but it's not the full solution. And we must make our uninstall procedure more robust/complete first (I am working on #642 right now for conda-standalone-based installers).