Jupyter kernel crashes because of a problem with pywin32

[johann-petrak]

Just installed the most recent Miniconda on a fresh Windows 10 system, created a Python 3.9 conda environment, installed a bunch of other packages and when I tried to run a Jupyter notebook, the Kernel crashed.

The problem was identical to what was reported here: jupyter/notebook#4980

import win32api
ImportError: DLL load failed: The specified procedure could not be found.

This issue also describes a workaround that also worked for me, which is running:

python [environment path]/Scripts/pywin32_postinstall.py -install

However this is hardly a way to make things work for ordinary user, as they would first need to search for the problem, find some closed isse, figure out the exact file the they need to use and run the command to fix this.

See also mhammond/pywin32#1507
This explains that there may be a problem with how Jupyter bundles the win32api code which may be responsible for the problem

[kevin-bates]

@johann-petrak - thanks for moving this issue from Notebook.

While catching up on the links within mhammond/pywin32#1507 it seems to me that we've come full-circle here and I'm sure @MSeal is shaking his head right now.

I've come to the same conclusion as Matt wrt where this issue lies and it seems like it is related to Anaconda packaging of pywin32. The other observation is that dependencies on Windows DLLs are inherently fragile when mixing python environments and this is what pywin32_postinstall.py "repairs".

I really don't understand what it is Jupyter should do here aside from documenting this in Notebook's Troubleshooting section. pywin32 is purely a third-party dependency of a python application and if some additional step is required depending on whether that dependency was installed from an installer or pip or in multiple environments shouldn't be the responsibility of the dependent application.

Since this is way outside my wheelhouse (pun intended), I'd like to ask @bollwyvl for his two cents as I respect his knowledge on all things related to Python packaging (among other areas).

[bollwyvl]

My recommendations to new users trying to get, and maintain, a working jupyter environment:

• start with miniforge or mambaforge
  • never install anything in the base environment, other than conda and/or mamba
• use and check into VCS an environment.yml to specify both conda (and pip, if applicable) dependencies
  • ensure it contains channels: [conda-forge, nodefaults] to avoid mixing binaries that we don't test with
  • if really paranoid, use and check the output of conda-lock to generate explicit description of your environment
• use the environment.yml (or lockfile) to keep an environment up-to-date with conda env update
• try to avoid conda install, it's easy to forget channels, etc.
• really try to avoid doing pip install, like, ever
  • due to the design of conda using filesystem hard links to share packages, this can screw up multiple environments that share the same package
  • with the new pip solver, one no longer has any idea what going to come out if all of the packages aren't solved at once...
    • it could upgrade pywin32, jedi, or any other number of dependencies without respecting what's already there

If problems do arise:

• try pip check
  • it's not perfect, but will concisely warn if at least pip thinks something is broken
• check the output of pip freeze and conda list --explicit
  • often it will be very obvious where something weird has happened
• if a real issue is encountered, include the above information, as otherwise it's really hard for maintainers to deduce how the state has occurred
  • nice people either post these as gists and link to them, or put them in <details><pre></pre></details> blocks

Some background, somewhat on the blunt side:

We, the Jupyter community, have no real insight into why Anaconda, Inc. makes decisions on their packaging. Those who aren't paying for Anaconda Enterprise, etc., are beta-testing their process and also don't have a lot of say in what happens.

Disclaimer: a number of Jupyter core developers, including yours truly, were employed by Anaconda, Inc. at some point. This is a long and complicated relationship, and a lot of peoples' feelings and livelihoods have been involved... it recommend not poking that particular bear too hard.

We are, however, invested in maintaining the whole stack for the conda-forge feedstocks, and strive really hard to test the daylights out of stuff before we ship it to anaconda.org, and if it does break, try really hard to mark stuff broken. For the most part, anaconda is repackaging this work on conda-forge, when they get around to it, with minor modifications.

[kevin-bates]

Thanks @bollwyvl!

@johann-petrak - I think the best we can do about pywin32's issue is to continue using python [environment path]/Scripts/pywin32_postinstall.py -install to fixup multi-env environments.

Closing. Should we find the need to address this issue differently in this repository, we can re-open it at that time. Thank you.

[johann-petrak]

OK so from previous comments in here and in the pywin32 issue it seems that ultimately this can only be dealt with by Anaconda, see issue conda-forge/pywin32-feedstock#18