How might it be possible to add IJulia to KNOWN_KERNELS?

[krinsman]

As far as I can tell, the value of argv in a kernel.json corresponding to an IJulia installation should be:

["/path/to/binary/julia", "-i", "--startup-file=yes", "--color=yes", "/path/to/install/location/of/IJulia/kernel.jl", "{connection_file}"]

Comparing this to what's already in envkernel.py, it seems like adding IJulia as a "known kernel" would amount to something similar to:

KNOWN_KERNELS = {
...
},
'ijulia' : {
'language' : 'julia',
'argv' : ['julia',
           '-i',
           '--startup-file=yes',
           '--color=yes',
           'kernel.jl',
           '{connection_file}'],
}
}

However, the command seems to require knowing the location of the file kernel.jl (corresponding seemingly to the install location of IJulia), and the above doesn't seem like it would fit with that.

I wanted to submit a PR along these lines, but I'm not sure how to deal with the issue of the kernel.jl file's location to make a PR that would actually work.

[rkdarst]

This is a good point when adding more kernels. Normally the Python kernels also hardcode the path to Python, but I could leave that out because envkernel would normally set up the PATH properly, so I could just assume it would be OK (for now). But the --python option sort of hacks around this and lets you manually set the path of the Python executable, something similar could be added for julia: --julia-kernejl-path, for example. Hackish but maybe OK to be able to make progress quickly? You can see example of using this at https://github.com/NordicHPC/envkernel/blob/master/envkernel.py#L165

One way I sort of got around this was with --kernel-template, which lets you install a kernel normally (which presumably sets the path correctly), then envkernel loads that spec and adds the wrapping to it. Still, this assumes the kernel is installable without envkernel - not always the case.

The immediate workaround is to use --kernel-cmd to specify the full argv, which you can compute yourself in the script that calls this.

To be fully general, we could add another option --kernel-args= which will do variable replacement, e.g. --kernel-args=kerneljl=/path/to/kerneljl,other_option=other_value. But this starts to seem like overengineering before we know how bad the problem will be. At this point maybe it would be better to make proper kernel-language hooks, but maybe that's really over-engineering.

I think the first option (--julia-kerneljs-path) said seems most reasonable to do with least effort. What do you think?

[krinsman]

@rkdarst I agree, that option makes the most sense to me too.

It seems somewhat unfortunate that kernelspec JSON's are not more similarly formatted between different languages' kernels, but maybe it wouldn't make sense to expect otherwise.

[rkdarst]

(more design background) And you know, the whole KNOWN_KERNELS itself was a bit of a hack. You'd think that envkernel shouldn't have to know about the kernel that is being installed. And it shouldn't, but it has to get that info from somewhere.

Here is how I thought the --kernel-template option would work:

module load my-conda               # at /path/to/conda
python -m ipykernel install --name=conda1 --prefix=/prefix/of/jupyterlab
# The kernel is now installed and will work, because it hardcodes the path 
# /path/to/conda/bin/python.  But running this won't set $PATH and other things
# to the conda env

envkernel lmod --name=conda1 --kernel-template=conda1 --prefix=/prefix/of/jupyterlab
# This reads in existing conda1 kernel, prepends the lmod-loading part to argv, then
# saves it back to the same name: conda1

This type of thing is completely general, and would work with any reasonably well-behaved kernel. I can imagine a problem with containers: you can install the kernel in the container that encodes the right paths, but getting that out is a problem. Though I guess you get get around that with mounting stuff cleverly.

It would be nice if envkernel could make the template automatically... but that would again require it to know every other kernel. So for now I guess the general method is to install the kernel yourself, then --template-kernel it.

What do you think?

[krinsman]

I guess to be honest I'm not sure what the --template-kernel option is supposed to do?

I think this is might be a little bit out of my depth. I'm slightly familiar with jupyter_client.kernelspec (e.g. see here https://github.com/krinsman/clonenotebooks/blob/master/clonenotebooks/cloners/cloners.py#L52 ) but don't deeply understand this.

(Actually this repository came up because we were considering calling envkernel to allow users, while cloning notebooks, to create kernelspecs that link to Docker and eventually probably also Shifter images.)

[rkdarst]

I saw this issue again, and also saw some old stuff I had written. I should link them:
https://github.com/NordicHPC/envkernel#other-kernels

I haven't re-read everything above, but the basic idea is "we can't know every kernel. So expect a user to install the kernel normally somehow, then we can --kernel-template=" to clone the argv to our new envkernel. This won't necessarily work in every case... but it does in my case at least and hopefully a lot of them.