prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories

[Micket]

Alternative approach, don't attempt to rewrite paths, instead generate them directly.

Will of course require some easyblocks (primarily pythonpackage and pythonbundle) to adapt by using these new options, though they should be trivial.

Note: very much a draft right now. Completely untested and probably forgot something important.

easyblock PR: easybuilders/easybuild-easyblocks#3343
easyconfig PR: easybuilders/easybuild-easyconfigs#20960

[Micket]

Suggested change

	if len(python_paths) > 1 and not use_ebpythonprefixes:
	raise EasyBuildError('Multiple python paths requires EBPYTHONPREFIXES: ' + ', '.join(python_paths))
	elif python_paths:
	if python_paths:

This is (as far as I have found, having tested just about every type of easyconfig i could find), which has an actual issue with the code right now

EasyBuild itself generates both:

/apps/Test2/software/EasyBuild/4.9.2/lib/python3.6/site-packages/
/apps/Test2/software/EasyBuild/4.9.2/lib64/python3.6/site-packages/

while the latter is empty, my glob'ing still picks it up, and having multiple PYTHONPATH's doesn't make any sense, this errors out.
There is technically nothing wrong here, those paths are both compatible with python3.6, so they would be fine to use.

I can just remove that extra check. Simplifies the code and you simply get what you ask for. If you have multiple paths for some reason, they are both added, no problem. This is just an extra check we added that were never cared about before, and it's in no way a requirement for this PR.

On the other hand, it's literally only EasyBuild that does this, and I think it shouldn't duplicate the installation directories. Perhaps it should be creating a lib <-> lib64 symlink like just about everything else?

I could also make the code more robust, checking if it's really the same python version etc...

[Micket]

I removed this check for now, to keep this as simple as possible. This check could be introduced later if we want it (but we never checked for this in the past, so this change really has nothing to do with EBPYTHONPREFIXES, especially considering it actually only is problem when EBPYTHONPREFIXES isn't used)

[Micket]

I've made some simplifications:

1. I just drop that extra check for python paths. We have never checked this before, it's not a requirement for this PR, and in practice it never has any real errors in anything we currently have (and it would be very strange if a real issue ever appeared).

2. Secondly, i dropped lib64 variant from the globbed paths, since this was never considered by our multi python dep code (sitecustomize.py) to begin with, so I will not consider it here either for consistency.

3. I changed the global prefer-ebpythonprefixes default to false. While I think everyone really ought to use this due to the problems caused by using PYTHONPATH, there is a problem enabling this by default for existing installations: if your dependencies use PYTHONPATH, you can't shadow any libraries as PYTHONPATH will take precedence. Fixing this requires you rebuild the old easyconfigs (module-only) with --prefer-ebpyhonprefixes, while this is easy, i suspect many would have issues with this.
   We shadow libraries like this regularly, example https://github.com/easybuilders/easybuild-easyconfigs/blob/187ba72edb9f3c71203b1bf02a2243cfd8d203af/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.9.1-foss-2022a.eb#L130
   so if you had build scipy bundle before, and not try to build that tensorflow, you would pick up the wrong version of gast (and fail the sanity check)

[Flamefire]

Compared to #4496 it looks like this doesn't handle the case where easyblocks (and also easyconfigs?) add $PYTHONPATH directly, does it?

[Micket]

Correct, I don't override those if people have their own custom easyblocks or easyconfigs where they keep using PYTHONPATH (my accompanying PRs remove it from easybuild-easyblocks and easybuild-easyconfigs)

[Flamefire]

I'm wondering: There can/should never be more than one here, can there? It would mean we'd have e.g. lib/python3.8 and lib/python2.6 and will add both to $PYTHONPATH which can't be right

[Micket]

A bit of history, went through a few iterations here, at some points i had lib/ and lib64/, which if not symlinked, could be both considered without being an error (and there was a few cases like that which falsely triggered an error), so I just got tired of dealing with that as well so I opted skip that error checking for now.

Since then, I've changed to only consider lib/ since that's the only thing EBPYTHONPREFIXES could handle anyway, so, yes, it's changed slightly. So yes, if someone makes a non PythonPackage/PythonBundle, which also does multi_deps for python, and they set force_python = True for some reason, there would be multiple pythonpaths which maybe (possible likely) would break things (though breaking things isn't guaranteed if the actual python modules installed happens to work with both).

[boegel]

So when using $EBPYTHONPREFIX, it's fine if there are multiple ones (there would be when using multi_deps), but not when using $PYTHONPATH, so we should make the latter an error.

I'll look into that, I'm working on this right now (implementing tests)

[Flamefire]

Can't we have either an "enum" dep_type=BUILD|RUNTIME|BOTH or at least use include_build=True, include_runtime=True because now we have a potentially "invalid" call: When both are True

Maybe we should at least rename to exclude_runtime, exclude_build which would be mostly backwards compatible and setting both to True seems more reasonable to return nothing

[boegel]

Let's follow up on that in a separate PR, I want to get this PR merged ASAP.

Maybe open an issue for it, so we can tag it with 5.0 milestone.

[boegel]

lgtm

[lexming]

Merging, thanks @Micket !