Bazel's Python stub is broken on Ubuntu 20.04 LTS

[philwo]

Description of the problem / feature request:

Bazel's uses a Python stub to make py_binary targets easily runnable: https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt?q=python_stub&ss=bazel

It uses the classic shebang-line #!/usr/bin/env python to invoke Python. This breaks on Ubuntu 20.04 LTS, because Ubuntu no longer provides a python executable - they only have /usr/bin/python2 and /usr/bin/python3.

This means that py_binary targets now fail with a message like this (from building Bazel itself on Ubuntu 20.04 LTS):

ERROR: /root/.cache/bazel/_bazel_root/ec321eb2cc2d0f8f91b676b6d4c66c29/external/test_WORKSPACE_files/BUILD:4:8: Couldn't build file external/test_WORKSPACE_files/archives.tar: PackageTar external/test_WORKSPACE_files/archives.tar failed (Exit 127) build_tar failed: error executing command bazel-out/host/bin/external/rules_pkg/build_tar @bazel-out/k8-fastbuild/bin/external/test_WORKSPACE_files/archives.args

Use --sandbox_debug to see verbose messages from the sandbox
/usr/bin/env: 'python': No such file or directory

Users can work around this by installing python-is-python2 or python-is-python3, which will install a system-wide symlink that redirects python to either of the versions, but neither seem to be installed by default, so we cannot rely on users having these.

What operating system are you running Bazel on?

Ubuntu 20.04 LTS

What's the output of bazel info release?

Bazel 3.2.0

[meisterT]

Probably it's a good time to add Ubuntu 20.04 to our CI after this bug is fixed.

[philwo]

@meisterT Yep. I discovered it during the preparations for that 😄

[davido]

Gerrit project would also like to use RBE with Ubuntu 20.04 image, see this request: [1].

This is needed, because Gerrit uses very recent Git core package (> 2.18) that is only available out of the box in Ubuntu 20.04.

Any help with moving forward with: [1] would be greatly appreciated.

Yes, I know, that we could easily create custom Ubuntu image for RBE, but a patch for that was staled: [2].

[1] bazelbuild/bazel-toolchains#870
[2] https://gerrit-review.googlesource.com/c/gerrit/+/260873

[cjds]

+1 on this. Experiencing this as well

[harinath]

This also appears to affect Debian unstable, which doesn't have the workaround packages mentioned above (python-is-python[23])

[fishy]

This also breaks Android rules.

$ bazel mobile-install :app --sandbox_debug
INFO: Invocation ID: 77a2fdb0-14a4-4628-9527-295368a6853d
INFO: Analyzed target //:app (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/fishy/work/notifbot/android/BUILD:7:15: Injecting mobile install stub application failed (Exit 127): process-wrapper failed: error executing command 
  (cd /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/sandbox/processwrapper-sandbox/501/execroot/notifbot && \
  exec env - \
    TMPDIR=/tmp \
  /home/fishy/.cache/bazel/_bazel_fishy/install/f1f06b37c54db45d54451bf6dea565c4/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/k8-py2-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/stubify_manifest '--mode=mobile_install' --input_manifest bazel-out/k8-fastbuild/bin/_merged/app/AndroidManifest.xml --output_manifest bazel-out/k8-fastbuild/bin/app_files/mobile_install/AndroidManifest.xml --output_datafile bazel-out/k8-fastbuild/bin/app_files/stub_application_data/stub_application_data.txt) process-wrapper failed: error executing command 
  (cd /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/sandbox/processwrapper-sandbox/501/execroot/notifbot && \
  exec env - \
    TMPDIR=/tmp \
  /home/fishy/.cache/bazel/_bazel_fishy/install/f1f06b37c54db45d54451bf6dea565c4/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/k8-py2-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/stubify_manifest '--mode=mobile_install' --input_manifest bazel-out/k8-fastbuild/bin/_merged/app/AndroidManifest.xml --output_manifest bazel-out/k8-fastbuild/bin/app_files/mobile_install/AndroidManifest.xml --output_datafile bazel-out/k8-fastbuild/bin/app_files/stub_application_data/stub_application_data.txt)
/usr/bin/env: 'python': No such file or directory
Target //:app failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.335s, Critical Path: 0.22s
INFO: 6 processes: 6 internal.
FAILED: Build did NOT complete successfully

I manually linked /usr/local/bin/python to python2, but because that command added exec env - to clear environment variables, /use/local/bin is no longer in $PATH and that no longer works.

[danieljanes]

Is there a chance to get this fixed in time for Bazel 4.0 (#12455)? If not, is this something that will become part of a 4.0 LTS patch release or will it have to wait until 5.0?

[lguangyu]

Is there a work around that during building the bazel itself, to permanently change default rule of python to invoke python3? or more generally, to use any python I want, even if it is in a non-standard path?

[fishy]

The workaround I use is to add these to my .bazelrc:

build --extra_toolchains=@rules_python//python:autodetecting_toolchain_nonstrict
mobile-install --extra_toolchains=@rules_python//python:autodetecting_toolchain_nonstrict

But that won't work for you if the thing you use actually requires python2 and won't work in python3.

[lguangyu]

I see.

For building bazel itself on python3-only systems I found a work around by changing file:
./src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java

// line 250:
String pythonExecutableName = OS.getCurrent() == OS.OPENBSD ? "python3" : "python";

to

String pythonExecutableName = "python3";

This will at least affect all scripts generated to build bazel iteself. I'm not sure if this also affects future builds. Previous build failures were caused by auto-generated scripts interpreter instruction line "#!/usr/bin/env python"

[blais]

I just burnt an hour on this problem after an Ubuntu 20.04 update.
Thanks for filing the ticket (please address!).

[caseyriebe]

Had a similar issue with building in a Dockerfile. I am able to get around it with:

RUN ln -s /usr/bin/python3 /usr/bin/python & \
    ln -s /usr/bin/pip3 /usr/bin/pip

[junyer]

Would it be fair to say that PR #11434 addresses this issue as well as issue #8685?

Oh, and PR #13414 for the autodetecting toolchains. :)

[philwo]

@junyer Thank you! Indeed it seems like this was fixed by the referenced PRs. :)