Unable to build bazel on systems without /usr/bin/python

[tchernobog]

Description of the problem / feature request:

Several distributions have started removing python2 support as it is moving to EOL. This means there is no /usr/bin/python anymore, only /usr/bin/python3.

In this scenario, Bazel itself fails to build.

The same problem also happens with many other packages in projects other than bazel, making it a critical issue.
Examples include any project using https://github.com/bazelbuild/rules_docker, https://github.com/bazelbuild/rules_k8s, etc.

$ bazel build //src:bazel-bin --sandbox_debug 
[...]
INFO: Analyzed target //src:bazel-bin (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/matteo/Projects/bazel/src/BUILD:300:9: Executing genrule //src:embedded_tools_jdk_minimal failed (Exit 127): linux-sandbox failed: error executing command 
  (cd /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel && \
  exec env - \
    LD_LIBRARY_PATH=:/home/matteo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/matteo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib \
    PATH=/home/matteo/.yarn/bin:/home/matteo/.config/yarn/global/node_modules/.bin:/home/matteo/.scripts:/home/matteo/.gem/ruby/2.5.0/bin:/home/matteo/.cargo/bin:/home/matteo/.local/bin:/bin:/usr/lib/ccache:/home/matteo/.cargo/bin:/home/matteo/.cargo/bin:/home/matteo/.yarn/bin:/home/matteo/.config/yarn/global/node_modules/.bin:/home/matteo/.scripts:/home/matteo/.gem/ruby/2.5.0/bin:/home/matteo/.cargo/bin:/home/matteo/.local/bin:/bin:/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/matteo/Android/Sdk/platform-tools:/home/matteo/.npm-packages/bin:/home/matteo/Android/Sdk/platform-tools:/home/matteo/.npm-packages/bin \
    TMPDIR=/tmp \
  /home/matteo/.cache/bazel/_bazel_matteo/install/f1f06b37c54db45d54451bf6dea565c4/linux-sandbox -t 15 -w /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel -w /tmp -w /dev/shm -D -- /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/k8-fastbuild/bin/src/embedded_tools_jdk_minimal.zip" bazel-out/k8-fastbuild/bin/src/embedded_tools_jdk_minimal.params') linux-sandbox failed: error executing command 
  (cd /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel && \
  exec env - \
    LD_LIBRARY_PATH=:/home/matteo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/matteo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib \
    PATH=/home/matteo/.yarn/bin:/home/matteo/.config/yarn/global/node_modules/.bin:/home/matteo/.scripts:/home/matteo/.gem/ruby/2.5.0/bin:/home/matteo/.cargo/bin:/home/matteo/.local/bin:/bin:/usr/lib/ccache:/home/matteo/.cargo/bin:/home/matteo/.cargo/bin:/home/matteo/.yarn/bin:/home/matteo/.config/yarn/global/node_modules/.bin:/home/matteo/.scripts:/home/matteo/.gem/ruby/2.5.0/bin:/home/matteo/.cargo/bin:/home/matteo/.local/bin:/bin:/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/matteo/Android/Sdk/platform-tools:/home/matteo/.npm-packages/bin:/home/matteo/Android/Sdk/platform-tools:/home/matteo/.npm-packages/bin \
    TMPDIR=/tmp \
  /home/matteo/.cache/bazel/_bazel_matteo/install/f1f06b37c54db45d54451bf6dea565c4/linux-sandbox -t 15 -w /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel -w /tmp -w /dev/shm -D -- /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/k8-fastbuild/bin/src/embedded_tools_jdk_minimal.zip" bazel-out/k8-fastbuild/bin/src/embedded_tools_jdk_minimal.params')
1605363390.986791260: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1605363390.986834318: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1605363390.987124783: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 35580
1605363390.987164752: src/main/tools/linux-sandbox-pid1.cc:434: Pid1Main started
1605363390.987246747: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1605363390.987402261: src/main/tools/linux-sandbox-pid1.cc:176: working dir: /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel
1605363390.987419564: src/main/tools/linux-sandbox-pid1.cc:208: writable: /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel
1605363390.987428985: src/main/tools/linux-sandbox-pid1.cc:208: writable: /tmp
1605363390.987437258: src/main/tools/linux-sandbox-pid1.cc:208: writable: /dev/shm
1605363390.987507247: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /
1605363390.987515827: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys
1605363390.987522845: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/kernel/security
1605363390.987532222: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup
1605363390.987539991: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/unified
1605363390.987547028: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/systemd
1605363390.987553556: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/blkio
1605363390.987560266: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/pids
1605363390.987566995: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/freezer
1605363390.987574131: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/perf_event
1605363390.987580469: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/net_cls,net_prio
1605363390.987586530: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/cpuset
1605363390.987618149: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/memory
1605363390.987625393: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/cpu,cpuacct
1605363390.987631199: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/rdma
1605363390.987637222: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/cgroup/devices
1605363390.987643064: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/pstore
1605363390.987649086: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/firmware/efi/efivars
1605363390.987656977: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/bpf
1605363390.987662973: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/kernel/debug
1605363390.987669688: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/kernel/tracing
1605363390.987675805: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /sys/fs/fuse/connections
1605363390.987683333: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /dev
1605363390.987689628: src/main/tools/linux-sandbox-pid1.cc:279: remount rw: /dev/shm
1605363390.987697566: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /dev/pts
1605363390.987704220: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /dev/hugepages
1605363390.987761580: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /dev/mqueue
1605363390.987769837: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /run
1605363390.987776196: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /run/lock
1605363390.987782080: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /run/rpc_pipefs
1605363390.987788418: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /run/user/1000
1605363390.987796015: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /run/user/1000/gvfs
1605363390.987802973: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /run/user/1000/doc
1605363390.987808049: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /proc
1605363390.987819294: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /proc/sys/fs/binfmt_misc
1605363390.987827529: src/main/tools/linux-sandbox-pid1.cc:300: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1605363390.987838805: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /proc/sys/fs/binfmt_misc
1605363390.987844018: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /home
1605363390.987847882: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /boot
1605363390.987863375: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /boot/efi
1605363390.987868832: src/main/tools/linux-sandbox-pid1.cc:279: remount ro: /var/lib/docker/btrfs
1605363390.987874326: src/main/tools/linux-sandbox-pid1.cc:279: remount rw: /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel
1605363390.987880466: src/main/tools/linux-sandbox-pid1.cc:279: remount rw: /home/matteo/.cache/bazel/_bazel_matteo/e2f5ccead425e4f421d6f6631094e721/sandbox/linux-sandbox/1199/execroot/io_bazel
1605363390.987884714: src/main/tools/linux-sandbox-pid1.cc:279: remount rw: /tmp
1605363390.987888519: src/main/tools/linux-sandbox-pid1.cc:279: remount rw: /dev/shm
1605363390.987933727: src/main/tools/linux-sandbox-pid1.cc:363: calling fork...
1605363390.988066524: src/main/tools/linux-sandbox-pid1.cc:393: child started with PID 2
/usr/bin/env: 'python': No such file or directory
1605363390.992526457: src/main/tools/linux-sandbox-pid1.cc:410: wait returned pid=2, status=0x7f00
1605363390.992537613: src/main/tools/linux-sandbox-pid1.cc:428: child exited normally with code 127
1605363390.993905011: src/main/tools/linux-sandbox.cc:233: child exited normally with code 127
Target //src:bazel-bin failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.268s, Critical Path: 0.05s
INFO: 6 processes: 6 internal.
FAILED: Build did NOT complete successfully

Feature requests: what underlying problem are you trying to solve with this feature?

Compiling anything that relies on python scripts using Bazel on a Linux distribution released in the past few months.
At least Debian unstable and Ubuntu 20.04 are affected.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

• Install Debian sid or Ubuntu 20.04.
• Check out bazel sources
• Run bazel build //src:bazel-bin

What operating system are you running Bazel on?

Distributor ID: Debian
Description: Debian GNU/Linux bullseye/sid
Release: unstable
Codename: sid

What's the output of bazel info release?

release 3.7.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/bazelbuild/bazel.git
ca6209f
ca6209f

Have you found anything relevant by searching the web?

#11554 seems related to the same issue.

[gkelly]

It's not just building bazel itself, it's running any py_binary rule as the stub (src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt) uses /usr/bin/env python to locate the system python before it can even resolve the actual toolchain.

It seems like it might make sense for the stub to use the configured python toolchain's python interpreter, and not depend on the host python at all.

This bug prevents a lot of bazel rules from working on Ubuntu 20.10.

[meteorcloudy]

Related issue is #8685

[adam-azarchs]

I think this was fixed with 2945ef5:

bazel/src/main/java/com/google/devtools/build/lib/starlarkbuildapi/python/PyRuntimeInfoApi.java

Line 48 in 2945ef5

static final String DEFAULT_STUB_SHEBANG = "#!/usr/bin/env python3";

.

[meteorcloudy]

Thanks! Closing due to fixing #8685