Skip to content

Commit

Permalink
Ensure a hermetic python install is used for pip_parse and upgrade …
Browse files Browse the repository at this point in the history
…`rules_python` (#1886)

Summary: Ensure a hermetic python install is used for `pip_parse` and
upgrade `rules_python`

This is prerequisite work for migrating our docker dev image to Ubuntu
24.04 (#1885). In that PR, I've been running builds from within the
containers built via packer to verify the upgrade is successful. This
change is required to address the following issues:
* The `pip_parse` for `ubuntu_package_deps` comes before the hermetic
python definition. This results in a `ModuleNotFoundError` error without
the `rules_python` upgrade (since python 3.12, shipped with 24.04,
[removed disutils](https://peps.python.org/pep-0632/)).
* The remaining `pip_parse` calls fail when run within a 24.04 container
since it causes python to upgrade to 3.12.

Note: `rules_python` 0.26.0 is the latest version we can easily upgrade
to. The next release has issues with our current version of
`rules_docker`. Since `rules_docker` is deprecated and we are at the
latest version, upgrading beyond 0.26.0 seemed like it would be a more
significant change.

Relevant Issues: N/A

Type of change: /kind dependencies

Test Plan: CI build should pass and running builds in containers from
#1885 succeed

---------

Signed-off-by: Dom Delnano <[email protected]>
  • Loading branch information
ddelnano authored May 1, 2024
1 parent 0888c38 commit 4c20933
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 5 deletions.
15 changes: 13 additions & 2 deletions WORKSPACE
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,10 @@ load("//bazel:pl_workspace.bzl", "pl_container_images", "pl_model_files", "pl_wo

pl_workspace_setup()

load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_toolchains")

py_repositories()

# The pip_deps rule cannot be loaded until we load all the basic packages in the Pixie
# workspace. Also, bazel requires that loads are done at the top level (not in a function), so
# we need to pull it out over here.
Expand Down Expand Up @@ -136,10 +140,12 @@ tf_workspace0()

pl_model_files()

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
name = "python3_10",
# Allow the root user to build the code base since this is a current requirement for
# building in a containerized environment. See https://github.com/bazelbuild/rules_python/pull/713
# for more details.
ignore_root_user_error = True,
# Available versions are listed in @rules_python//python:versions.bzl.
# We recommend using the same version your team is already standardized on.
python_version = "3.10",
Expand All @@ -160,6 +166,7 @@ vizier_api_install_deps()

pip_parse(
name = "pxapi_python_doc_deps",
python_interpreter_target = interpreter,
requirements_lock = "//src/api/python/doc:requirements.bazel.txt",
)

Expand Down Expand Up @@ -250,6 +257,7 @@ go_download_sdk(

pip_parse(
name = "amqp_gen_reqs",
python_interpreter_target = interpreter,
requirements_lock = "//src/stirling/source_connectors/socket_tracer/protocols/amqp/amqp_code_generator:requirements.bazel.txt",
)

Expand All @@ -259,6 +267,7 @@ amp_gen_install_deps()

pip_parse(
name = "protocol_inference",
python_interpreter_target = interpreter,
requirements_lock = "//src/stirling/protocol_inference:requirements.bazel.txt",
)

Expand All @@ -275,6 +284,7 @@ py_image_repos()

pip_parse(
name = "amqp_bpf_test_requirements",
python_interpreter_target = interpreter,
requirements_lock = "//src/stirling/source_connectors/socket_tracer/testing/containers/amqp:requirements.bazel.txt",
)

Expand All @@ -300,6 +310,7 @@ px_deps_pinned_maven_install()

pip_parse(
name = "mongodb_bpf_test_requirements",
python_interpreter_target = interpreter,
requirements_lock = "//src/stirling/source_connectors/socket_tracer/testing/containers/mongodb:requirements.bazel.txt",
)

Expand Down
6 changes: 3 additions & 3 deletions bazel/repository_locations.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -457,9 +457,9 @@ REPOSITORY_LOCATIONS = dict(
urls = ["https://github.com/bazelbuild/rules_foreign_cc/archive/refs/tags/0.8.0.tar.gz"],
),
rules_python = dict(
sha256 = "cdf6b84084aad8f10bf20b46b77cb48d83c319ebe6458a18e9d2cebf57807cdd",
strip_prefix = "rules_python-0.8.1",
urls = ["https://github.com/bazelbuild/rules_python/archive/refs/tags/0.8.1.tar.gz"],
sha256 = "9d04041ac92a0985e344235f5d946f71ac543f1b1565f2cdbc9a2aaee8adf55b",
strip_prefix = "rules_python-0.26.0",
urls = ["https://github.com/bazelbuild/rules_python/archive/refs/tags/0.26.0.tar.gz"],
),
rules_jvm_external = dict(
urls = ["https://github.com/bazelbuild/rules_jvm_external/archive/refs/tags/5.2.tar.gz"],
Expand Down

0 comments on commit 4c20933

Please sign in to comment.