[bazel] Should minimize unneeded runfiles data?

[EricCousineau-TRI]

In Anzu, we have switched some rules to @rules_python (before, we had a monolith). As a side effect, .runfiles enumeration is becoming somewhat of a bottleneck for build time and CI performance.

In a related thread, @jwnimmer-tri mentioned:

FYI I also noticed while profiling that ROS2 is worse than it used to be. I see programs with 7000+ runfiles that are symlinks to stuff in /opt/ros, including symlinks to *.cmake files. Something there has gone sideways, and is probably a source of the ever-growing pain of bazel build //....

Some options:

• Try to eliminate unneeded files from data = [] deps in generated BUILD file.
• (heavy hammer, may need some crafting) Use declare_directory to consolidate when possible. I'm not sure what this looks like when using existing system folder.

@jwnimmer-tri @ggould-tri @IanTheEngineer Any other thoughts?

[jwnimmer-tri]

The first step in optimizing something is to profile it. So I'd suggest the next action here would be to take one specific program in Anzu that has what seems like too many ROS-related runfiles, figure out which of them are spurious, hunt down which bzl code added the spurious things, and work to try to fix it to be more precise in what it lays claim to.

[jwnimmer-tri]

A nice mechanism to find runfile packrats is to look at the text files named bazel-bin/mypkg/myprog.runfiles/MANIFEST. That file has one line per runfile, and you can pretty easily grep/filter it to zoom in on ROS runfiles versus other external's runfiles to find hot spots.

[jwnimmer-tri]

Here's a flavor of some of the most common filenames in the 7,000 files in a typical ROS-using runfiles tree:

    296 package.dsv
    296 ament_prefix_path.sh
    292 path.dsv
    292 local_setup.zsh
    292 local_setup.sh
    292 local_setup.dsv
    292 local_setup.bash
    292 ament_prefix_path.dsv
    242 __init__.py
    196 ament_cmake_export_dependencies-extras.cmake
    188 ament_cmake_export_targets-extras.cmake
    164 ament_cmake_export_include_directories-extras.cmake
    160 library_path.sh
    160 library_path.dsv
    156 ament_cmake_export_libraries-extras.cmake
    124 pythonpath.sh
    124 pythonpath.dsv
...

It doesn't seem to me like running a simple ROS-using Anzu application really needs all of those *.cmake or *.dsv or *.{ba,}sh files?