bazel run uses wrong(?) directory for includes

[rileym-arcules]

After reading through the issues list, I believe this is similar to #123, but not exactly.

Some background:

We used to use an old make build system with modular structure where the src directory would be removed from includes:

ExampleModule
├── module.mak
└── src
    ├── Example.cpp
    ├── Example.h
    └── Makefile

Other modules would include Example.h using #include <ExampleModule/Example.h>.

Problem

This modular structure was super simple with bazel because of the strip_include_prefix. Now we see this interferes with the extractor, but we can't remove strip_include_prefix as we want to limit public headers (like google/glog#926, can't use include).

Example compile_commands.json output from bazel run @hedron_compile_commands//:refresh_all:

[
  {
    "file": "OtherModule/src/OtherExample.cpp"
    "arguments": [
      ....
      "-Ibazel-out/k8-dbg/bin/ExampleModule/_virtual_includes/lib",   // In here is the ExampleModule/Example.h symlink
      ....
    ],
    ....
  }
]

However, even after a bazel build of our target, that path doesn't exist. It's actually at
bazel-out/k8-dbg-ST-4a519fd6d3e4/bin/ExampleModule/_virtual_includes/lib.

Work around

If I run the tool manually in our workspace root (not suggested per documentation), using

PYTHONPATH=bazel-bin BUILD_WORKSPACE_DIRECTORY=$(pwd) python3 -c "import refresh_compile_commands as m; m.main()"

the compile_commands.json is different, and uses the desired path (bazel-out/k8-dbg-ST-4a519fd6d3e4/bin/ExampleModule/_virtual_includes/lib).

[cpsauer]

Hey, Riley! Good to meet you. Thanks for writing in, working to trace this back, and providing good details.

It looks like the include paths differ by (what I think is) a configuration hash. I'm not sure offhand why, especially given your direct run succeeds... I don't have time to dig in too much today, unfortunately, but I thought I should tap back quickly, if incompletely, just yet in case it helps us move it along.

Could I ask you to check in on two things:
(1) Just to make sure this isn't arising because of flag or configuration-transition differences, could I ask you to try running this tool from a refresh_compile_commands target (like in the README section on flags and configuration/transitions), specifying just one target and no additional flags? And then verifying that running that refresh_compile_commands target and building the same target specified (w/o flags) yields the same problem?
(2) If that doesn't resolve things, could I ask you to see if the environment differs between the two invocations? That is, just add a line to the top of main() that prints out the environment and see if it differs between the invocations?

Cheers,
Chris

[rileym-arcules]

Nice to meet you as well, and thank you for the timely response!

As far as 1), we're not using any special command flags or options during build or when running refresh_all:

refresh_compile_commands(
    name = "refresh_compile_commands",
    targets = "//:example"
)

using exactly bazel run @hedron_compile_commands//:refresh_all.

We do have a .bazelrc, although none of those options look like they would cause the effect we're seeing:

build -c dbg \
      --color=yes \
      --stamp \
      --workspace_status_command=Build/Bazel/get_workspace_status \
      --cxxopt="-DMAKEENV_SYSTEM_linux_gcc" \
      --cxxopt="-DMAKEENV_GCC_PLATFORM_x86_64" \
      --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" \
      --cxxopt="-std=c++17" \
      --cxxopt="-Wno-parentheses" \
      --cxxopt="-Wno-unknown-pragmas" \
      --cxxopt="-Wno-unused-local-typedefs" \
      --cxxopt="-Wno-deprecated-declarations" \
      --cxxopt="-Wno-class-conversion" \
      --cxxopt="-Wno-unused-function" \
      --cxxopt="-Wno-unused-variable" \
      --cxxopt="-Wno-placement-new" \
      --cxxopt="-Wno-terminate"

On 2), there are some differences when running. Please note I removed things that don't pertain to the issue. Running via bazel run:

{
  "RUNFILES_MANIFEST_FILE": "/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all.runfiles_manifest",
  "SHLVL": "2",
  "BUILD_WORKING_DIRECTORY": "/home/rileym/gitlab/src/project",
  "PYTHONSAFEPATH": "1",
  "_": "/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all",
  "PATH": "/home/rileym/.vscode-server/bin/4cb974a7aed77a74c7813bdccd99ee0d04901215/bin/remote-cli:/home/rileym/go/bin:/usr/local/go/bin:/home/rileym/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin",
  "BUILD_WORKSPACE_DIRECTORY": "/home/rileym/gitlab/src/project",
  "PWD": "/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all.runfiles/project",
  "PYTHONPATH": "/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all.runfiles:/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all.runfiles/:/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all.runfiles/bazel_tools:/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all.runfiles/project:/home/rileym/.cache/bazel/_bazel_rileym/567c132ed103c5789b82d69c88fbdf82/execroot/project/bazel-out/k8-dbg/bin/external/hedron_compile_commands/refresh_all.runfiles/hedron_compile_commands"
}

When running manually:

{
  "BUILD_WORKSPACE_DIRECTORY": "/home/rileym/gitlab/src/project",
  "PYTHONPATH": "bazel-bin",
  "PWD": "/home/rileym/gitlab/src/project",
  "SHLVL": "3",
  "PATH": "/home/rileym/.vscode-server/bin/4cb974a7aed77a74c7813bdccd99ee0d04901215/bin/remote-cli:/home/rileym/go/bin:/usr/local/go/bin:/home/rileym/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin",
  "OLDPWD": "/home/rileym/gitlab/src/project",
  "_": "/usr/bin/python3"
}

[rileym-arcules]

OH. Turns out it's the difference between running

bazel run @hedron_compile_commands//:refresh_all

and

bazel run :refresh_compile_commands

Not sure entirely why the former causes the problem, but alas.