Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bant won't detect dependency that should be removed #14

Closed
lpawelcz opened this issue Aug 21, 2024 · 4 comments
Closed

Bant won't detect dependency that should be removed #14

lpawelcz opened this issue Aug 21, 2024 · 4 comments

Comments

@lpawelcz
Copy link

In google/xls#1550 we discussed a situation where indata_generator.{h,cc} files there is no absl/status/status.h included and there is a dependency in the BUILD file for it ("@com_google_absl//absl/status").
The problem is that bant v0.1.6-1 built from the current upstream main (d033e1a) does not detect this inconsistency and it does not try to remove it.

Here is the output of bant dwyu -vvv //xls/modules/zstd:data_generator:

Dependcy graph: Did not find these packages                                                                                                                                                                                                                              10:10:05 [13/18190]
        @bazel_tools//tools/cpp/runfiles                                                                                                                                                                                                                                                    
        @com_google_absl//absl/algorithm
        @com_google_absl//absl/base              
        @com_google_absl//absl/container            
        @com_google_absl//absl/debugging                                                                                                      
        @com_google_absl//absl/flags                                                                                                          
        @com_google_absl//absl/log
        @com_google_absl//absl/status           
        @com_google_absl//absl/strings           
        @com_google_absl//absl/synchronization 
        @com_google_absl//absl/time           
        @com_google_absl//absl/types                               
        @com_google_protobuf//                                   
        @com_google_protobuf//src/google/protobuf/io
        @com_googlesource_code_re2//

Dependency graph: Did not find these targets
        @bazel_tools//tools/cpp/runfiles
        @com_google_absl//absl/algorithm:container
        @com_google_absl//absl/base:core_headers
        @com_google_absl//absl/base:log_severity
        @com_google_absl//absl/container:fixed_array
        @com_google_absl//absl/container:flat_hash_map
        @com_google_absl//absl/debugging:stacktrace
        @com_google_absl//absl/debugging:symbolize
        @com_google_absl//absl/flags:flag
        @com_google_absl//absl/log:check
        @com_google_absl//absl/log:flags
        @com_google_absl//absl/log:globals
        @com_google_absl//absl/log
        @com_google_absl//absl/log:log_entry
        @com_google_absl//absl/log:log_sink
        @com_google_absl//absl/status
        @com_google_absl//absl/status:statusor
        @com_google_absl//absl/strings:cord
        @com_google_absl//absl/strings:str_format
        @com_google_absl//absl/strings:string_view
        @com_google_absl//absl/strings
        @com_google_absl//absl/synchronization
        @com_google_absl//absl/time
        @com_google_absl//absl/types:span
        @com_google_protobuf//:protobuf
        @com_google_protobuf//src/google/protobuf/io
        @com_google_protobuf//src/google/protobuf/io:tokenizer
        @com_googlesource_code_re2//:re2
        
Dependency graph expanded build file# from initial 1 to 5; 15 targets and 42 that depend on these.
xls/modules/zstd/data_generator.cc:27:11-36: unknown provider for absl/algorithm/container.h -- Missing or from non-standard bazel-rule ?
xls/modules/zstd/data_generator.cc:28:11-32: unknown provider for absl/status/statusor.h -- Missing or from non-standard bazel-rule ?
xls/modules/zstd/data_generator.cc:29:11-32: unknown provider for absl/strings/str_cat.h -- Missing or from non-standard bazel-rule ?
xls/modules/zstd/data_generator.cc:30:11-33: unknown provider for absl/strings/str_join.h -- Missing or from non-standard bazel-rule ?
xls/modules/zstd/data_generator.cc:31:11-26: unknown provider for absl/time/time.h -- Missing or from non-standard bazel-rule ?
xls/modules/zstd/data_generator.cc:32:11-27: unknown provider for absl/types/span.h -- Missing or from non-standard bazel-rule ?
xls/modules/zstd/data_generator.cc:33:11-38: #include "xls/common/file/filesystem.h"
xls/modules/zstd/data_generator.cc:33:11-38:    | //xls/common/file:filesystem
xls/modules/zstd/data_generator.cc:34:11-44: #include "xls/common/file/get_runfile_path.h"
xls/modules/zstd/data_generator.cc:34:11-44:    | //xls/common/file:get_runfile_path
xls/modules/zstd/data_generator.cc:35:11-43: #include "xls/common/status/status_macros.h"
xls/modules/zstd/data_generator.cc:35:11-43:    | //xls/common/status:status_macros
xls/modules/zstd/data_generator.cc:36:11-33: #include "xls/common/subprocess.h"
xls/modules/zstd/data_generator.cc:36:11-33:    | //xls/common:subprocess
xls/modules/zstd/BUILD:150:14-30: ^... in source 'data_generator.cc' referenced by //xls/modules/zstd:data_generator
xls/modules/zstd/data_generator.h:21:11-32: unknown provider for absl/status/statusor.h -- Missing or from non-standard bazel-rule ?
xls/modules/zstd/BUILD:151:14-29: ^... in source 'data_generator.h' referenced by //xls/modules/zstd:data_generator
xls/modules/zstd/BUILD:160:10-51:: Unsure what @com_google_absl//absl/algorithm:container provides, but there are also unaccounted headers. Won't remove.
xls/modules/zstd/BUILD:161:10-38:: Unsure what @com_google_absl//absl/status provides, but there are also unaccounted headers. Won't remove.
xls/modules/zstd/BUILD:162:10-47:: Unsure what @com_google_absl//absl/status:statusor provides, but there are also unaccounted headers. Won't remove.
xls/modules/zstd/BUILD:163:10-39:: Unsure what @com_google_absl//absl/strings provides, but there are also unaccounted headers. Won't remove.
xls/modules/zstd/BUILD:164:10-36:: Unsure what @com_google_absl//absl/time provides, but there are also unaccounted headers. Won't remove.
xls/modules/zstd/BUILD:165:10-42:: Unsure what @com_google_absl//absl/types:span provides, but there are also unaccounted headers. Won't remove.
Checked DWYU on 1 targets.
Arena: 4612 allocations in 1 blocks; 0.169293 MB.
BUILD file glob walk 23 files/directories in 0.130ms
read(BUILD)       5 BUILD files with 53.00 KiB in 0.067ms (821.57 MB/sec)
Parse & build AST 5 BUILD files with 53.00 KiB in 2.482ms (22.18 MB/sec)
Elaborated 5 packages in 1.016ms
Dependency follow iterations 5 rounds in 3.620ms
  - of which exist-check 4 BUILD files in 0.009ms
  - of which glob() walking 93 files in 0.769ms
DWYU preparation 86 indexed targets in 1.807ms
read(C++ source) 2 sources with 4.00 KiB in 0.017ms (287.76 MB/sec)
Grep'ed for #inc 2 sources with 4.00 KiB in 2.404ms (2.03 MB/sec)
@hzeller
Copy link
Owner

hzeller commented Aug 21, 2024 via email

@lpawelcz
Copy link
Author

Ohhh, now I see what was the problem.

That almost looks like the workspace with all the expanded external
projects is not there entirely, e.g. due to no bazel build before.

This is 'sort of' the case for me. For XLS development I have pretty complicated build environment that resulted in broken symlinks to bazel cache from the perspective of bant . That is why bant acted like there was no bazel build executed before.

Now knowing that I was able to get bant to work with my setup and I indeed see the line that was missing:

buildozer 'remove deps @com_google_absl//absl/status' //xls/modules/zstd:data_generator

We can close this now as this was entirely an user error. Thanks for taking an interest and helping me out @hzeller!

@hzeller
Copy link
Owner

hzeller commented Aug 21, 2024

Ha, now I am curious how your set-up is. Is this something that bant could/should support ? Is it some RBE set-up (which I have not played with yet).

@lpawelcz
Copy link
Author

I perform my build inside a docker container with volumes bound to my XLS workspace and bazel cache directories. In the container, the paths to cache are different than on my machine so after the build I end up with symlinks to bazel-bin etc. pointing to paths from my docker container which are not valid outside of the container.
My error was that I used bant outside of the container and it couldn't reach the cache directories through the broken symlinks. I just had to get bant and use it inside the container when the symlinks were valid and it worked just fine then.

The container is required to ensure hermeticity of the build environment (bazel uses e.g. host's glibc ) for maximizing cache hits on shared bazel remote cache server.

I think that assuming bazel-* symlinks point to valid paths is OK. I encountered this error only because of my pretty non-standard setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hzeller @lpawelcz