compiling BCC stype C code into ELF format

[Xichen96]

I am trying to generate an ELF file from BCC compiler. I am able to generate an ELF file successfully, but it was not runnable. The current problem is with the "helpers" section. I have noticed that there is no "helpers" section in a clang compiled object file. Currently the error reports "libbpf: Program '.bpf.fn.xdp_root' contains unrecognized relo data pointing to section 3
Segmentation fault (core dumped)", and I am guessing the problem is with a helper function "bpf_tail_call". Does anyone have any insight?

[yonghong-song]

Probably provide a reproducible test case so people can help.

[Xichen96]

https://github.com/Xichen96/bcc/tree/bcc-compile-elf
I pushed my changes to my forked repo on branch bcc-compile-elf.

[Xichen96]

Aside from the code I added, test codes are in elf_test directory, in which xdp_root.o.bak is the clang compiled elf file, and running main.go will compile the bpf/xdp_root.c file info an elf (some path might be messed up). Compare to the clang compiled elf, there is a bit of difference. First thing would be the symbol table keeps a section id for every entry to keep track of which section the symbol belongs to, and it is all messed up in my compiled version. Second thing is the "helpers" section which does not exist in the clang compiled version. btw I have to substitute the SEC(maps) style of definition to BPF_HASH_MAP style of definition.

[Xichen96]

I am worked through a few problems already, but this one I just don't know how to fix. Any input is appreciated.

[anakryiko]

Do you have a high-level description on what you are trying to achieve and how do you go about it? We've been floating the idea of generating ELF object file compatible with libbpf format and then using libbpf to actually load it (to gain all the features of libbpf that are missing from BCC, among other things). I'm curious if what you are doing is similar in spirit and implementation-wise. Thanks.

[Xichen96]

bcc_common has this function bpf_module_create_c and bpf_module_create_c_from_string. They compiles c style bpf code with llvm MCJIT class, which takes in a MemoryManager and keep the generated code in it, and then loads the maps and programs and stuff. What I did was to implement a child class of the memory manager, which initialize a new ELF object with libelf, let MCJIT fill in the blanks in the ELF, and finish generating the object on exit. The whole initialization will stop midway and not load the maps and programs if a path is provided. So far the problem is that the generated ELF is not quite loadable. The detail of the problem I have provided above. You can test out the code in the link above.

[anakryiko]

I think the right and clean approach is to dive a bit deeper and make BCC use libbpf as BPF loader completely. This would require changing internal ELF composition to be compatible with libbpf ELF. Few high level things:

1. All entry-point BPF programs go into a ELF section where name specifies BPF hook (e.g., "tp/syscall/sys_enter_nanosleep", "kprobe/some_kernel_func", etc);
2. All BPF map definitions are gathered in a single ELF section ".maps" and follows libbpf format (this should be rather simple, given all maps are defined through macros and macro impl can be easily updated)
3. All helper functions are put together into a default ".text" section.
4. .BTF/.BTF.ext stay in ".BTF" and ".BTF.ext" sections as is.

That's pretty much it. I'm not very familiar with current BCC ELF layout, but I think libbpf ELF layout is much simpler and shouldn't be too problematic to convert to.

The benefits of having libbpf as a BPF loader for BCC are multiple. Global variables and full bpf-to-bpf helper function calls are few obvious ones.

Thoughts?

[yonghong-song]

Thanks @Xichen96 @anakryiko Totally agreed that reusing libbpf as the loader will be beneficial so bcc can focus on what most useful for it, python frontend, symbolization, usdt, etc. The biggest challenge will be backward compatibility. Will try to do a more thorough analysis/prototype sometime later.

[Littlefisher619]

Hi!

I'm interested about and trying to contribute to the similar things these days, but I have no ideas what things has going on now in 2022. It seems the tools developments has been moved to libbpf-tools, and new bcc tools is not encouraged. However, many tools implemented with bcc still exists today, so maybe it's still valuable to generating ELF object file compatible with libbpf format, and then using libbpf to actually load it? This may enable new features from libbpf easily and allow bcc to do aot compile.

Does anyone has context about how the bcc community discussed about this issue today? I have go through the commits and issues, but haven't found more of them. Thanks very much!

some possible related issues:

• support load objects #2564
• bcc internals: Keep track of BPF progs using function instead of section #3660
• 20fd1c8
• [DRAFT Proposal] bcc / tools repo evolution #3976

[spumer]

I'm waiting for this feature and looking into https://github.com/redhat-et/bpfd and want to create object file via bcc and then put into bpfd.