add option to generate an ELF file together with the bootable image

[wucke13]

The ELF should contain the sections with information on target load addresses so that the ELF file and the entry point (all standard metadata supported by ELF AFAIK) can be directly loaded with a debugger (i.e. GDB's load). This is primarily easing work with debuggers and tracing hardware.

The Binary itself can of course be trivially extracted from the respective ELF, but no the other way round.

[Ivan-Velickovic]

I'm confused because can't you already load the ELFs into GDB? E.g add-symbol-file myprogram.elf.

I'm not sure how a single ELF would work, since it would have multiple ELFs nested inside of it, how does GDB know which one to debug?

[Ivan-Velickovic]

Do you mean a single ELF for the whole system or are you looking to debug a particular component such as the loader, initial task or kernel?

[wucke13]

I'm confused because can't you already load the ELFs into GDB

Yes, that works. However, for a larger deployment with dozens if not hundreds of components each bringing their own ELF this might not scale, especially when verifying integration or debugging problems going across protection domains.

I'm not sure how a single ELF would work, since it would have multiple ELFs nested inside of it, how does GDB know which one to debug?

I think one would merge the ELF files into one, which has different sections per protection domain each with code and debug symbols. I'm unsure about the specifics, tagging @moritz-meier for further input.

Do you mean a single ELF for the whole system

This. The individual ELF files all can be found already scattered across the local builddir and the MICROKIT_SDK path, right?

[Ivan-Velickovic]

I think one would merge the ELF files into one, which has different sections per protection domain each with code and debug symbols. I'm unsure about the specifics, tagging @moritz-meier for further input.

Sure, although I'm still not quite sure how that would end up working. I don't know how GDB would be able to interpret the ELF as actually being multiple ELFs, each in a separate address space.

At TS we have been working on a debugger component for GDB (https://github.com/au-ts/libgdb) and I could envision that component having system metadata/ELFs embedded so that if you didn't have access to the individual ELFs or something you could still debug the system.

I am not familiar with other debuggers/tracing hardware so not sure what is needed there.

The individual ELF files all can be found already scattered across the local builddir and the MICROKIT_SDK path, right?

Yes.

[wucke13]

Sure, although I'm still not quite sure how that would end up working. I don't know how GDB would be able to interpret the ELF as actually being multiple ELFs, each in a separate address space.

Why does it need to interpret as multiple ELFs? In the end, there is multiple code sections, each loaded to different addresses in physical memory, and each code section also is accompanied by individual DWARF. The fact that this could also be represented by multiple, individual ELF files should be irrelevant? I'm aware of a (unfortunately closed source, hence I can not share) example of an OS doing this, they use a small python script to generate a linker script that just links together the different binaries, each as an individual section, and each with the target load address in the metadata.

With that ELF, for example in GDB or in Xilinx' xsct you can do a single load, without specifying addresses, and all is set (and the debugger has the source code).

[moritz-meier]

Would be nice if the microkit tool would generate an ELF file and not a raw binary. Loading raw binaries is always a hassle because you need to specify the load and entry addresses explicitly.

[Ivan-Velickovic]

Why does it need to interpret as multiple ELFs?

Sorry, meant to say "multiple programs".

[Ivan-Velickovic]

Would be nice if the microkit tool would generate an ELF file and not a raw binary. Loading raw binaries is always a hassle because you need to specify the load and entry addresses explicitly.

I've found the raw binaries to be more reliable but I can look into how difficult it would be to generate an ELF (but I guess that's a separate thing to this discussion).

[alwin-joshy]

Why does it need to interpret as multiple ELFs? In the end, there is multiple code sections, each loaded to different addresses in physical memory, and each code section also is accompanied by individual DWARF.

Could you explain this a little bit more? The microkit binary image consists of the binary of the loader, with all the ELFS for the different PDS unpacked and appended to the end of it. I imagine the things that you generally want to use GDB for are the PDs themselves, not the loader or the monitor, so I don't really understand what you get from emitting the loader.img file as an ELF.

When the loader/monitor start, they create a bunch of VSpaces for each of the protection domains and map the relevant data for each PD into the vspace, often with conflicting virtual address (i.e. code usually starts at something like 0x200000 on AARCH64), so I don't understand how you can avoid specifically loading the ELF file of the particular PD(s) that you are interested in debugging. Edit: Unless you mean that in the process of stitching all of the ELFs together you kind of shift them around in such a way that they are loaded in non-conflicting addresses? But considering that the tool doesn't compile the PD ELFs, would it have to require that they are compiled position-independently or something?

[wucke13]

First, disclaimer: I might be wrong on the following, this is my (limited) understanding.

Each memory segment in the ELF program header table has two fields in their metadata which are relevant, p_paddr (physical address of that segment in memory) and p_vaddr (virtual address of that segment in memory). My observation is, that if an ELF is created with multiple such segments, one can simply load the entire ELF in GDB (when a suitable target is attached, maybe QEMU or via a JTAG probe) or one can use other debug toolchains such as xsct to load all in one go to the target.

Now, for on-target (HW) debugging tools, having one ELF that contains both the different code segments and the debug information for all of them is super useful.

A specific use-case would be on-chip hardware tracing; where one needs the ELF files (in particular the DWARF) to reconstruct the coverage of the execution trace.

Mostly all of the above today can be achieved manually, but that is quiet tedious, and some tools don't play well with loading multiple ELFs.
And, at least from the high-level glance that I have, the data represented in this one ELF would be equivalent to the data in boot IMG. Incidentally, this should also allow to launch the system.elf simply via the -kernel flag from QEMU, no need to specify loader addresses etc.

@moritz-meier I'd be happy for more input from your end.

[alwin-joshy]

My observation is, that if an ELF is created with multiple such segments, one can simply load the entire ELF in GDB (when a suitable target is attached, maybe QEMU or via a JTAG probe) or one can use other debug toolchains such as xsct to load all in one go to the target.

I see, I wasn't aware of GDB being able to do this. My experiments have been using the GDB remote protocol, which seems to talk about things entirely using virtual addresses. I had a look at some of the ELFs that I've been using in for example systems and it seems like the p_paddr = p_vaddr by default. Does this mean that when generating the final ELF, we would read in each of the headers in the PD ELFs passed in to the tool, adjust their physical address to where they will actually be placed, and append the modified headers to our final ELF?