Lets try to have a specification for what is needed to use as few changes in
the kernel building process and in pahole to generate BTF for all .o files in
the kernel build process and then dedup the combined archive put together by an
unmodified ld (linker), one that is unaware of BTF.

So if we build a kernel with this patch:

$ git diff
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 1d581ba5df66f2b5..ad9e788910636715 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -240,7 +240,7 @@ cmd_ld_single = $(if $(objtool-enabled)$(is-single-obj-m), ; $(LD) $(ld_flags) -
 endif
 
 quiet_cmd_cc_o_c = CC $(quiet_modtag)  $@
-      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \
+      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< && ${PAHOLE} --btf_encode ${PAHOLE_FLAGS} $@ \
                $(cmd_ld_single) \
                $(cmd_objtool)
 
$

We will have BTF and DWARF in all its .o files. At some point we can strip the
DWARF info right after the above step or gcc will generate BTF directly, no DWARF involved.

Then, looking at the first .o in the DWARF section for this vmlinux:

  $ readelf -wi ../build/v6.16.0-rc6+/vmlinux
  Contents of the .debug_info section:
  
    Compilation Unit @ offset 0:
     Length:        0x2ceed (32-bit)
     Version:       5
     Unit Type:     DW_UT_compile (1)
     Abbrev Offset: 0
     Pointer Size:  8
   <0><c>: Abbrev Number: 253 (DW_TAG_compile_unit)
      <12>   DW_AT_language    : 29	(C11)
      <13>   DW_AT_name        : (indirect line string, offset: 0): /home/acme/git/linux/init/main.c
      <17>   DW_AT_comp_dir    : (indirect line string, offset: 0x21): /home/acme/git/build/v6.16.0-rc6+
  <SNIP - also the DW_AT_producer, too long>

We see that init/main.o is the first CU (Compile Unit).

So lets see how many types are in the DWARF info for this vmlinux file:

  $ pahole -F dwarf --sizes /home/acme/git/build/v6.16.0-rc6+/vmlinux | wc -l
  11470
  $

But there are way fewer types in the .BTF section:

  $ pahole -F btf --sizes /home/acme/git/build/v6.16.0-rc6+/vmlinux | wc -l
  589
  $

Because it is a series of .BTF from all its .o files, concatenated by a BTF
unaware linker, and the first, even BTF not having a DW_TAG_compile_unit with
its DW_AT_name in there, is:

  $ pahole -F dwarf --sizes /home/acme/git/build/v6.16.0-rc6+/init/main.o  | wc -l
  589
  $

To confirm that, lets compare the BTF pahole output for vmlinux (where it will
see just the first CU, main.o), with the BTF info for main.o:

  $ pahole -F btf /home/acme/git/build/v6.16.0-rc6+/vmlinux > /tmp/vmlinux
  $ pahole -F btf /home/acme/git/build/v6.16.0-rc6+/init/main.o > /tmp/main.o
  $ diff -u /tmp/vmlinux /tmp/main.o 
  $

So now its a matter of noticing that the .BTF info for vmlinux is not so short:

  $ readelf -SW /home/acme/git/build/v6.16.0-rc6+/vmlinux | grep '\.BTF\>'
    [15] .BTF              PROGBITS        ffffffff82fe9000 21e9000 16ae1449 00   A  0   0  1
  $ readelf -SW /home/acme/git/build/v6.16.0-rc6+/init/main.o | grep '\.BTF\>'
    [218] .BTF              PROGBITS        0000000000000000 0b92f5 03a3d1 00      0   0  1
  $

  $ readelf -SW /home/acme/git/build/v6.16.0-rc6+/init/main.o | grep -w Name
    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  $ 

I.e. 0x16ae1449 > 0x03a3d1, 380.507.209 > 238.545

This total concatenated .BTF section is of course way bigger than a typical /sys/kernel/btf/vmlinux:

  $ ls -la /sys/kernel/btf/vmlinux 
  -r--r--r--. 1 nobody nobody 6585269 Jul 16 10:26 /sys/kernel/btf/vmlinux
  $ ls -lah /sys/kernel/btf/vmlinux 
  -r--r--r--. 1 nobody nobody 6.3M Jul 21 13:00 /sys/kernel/btf/vmlinux
  $

We need to deduplicate it, how to most efficiently do it is now the challenge.

We could first look at the size of the .BTF section in a vmlinux, then try to
load it, ask for its raw size and upon noticing it is more than the .BTF ELF
section size, taking into account its header, etc, calculate the offset into
this section for the next .BTF section and go on loading the next one and
adding its contents to the previous one, like we do iterating DWARF CUs in
pahole.