Bug 1218163

Summary: debugging symbols missing in debug kernel
Product: [openSUSE] openSUSE Tumbleweed Reporter: Martin Clauss <martin>
Component: KernelAssignee: openSUSE Kernel Bugs <kernel-bugs>
Status: RESOLVED INVALID QA Contact: E-mail List <qa-bugs>
Severity: Minor    
Priority: P5 - None CC: jslaby, martin, tiwai
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Martin Clauss 2023-12-18 15:39:40 UTC
Hi :)

The debugging symbols are missing in the debug kernel.

Steps to reproduce:

```
# update and install
sudo zypper dup

sudo zypper install kernel-debug

# some additional information
uname -a
Linux opensuse-Tumbleweed-x86-64 6.6.6-1-default #1 SMP PREEMPT_DYNAMIC Mon Dec 11 09:46:39 UTC 2023 (a946a9f) x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release
NAME="openSUSE Tumbleweed"
# VERSION="20231215"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20231215"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20231215"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"
```

After the installation of `kernel-debug` a new kernel appears under `/usr/lib/modules/6.6.6-1-debug/vmlinux.xz`. Decompress the kernel with `xz -d vmlinux.xz`. Show file information about the kernel with `file vmlinux` (note this is the decompressed kernel):

```
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=9d90cec54c25e489b986b1c3a586c994e6b4842a, not stripped
```

`not stripped` is good but `with debug_info` is missing. Also `readelf` cannot find any information:

```
readelf --debug-dump=info vmlinux
<no output here...>
```

The sections look like this:

```
readelf -S vmlinux
There are 48 section headers, starting at offset 0x40779d0:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         ffffffff81000000  00200000
       0000000001000000  0000000000000000  AX       0     0     4096
  [ 2] .rodata           PROGBITS         ffffffff82000000  01200000
       00000000006f7006  0000000000000000  WA       0     0     4096
  [ 3] .pci_fixup        PROGBITS         ffffffff826f7010  018f7010
       0000000000003b70  0000000000000000   A       0     0     16
  [ 4] .tracedata        PROGBITS         ffffffff826fab80  018fab80
       0000000000000078  0000000000000000   A       0     0     1
  [ 5] .printk_index     PROGBITS         ffffffff826fabf8  018fabf8
       0000000000012f70  0000000000000000  WA       0     0     8
  [ 6] __ksymtab         PROGBITS         ffffffff8270db68  0190db68
       00000000000110d0  0000000000000000   A       0     0     4
  [ 7] __ksymtab_gpl     PROGBITS         ffffffff8271ec38  0191ec38
       000000000001206c  0000000000000000   A       0     0     4
  [ 8] __kcrctab         PROGBITS         ffffffff82730ca4  01930ca4
       0000000000005af0  0000000000000000   A       0     0     1
  [ 9] __kcrctab_gpl     PROGBITS         ffffffff82736794  01936794
       0000000000006024  0000000000000000   A       0     0     1
  [10] __ksymtab_strings PROGBITS         ffffffff8273c7b8  0193c7b8
       000000000003a63b  0000000000000001 AMS       0     0     1
  [11] __init_rodata     PROGBITS         ffffffff82776e00  01976e00
       00000000000000b0  0000000000000000   A       0     0     32
  [12] __param           PROGBITS         ffffffff82776eb0  01976eb0
       00000000000031d8  0000000000000000   A       0     0     8
  [13] __modver          PROGBITS         ffffffff8277a088  0197a088
       0000000000000288  0000000000000000  WA       0     0     8
  [14] __ex_table        PROGBITS         ffffffff8277a310  0197a310
       0000000000001c20  0000000000000000   A       0     0     4
  [15] .notes            NOTE             ffffffff8277bf30  0197bf30
       00000000000001d8  0000000000000000   A       0     0     4
  [16] .BTF              PROGBITS         ffffffff8277c108  0197c108
       00000000004c7bec  0000000000000000   A       0     0     1
  [17] .BTF_ids          PROGBITS         ffffffff82c43cf4  01e43cf4
       0000000000000b00  0000000000000000   A       0     0     1
  [18] .data             PROGBITS         ffffffff82e00000  02000000
       00000000003092c0  0000000000000000  WA       0     0     8192
  [19] __bug_table       PROGBITS         ffffffff831092c0  023092c0
       000000000001c5f0  0000000000000000  WA       0     0     1
  [20] .orc_header       PROGBITS         ffffffff831258b0  023258b0
       0000000000000014  0000000000000000   A       0     0     4
  [21] .orc_unwind_ip    PROGBITS         ffffffff831258c4  023258c4
       00000000001a81f4  0000000000000000   A       0     0     1
  [22] .orc_unwind       PROGBITS         ffffffff832cdab8  024cdab8
       000000000027c2ee  0000000000000000   A       0     0     1
  [23] .orc_lookup       NOBITS           ffffffff83549da8  02749da6
       0000000000040004  0000000000000000  WA       0     0     1
  [24] .vvar             PROGBITS         ffffffff8358a000  0278a000
       0000000000001000  0000000000000000  WA       0     0     16
  [25] .data..percpu     PROGBITS         0000000000000000  02800000
       000000000003d000  0000000000000000  WA       0     0     16384
  [26] .init.text        PROGBITS         ffffffff835c8000  029c8000
       00000000000a53e4  0000000000000000  AX       0     0     16
  [27] .altinstr_aux     PROGBITS         ffffffff8366d3e4  02a6d3e4
       00000000000035a6  0000000000000000  AX       0     0     1
  [28] .init.data        PROGBITS         ffffffff83672000  02a72000
       00000000001d6208  0000000000000000  WA       0     0     8192
  [29] .x86_cpu_dev.init PROGBITS         ffffffff83848208  02c48208
       0000000000000028  0000000000000000   A       0     0     8
  [30] .parainstructions PROGBITS         ffffffff83848230  02c48230
       0000000000010870  0000000000000000   A       0     0     8
  [31] .retpoline_sites  PROGBITS         ffffffff83858aa0  02c58aa0
       00000000000095f0  0000000000000000   A       0     0     1
  [32] .return_sites     PROGBITS         ffffffff83862090  02c62090
       00000000000391f4  0000000000000000   A       0     0     1
  [33] .call_sites       PROGBITS         ffffffff8389b288  02c9b288
       00000000000e9958  0000000000000000   A       0     0     1
  [34] .ibt_endbr_seal   PROGBITS         ffffffff83984be0  02d84be0
       000000000000b34c  0000000000000000   A       0     0     1
  [35] .altinstructions  PROGBITS         ffffffff8398ff30  02d8ff30
       00000000000110d6  0000000000000000   A       0     0     1
  [36] .altinstr_re[...] PROGBITS         ffffffff839a1006  02da1006
       000000000000353a  0000000000000000  AX       0     0     1
  [37] .apicdrivers      PROGBITS         ffffffff839a4540  02da4540
       0000000000000030  0000000000000000  WA       0     0     8
  [38] .exit.text        PROGBITS         ffffffff839a4570  02da4570
       0000000000001ebb  0000000000000000  AX       0     0     16
  [39] .smp_locks        PROGBITS         ffffffff839a7000  02da7000
       000000000000a000  0000000000000000   A       0     0     4
  [40] .data_nosave      PROGBITS         ffffffff839b1000  02db1000
       0000000000001000  0000000000000000  WA       0     0     4
  [41] .bss              NOBITS           ffffffff839b2000  02db2000
       000000000044e000  0000000000000000  WA       0     0     4096
  [42] .brk              NOBITS           ffffffff83e00000  02db2000
       0000000000030000  0000000000000000  WA       0     0     1
  [43] .init.scratch     PROGBITS         ffffffff84000000  03400000
       0000000000400000  0000000000000000  WA       0     0     32
  [44] .gnu_debuglink    PROGBITS         0000000000000000  03800000
       0000000000000014  0000000000000000           0     0     4
  [45] .symtab           SYMTAB           0000000000000000  03800018
       0000000000501d38  0000000000000018          46   159778     8
  [46] .strtab           STRTAB           0000000000000000  03d01d50
       0000000000375a61  0000000000000000           0     0     1
  [47] .shstrtab         STRTAB           0000000000000000  040777b1
       000000000000021c  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), l (large), p (processor specific)
```

Expected result:

I would like to have debug symbols when installing the `kernel-debug` package.

Thanks a lot!
Comment 1 Takashi Iwai 2024-01-02 12:00:47 UTC
Have you installed kernel-debug-debuginfo and kernel-debug-debugsource?
Comment 2 Martin Clauss 2024-01-03 13:17:57 UTC
(In reply to Takashi Iwai from comment #1)
> Have you installed kernel-debug-debuginfo and kernel-debug-debugsource?

Sorry, my bad. I did not enable the corresponding repos before so I did not find the packages that you mentioned. I thought this documentation was out of date and that the `kernel-debug` package contains everything I need.

Thanks! :)
Comment 3 Martin Clauss 2024-01-04 14:55:27 UTC
I have one more question :)

The debug kernel is missing some of the section contents (e. g. .rodata or .text):

readelf -S --wide /usr/lib/debug/usr/lib/modules/6.6.7-1-debug/vmlinux.debug
There are 57 section headers, starting at offset 0xb82d388:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .text             NOBITS          ffffffff81000000 200000 1000000 00  AX  0   0 4096
  [ 2] .rodata           NOBITS          ffffffff82000000 200000 6f8006 00  WA  0   0 4096
  [ 3] .pci_fixup        NOBITS          ffffffff826f8010 200000 003b70 00   A  0   0 16
  [ 4] .tracedata        NOBITS          ffffffff826fbb80 200000 000078 00   A  0   0  1
  [ 5] .printk_index     NOBITS          ffffffff826fbbf8 200000 012f80 00  WA  0   0  8
  [ 6] __ksymtab         NOBITS          ffffffff8270eb78 200000 0110dc 00   A  0   0  4
  [ 7] __ksymtab_gpl     NOBITS          ffffffff8271fc54 200000 01206c 00   A  0   0  4
  [ 8] __kcrctab         NOBITS          ffffffff82731cc0 200000 005af4 00   A  0   0  1
  [ 9] __kcrctab_gpl     NOBITS          ffffffff827377b4 200000 006024 00   A  0   0  1
  [10] __ksymtab_strings NOBITS          ffffffff8273d7d8 200000 03a65e 01 AMS  0   0  1
  [11] __init_rodata     NOBITS          ffffffff82777e40 200000 0000b0 00   A  0   0 32
  [12] __param           NOBITS          ffffffff82777ef0 200000 0031d8 00   A  0   0  8
  [13] __modver          NOBITS          ffffffff8277b0c8 200000 000288 00  WA  0   0  8
  [14] __ex_table        NOBITS          ffffffff8277b350 200000 001c20 00   A  0   0  4
  [15] .notes            NOTE            ffffffff8277cf70 197cf70 0001d8 00   A  0   0  4
  [16] .BTF              NOBITS          ffffffff8277d148 197d148 4c7da3 00   A  0   0  1
  [17] .BTF_ids          NOBITS          ffffffff82c44eec 197d148 000b00 00   A  0   0  1
  [18] .data             NOBITS          ffffffff82e00000 1a00000 3092c0 00  WA  0   0 8192
  [19] __bug_table       NOBITS          ffffffff831092c0 1a00000 01c5b4 00  WA  0   0  1
  [20] .orc_header       NOBITS          ffffffff83125874 1a00000 000014 00   A  0   0  4
  [21] .orc_unwind_ip    NOBITS          ffffffff83125888 1a00000 1a8288 00   A  0   0  1
  [22] .orc_unwind       NOBITS          ffffffff832cdb10 1a00000 27c3cc 00   A  0   0  1
  [23] .orc_lookup       NOBITS          ffffffff83549edc 1a00000 040004 00  WA  0   0  1
  [24] .vvar             NOBITS          ffffffff8358a000 1a00000 001000 00  WA  0   0 16
  [25] .data..percpu     NOBITS          0000000000000000 1a00000 03d000 00  WA  0   0 16384
  [26] .init.text        NOBITS          ffffffff835c8000 19c8000 0a5434 00  AX  0   0 16
  [27] .altinstr_aux     NOBITS          ffffffff8366d434 19c8000 0035b8 00  AX  0   0  1
  [28] .init.data        NOBITS          ffffffff83672000 19c8000 1d6228 00  WA  0   0 8192
  [29] .x86_cpu_dev.init NOBITS          ffffffff83848228 19c8000 000028 00   A  0   0  8
  [30] .parainstructions NOBITS          ffffffff83848250 19c8000 010860 00   A  0   0  8
  [31] .retpoline_sites  NOBITS          ffffffff83858ab0 19c8000 009600 00   A  0   0  1
  [32] .return_sites     NOBITS          ffffffff838620b0 19c8000 03920c 00   A  0   0  1
  [33] .call_sites       NOBITS          ffffffff8389b2c0 19c8000 0e98cc 00   A  0   0  1
  [34] .ibt_endbr_seal   NOBITS          ffffffff83984b90 19c8000 00b350 00   A  0   0  1
  [35] .altinstructions  NOBITS          ffffffff8398fee0 19c8000 011082 00   A  0   0  1
  [36] .altinstr_replacement NOBITS          ffffffff839a0f62 19c8000 00351f 00  AX  0   0  1
  [37] .apicdrivers      NOBITS          ffffffff839a4488 19c8000 000030 00  WA  0   0  8
  [38] .exit.text        NOBITS          ffffffff839a44c0 19c8000 001ebb 00  AX  0   0 16
  [39] .smp_locks        NOBITS          ffffffff839a7000 19c8000 00a000 00   A  0   0  4
  [40] .data_nosave      NOBITS          ffffffff839b1000 19c8000 001000 00  WA  0   0  4
  [41] .bss              NOBITS          ffffffff839b2000 19c8000 44e000 00  WA  0   0 4096
  [42] .brk              NOBITS          ffffffff83e00000 19c8000 030000 00  WA  0   0  1
  [43] .init.scratch     NOBITS          ffffffff84000000 19c8000 400000 00  WA  0   0 32
  [44] .debug_aranges    PROGBITS        0000000000000000 197d150 02b8b0 00      0   0 16
  [45] .debug_info       PROGBITS        0000000000000000 19a8a00 5695db9 00      0   0  1
  [46] .debug_abbrev     PROGBITS        0000000000000000 703e7b9 4856f0 00      0   0  1
  [47] .debug_line       PROGBITS        0000000000000000 74c3ea9 162665b 00      0   0  1
  [48] .debug_frame      PROGBITS        0000000000000000 8aea508 2bf1e0 00      0   0  8
  [49] .debug_str        PROGBITS        0000000000000000 8da96e8 3fab91 01  MS  0   0  1
  [50] .debug_line_str   PROGBITS        0000000000000000 91a4279 01946a 01  MS  0   0  1
  [51] .debug_loclists   PROGBITS        0000000000000000 91bd6e3 1a58bc3 00      0   0  1
  [52] .debug_rnglists   PROGBITS        0000000000000000 ac162a6 3898d7 00      0   0  1
  [53] .comment          PROGBITS        0000000000000000 af9fb7d 000056 01  MS  0   0  1
  [54] .symtab           SYMTAB          0000000000000000 af9fbd8 511770 18     55 162442  8
  [55] .strtab           STRTAB          0000000000000000 b4b1348 37bda6 00      0   0  1
  [56] .shstrtab         STRTAB          0000000000000000 b82d0ee 000293 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), l (large), p (processor specific)

The offset is the same (200000) for a lot of those sections...
I would need this information for my purpose. Is there a reason why this information is missing? Is there a way to obtain a "complete" kernel with debug information?

Thanks a lot!

Best
Martin
Comment 4 Jiri Slaby 2024-01-11 07:14:22 UTC
(In reply to Martin Clauss from comment #3)
> Is there a way to obtain a "complete" kernel with
> debug information?

There is no "complete" kernel. You have vmlinux and vmlinux.debug. I.e. they are separate and you and your tools should work with vmlinux (having vmlinux.debug in the same dir).