Bugzilla – Bug 1217074
crash: invalid structure member offset: kmem_cache_s_num
Last modified: 2023-12-01 14:37:34 UTC
We did standard test scenario for kdump and crash analysis, but analysis on 15-SP6 fails with "crash: invalid structure member offset: kmem_cache_s_num" # echo exit | crash `ls -1t /var/crash/*/vmcore | head -n1` /boot/vmlinux-6.4.0-150600.1-default.gz crash 7.3.1 Copyright (C) 2002-2021 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2021 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. NOTE: stdin: not a tty GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [526MB]: patching 217087 gdb minimal_symbol values WARNING: kernel version inconsistency between vmlinux and dumpfile please wait... (gathering kmem slab cache data) crash: invalid structure member offset: kmem_cache_s_num FILE: memory.c LINE: 9619 FUNCTION: kmem_cache_init() [/usr/bin/crash] error trace: 558b352c3106 => 558b35299777 => 558b353673e4 => 558b3536735d content of /var/crash can be downloaded from http://10.100.12.105:8000/15-sp6-crash/crash.tgz
Another issue was observer related to same crash scenario on aarch64: NOTE: stdin: not a tty WARNING: VA_BITS: calculated: 46 vmcoreinfo: 48 GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-unknown-linux-gnu"... WARNING: kernel relocated [73967149MB]: patching 196331 gdb minimal_symbol values crash: seek error: kernel virtual address: ffffc68ae3c55600 type: "kernel_config_data" WARNING: cannot read kernel_config_data crash: seek error: kernel virtual address: ffffc68ae539eb10 type: "possible" WARNING: cannot read cpu_possible_map crash: seek error: kernel virtual address: ffffc68ae539ebd0 type: "present" WARNING: cannot read cpu_present_map crash: seek error: kernel virtual address: ffffc68ae539eab0 type: "online" WARNING: cannot read cpu_online_map crash: seek error: kernel virtual address: ffffc68ae539ec38 type: "active" WARNING: cannot read cpu_active_map crash: seek error: kernel virtual address: ffffc68ae576d9a8 type: "shadow_timekeeper xtime_sec" crash: seek error: kernel virtual address: ffffc68ae56ae4e8 type: "init_uts_ns"
(In reply to Petr Cervinka from comment #1) > Another issue was observer related to same crash scenario on aarch64: > > > NOTE: stdin: not a tty > > WARNING: VA_BITS: calculated: 46 vmcoreinfo: 48 > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "aarch64-unknown-linux-gnu"... > > WARNING: kernel relocated [73967149MB]: patching 196331 gdb minimal_symbol > values > crash: seek error: kernel virtual address: ffffc68ae3c55600 type: > "kernel_config_data" > WARNING: cannot read kernel_config_data > crash: seek error: kernel virtual address: ffffc68ae539eb10 type: "possible" > WARNING: cannot read cpu_possible_map > crash: seek error: kernel virtual address: ffffc68ae539ebd0 type: "present" > WARNING: cannot read cpu_present_map > crash: seek error: kernel virtual address: ffffc68ae539eab0 type: "online" > WARNING: cannot read cpu_online_map > crash: seek error: kernel virtual address: ffffc68ae539ec38 type: "active" > WARNING: cannot read cpu_active_map > crash: seek error: kernel virtual address: ffffc68ae576d9a8 type: > "shadow_timekeeper xtime_sec" > crash: seek error: kernel virtual address: ffffc68ae56ae4e8 type: > "init_uts_ns" Also broken in other places --- WARNING: VA_BITS: calculated: 47 vmcoreinfo: 48 GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-unknown-linux-gnu"... WARNING: kernel relocated [51248248MB]: patching 196641 gdb minimal_symbol values crash: seek error: kernel virtual address: ffffb0e048795910 type: "kernel_config_data" WARNING: cannot read kernel_config_data crash: seek error: kernel virtual address: ffffb0e049eeeb10 type: "possible" WARNING: cannot read cpu_possible_map crash: seek error: kernel virtual address: ffffb0e049eeebd0 type: "present" WARNING: cannot read cpu_present_map crash: seek error: kernel virtual address: ffffb0e049eeeab0 type: "online" WARNING: cannot read cpu_online_map crash: seek error: kernel virtual address: ffffb0e049eeec38 type: "active" WARNING: cannot read cpu_active_map crash: seek error: kernel virtual address: ffffb0e04a2bf168 type: "shadow_timekeeper xtime_sec" crash: seek error: kernel virtual address: ffffb0e04a1ff0a8 type: "init_uts_ns" crash: /var/tmp/vmlinux-6.4.0-150600.2-default.gz_aeVCl7 and /var/crash/2023-11-17-05-16/vmcore do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details. ---
(In reply to Petr Cervinka from comment #0) > We did standard test scenario for kdump and crash analysis, but analysis on > 15-SP6 fails with "crash: invalid structure member offset: > kmem_cache_s_num" [ ... ] > WARNING: kernel relocated [526MB]: patching 217087 gdb minimal_symbol values > WARNING: kernel version inconsistency between vmlinux and dumpfile Can you please double-check that the dumpfile is really consistent with the vmlinux?
Let's add David Mair. AFAIK, he has upgraded crash to the latest version in 15sp6.
(In reply to Jiri Kosina from comment #3) > Can you please double-check that the dumpfile is really consistent with the > vmlinux? vmlinux file was not changed during the test. It is same all the time for dump and follow up analysis.
Also, I would recommend to update crash to 8.0.4 to same version as it in Tumbleweed. Version 8.0.4 fixed many issues with crash there (like https://bugzilla.suse.com/show_bug.cgi?id=1190434 which is not just for usr merge but fixed few follow issues).
I haven't been able to submit for 15-SP6 yet, for several reasons. It's in Factory head built for Tumbleweed. The crash-gcore package probably won't install due to different kernel versions in Tumbleweed and 15-SP6. Otherwise, the Tumbleweed 8.0.4 version is adequate for testing the reported problem.
David, we submitted for you: https://build.suse.de/request/show/313995
Setting it as resolved. One can always reopen if suspects something is not going