Bugzilla – Bug 1223440
crash fails to find MAGIC_START and refuses to load vmcore
Last modified: 2024-05-08 17:37:30 UTC
Created attachment 874526 [details] vmcore from SLES-12-SP5 that cannot be opened using crash I have tried opening the vmcore on both SLES-15-SP4 and SLES-12-SP5 without success. # crash vmcore -f vmlinux-4.12.14-122.144-default.gz vmlinux-4.12.14-122.144-default.debug crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [128MB]: patching 83518 gdb minimal_symbol values WARNING: could not find MAGIC_START! WARNING: cannot read linux_banner string crash: vmlinux-4.12.14-122.144-default.debug and vmcore do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details. Crash tried with -d 10: # crash vmcore -d 10 -f vmlinux-4.12.14-122.144-default.gz vmlinux-4.12.14-122.144-default.debug crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. compressed kdump: header->utsname.machine: x86_64 compressed kdump: memory bitmap offset: 4000 diskdump_data: filename: vmcore flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) dfd: 3 ofp: 0 machine_type: 62 (EM_X86_64) header: 171bbf0 signature: "KDUMP " header_version: 6 utsname: sysname: Linux nodename: geroescl1n3 release: 4.12.14-122.144-default version: #1 SMP Mon Dec 12 09:21:50 UTC 2022 (bccc371) machine: x86_64 domainname: (none) timestamp: tv_sec: 640acf09 tv_usec: 0 status: 2 (DUMP_DH_COMPRESSED_LZO) block_size: 4096 sub_hdr_size: 3 bitmap_blocks: 2080 max_mapnr: 34078720 total_ram_blocks: 0 device_blocks: 0 written_blocks: 0 current_cpu: 0 nr_cpus: 24 tasks[nr_cpus]: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 sub_header: 0 (n/a) sub_header_kdump: 171cc00 phys_base: de2000000 dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE) split: 0 start_pfn: (unused) end_pfn: (unused) offset_vmcoreinfo: 12768 (0x31e0) size_vmcoreinfo: 1994 (0x7ca) OSRELEASE=4.12.14-122.144-default NUMBER(SUSE_PRODUCT_CODE)=17564928 PAGESIZE=4096 SYMBOL(init_uts_ns)=ffffffff8a0132a0 SYMBOL(node_online_map)=ffffffff8a22a700 SYMBOL(swapper_pg_dir)=ffffffff8a00a000 SYMBOL(_stext)=ffffffff89000000 SYMBOL(vmap_area_list)=ffffffff8a11c250 SYMBOL(mem_section)=ffffffff8a9c3980 LENGTH(mem_section)=4096 SIZE(mem_section)=32 OFFSET(mem_section.section_mem_map)=0 SIZE(page)=64 SIZE(pglist_data)=171968 SIZE(zone)=1472 SIZE(free_area)=88 SIZE(list_head)=16 SIZE(nodemask_t)=128 OFFSET(page.flags)=0 OFFSET(page._refcount)=28 OFFSET(page.mapping)=8 OFFSET(page.lru)=32 OFFSET(page._mapcount)=24 OFFSET(page.private)=48 OFFSET(page.compound_dtor)=40 OFFSET(page.compound_order)=44 OFFSET(page.compound_head)=32 OFFSET(pglist_data.node_zones)=0 OFFSET(pglist_data.nr_zones)=171232 OFFSET(pglist_data.node_start_pfn)=171240 OFFSET(pglist_data.node_spanned_pages)=171256 OFFSET(pglist_data.node_id)=171264 OFFSET(zone.free_area)=192 OFFSET(zone.vm_stat)=1280 OFFSET(zone.spanned_pages)=120 OFFSET(free_area.free_list)=0 OFFSET(list_head.next)=0 OFFSET(list_head.prev)=8 OFFSET(vmap_area.va_start)=0 OFFSET(vmap_area.list)=48 LENGTH(zone.free_area)=11 SYMBOL(log_buf)=ffffffff8a068d08 SYMBOL(log_buf_len)=ffffffff8a068d00 SYMBOL(log_first_idx)=ffffffff8a898220 SYMBOL(clear_idx)=ffffffff8a8981f4 SYMBOL(log_next_idx)=ffffffff8a898210 SIZE(printk_log)=16 OFFSET(printk_log.ts_nsec)=0 OFFSET(printk_log.len)=8 OFFSET(printk_log.text_len)=10 OFFSET(printk_log.dict_len)=12 LENGTH(free_area.free_list)=5 NUMBER(NR_FREE_PAGES)=0 NUMBER(PG_lru)=5 NUMBER(PG_private)=12 NUMBER(PG_swapcache)=9 NUMBER(PG_swapbacked)=18 NUMBER(PG_slab)=8 NUMBER(PG_hwpoison)=22 NUMBER(PG_head_mask)=32768 NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128 NUMBER(HUGETLB_PAGE_DTOR)=2 NUMBER(phys_base)=59626225664 SYMBOL(init_top_pgt)=ffffffff8a00a000 SYMBOL(node_data)=ffffffff8a226280 LENGTH(node_data)=1024 KERNELOFFSET=8000000 NUMBER(KERNEL_IMAGE_SIZE)=1073741824 NUMBER(sme_mask)=0 CRASHTIME=1678429961 offset_note: 4200 (0x1068) size_note: 10564 (0x2944) notes_buf: 171dc10 num_prstatus_notes: 24 notes[0]: 171dc10 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000000 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000000ef0 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000000 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffffff8a003e20 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[1]: 171dd74 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000001 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000014 R10: 0000000000000f10 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000001 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52801d7e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[2]: 171ded8 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000002 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000000415 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000002 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52801dfe88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[3]: 171e03c (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000003 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000003f4c R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000003 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52801e7e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[4]: 171e1a0 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000004 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000001ea2 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000004 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52801efe88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[5]: 171e304 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000005 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000014 R10: 0000000000001cb4 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000005 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52801f7e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[6]: 171e468 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000006 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 000000000000011b R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000006 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba528644be88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[7]: 171e5cc (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 18653 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: 000000000000038d R14: 0000000000000000 R13: 000000000001eb98 R12: ffffffffc0876ce0 RBP: ffff939f02d6b8a8 RBX: ffff939f02d6b788 R11: ffff939b3a196300 R10: ffffba532be87bf0 R9: ffffba532be87d58 R8: 0000000000000000 RAX: 0000000000540101 RCX: 0000000000200000 RDX: ffff93ba73ce3e80 RSI: 0000000000000000 RDI: ffffffffc0bf3cc0 RIP: ffffffff890e6298 RFLAGS: 0000000000000202 RSP: ffffba532be87bf8 FS_BASE: 00007f3da06ab700 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[8]: 171e730 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 4916 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffff93aa3a26b5b8 R14: ffffba528d0efaa8 R13: ffff93a9fc678140 R12: 0000000000000000 RBP: ffffba528bf0fed0 RBX: ffffba528d6177b0 R11: 0000000000000018 R10: 0000000000000286 R9: 0000000000000000 R8: 0000000000000000 RAX: 0000000000540101 RCX: 0000000000000101 RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffffc0bf3cc0 RIP: ffffffff890e62fd RFLAGS: 0000000000000202 RSP: ffffba528bf0feb0 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[9]: 171e894 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 18649 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: 00000000000001f2 R14: 0000000000000000 R13: 000000000001f237 R12: ffffffffc0876ce0 RBP: ffff939fefe4b1a8 RBX: ffff939fefe4b088 R11: ffffffffffffff9c R10: ffffba532be7fbf0 R9: ffffba532be7fd58 R8: 0000000000000000 RAX: 0000000000000000 RCX: 0000000000280000 RDX: ffff93ba73d23e80 RSI: ffff93aa3fba3e80 RDI: ffffffffc0bf3cc0 RIP: ffffffff890e6285 RFLAGS: 0000000000000246 RSP: ffffba532be7fbf8 FS_BASE: 00007f704c8d5700 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[10]: 171e9f8 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 000000000000000a R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000000209 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 000000000000000a RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba528646be88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[11]: 171eb5c (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 000000000000000b R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 00000000000044af R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 000000000000000b RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba5286473e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[12]: 171ecc0 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 18646 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: 0000000000000397 R14: 0000000000000000 R13: 000000000001eab6 R12: ffffffffc0876ce0 RBP: ffff939b8df39168 RBX: ffff939b8df39048 R11: ffffffffffffff9c R10: ffffffffffffffb7 R9: ffffba532bb2fd58 R8: 0000000000000000 RAX: 0000000000000000 RCX: 0000000000340000 RDX: ffff93aa3fba3e80 RSI: ffff93ba73ce3e80 RDI: ffffffffc0bf3cc0 RIP: ffffffff890e6280 RFLAGS: 0000000000000246 RSP: ffffba532bb2fbf8 FS_BASE: 00007fee9bb5b700 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[13]: 171ee24 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 000000000000000d R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 00000000000052a1 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 000000000000000d RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba5286483e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[14]: 171ef88 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 4845 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffff93aa2f31c080 R14: ffff93aa2f31c080 R13: ffff93aa2f31c080 R12: 0000000000000018 RBP: 000000000001871e RBX: ffff93a39281ed10 R11: 0000000000aaaaaa R10: 00000000000003ff R9: 0000000000051d42 R8: 0000000000000000 RAX: 0000000000000063 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffff93aa3fbd7948 RDI: ffff93aa3fbd7948 RIP: ffffffffc07da651 RFLAGS: 0000000000010296 RSP: ffffba528cad7e80 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[15]: 171f0ec (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 000000000000000f R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000004e34 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 000000000000000f RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba5286493e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[16]: 171f250 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000010 R13: 0000000000000002 R12: 0000000000000010 RBP: 0000000000000003 RBX: 0000000000000004 R11: 0000000000000018 R10: 0000000000000ef7 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000010 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000010 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba528649be88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[17]: 171f3b4 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000011 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000004008 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000011 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52864a3e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[18]: 171f518 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000012 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000000479 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000012 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52864abe88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[19]: 171f67c (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000013 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000003094 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000013 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52864b3e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[20]: 171f7e0 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 18650 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: 0000000000000393 R14: 0000000000000000 R13: 000000000001f237 R12: ffffffffc0876ce0 RBP: ffff939f7517aba8 RBX: ffff939f7517aa88 R11: ffffffffffffff9c R10: ffffffffffffffbb R9: ffffba532a693d58 R8: 0000000000000000 RAX: 0000000000000000 RCX: 0000000000540000 RDX: ffff93aa3fca3e80 RSI: ffff93ba73d23e80 RDI: ffffffffc0bf3cc0 RIP: ffffffff890e6285 RFLAGS: 0000000000000246 RSP: ffffba532a693bf8 FS_BASE: 00007fd6dde71700 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[21]: 171f944 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000015 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 00000000000003fe R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000015 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52864c3e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[22]: 171faa8 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000016 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000000f00 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000016 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52864cbe88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 notes[23]: 171fc0c (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 0 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: ffffffff8a1714c0 R14: 0000000000000017 R13: 0000000000000003 R12: 0000000000000020 RBP: 0000000000000004 RBX: 0000000000000008 R11: 0000000000000018 R10: 0000000000006555 R9: ffffffe91cc49f1d R8: 0000000000000004 RAX: 0000000000000020 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff8a1714c0 RDI: 0000000000000017 RIP: ffffffff89754355 RFLAGS: 0000000000000046 RSP: ffffba52864d3e88 FS_BASE: 0000000000000000 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 snapshot_task: 0 num_qemu_notes: 0 NOTE offsets: 1068 (NT_PRSTATUS) 11cc (NT_PRSTATUS) 1330 (NT_PRSTATUS) 1494 (NT_PRSTATUS) 15f8 (NT_PRSTATUS) 175c (NT_PRSTATUS) 18c0 (NT_PRSTATUS) 1a24 (NT_PRSTATUS) 1b88 (NT_PRSTATUS) 1cec (NT_PRSTATUS) 1e50 (NT_PRSTATUS) 1fb4 (NT_PRSTATUS) 2118 (NT_PRSTATUS) 227c (NT_PRSTATUS) 23e0 (NT_PRSTATUS) 2544 (NT_PRSTATUS) 26a8 (NT_PRSTATUS) 280c (NT_PRSTATUS) 2970 (NT_PRSTATUS) 2ad4 (NT_PRSTATUS) 2c38 (NT_PRSTATUS) 2d9c (NT_PRSTATUS) 2f00 (NT_PRSTATUS) 3064 (NT_PRSTATUS) offset_eraseinfo: 0 (0x0) size_eraseinfo: 0 (0x0) start_pfn_64: (unused) end_pfn_64: (unused) max_mapnr_64: 34078720 (0x2080000) data_offset: 824000 block_size: 4096 block_shift: 12 bitmap: 7f933f049010 bitmap_len: 8519680 max_mapnr: 34078720 (0x2080000) dumpable_bitmap: 7f933e828010 byte: 0 bit: 0 compressed_page: 17609a0 curbufptr: 0 page_cache_hdr[0]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1750990 pg_hit_count: 0 page_cache_hdr[1]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1751990 pg_hit_count: 0 page_cache_hdr[2]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1752990 pg_hit_count: 0 page_cache_hdr[3]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1753990 pg_hit_count: 0 page_cache_hdr[4]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1754990 pg_hit_count: 0 page_cache_hdr[5]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1755990 pg_hit_count: 0 page_cache_hdr[6]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1756990 pg_hit_count: 0 page_cache_hdr[7]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1757990 pg_hit_count: 0 page_cache_hdr[8]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1758990 pg_hit_count: 0 page_cache_hdr[9]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 1759990 pg_hit_count: 0 page_cache_hdr[10]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 175a990 pg_hit_count: 0 page_cache_hdr[11]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 175b990 pg_hit_count: 0 page_cache_hdr[12]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 175c990 pg_hit_count: 0 page_cache_hdr[13]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 175d990 pg_hit_count: 0 page_cache_hdr[14]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 175e990 pg_hit_count: 0 page_cache_hdr[15]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 175f990 pg_hit_count: 0 page_cache_buf: 1750990 evict_index: 0 evictions: 0 accesses: 0 cached_reads: 0 valid_pages: 1740580 readmem: read_diskdump() GETBUF(44 -> 0) NOTE: gnu_debuglink file: vmlinux-4.12.14-122.144-default.debug crc32: 5f2353bd GETBUF(35 -> 1) vmlinux-4.12.14-122.144-default.debug: CRC matches FREEBUF(1) FREEBUF(0) KASLR: _stext from vmlinux-4.12.14-122.144-default.gz_f9ql5h: ffffffff81000000 _stext from vmcoreinfo: ffffffff89000000 relocate: 8000000 (128MB) crash: pv_init_ops exists: ARCH_PVOPS VMCOREINFO: NUMBER(phys_base): 59626225664 -> de2000000 gdb vmlinux-4.12.14-122.144-default.debug GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... GETBUF(288 -> 0) GETBUF(1500 -> 1) WARNING: kernel relocated [128MB]: patching 83518 gdb minimal_symbol values FREEBUF(1) FREEBUF(0) <readmem: ffffffff8a04ede8, KVADDR, "page_offset_base", 8, (FOE), d45a28> <read_diskdump: addr: ffffffff8a04ede8 paddr: dec04ede8 cnt: 8> read_diskdump: paddr/pfn: dec04ede8/dec04e -> cache physical page: dec04e000 <readmem: ffffffff89c2a2e0, KVADDR, "kernel_config_data", 32768, (ROE), 2f297b0> <read_diskdump: addr: ffffffff89c2a2e0 paddr: debc2a2e0 cnt: 3360> read_diskdump: paddr/pfn: debc2a2e0/debc2a -> cache physical page: debc2a000 <read_diskdump: addr: ffffffff89c2b000 paddr: debc2b000 cnt: 4096> read_diskdump: paddr/pfn: debc2b000/debc2b -> cache physical page: debc2b000 <read_diskdump: addr: ffffffff89c2c000 paddr: debc2c000 cnt: 4096> read_diskdump: paddr/pfn: debc2c000/debc2c -> cache physical page: debc2c000 <read_diskdump: addr: ffffffff89c2d000 paddr: debc2d000 cnt: 4096> read_diskdump: paddr/pfn: debc2d000/debc2d -> cache physical page: debc2d000 <read_diskdump: addr: ffffffff89c2e000 paddr: debc2e000 cnt: 4096> read_diskdump: paddr/pfn: debc2e000/debc2e -> cache physical page: debc2e000 <read_diskdump: addr: ffffffff89c2f000 paddr: debc2f000 cnt: 4096> read_diskdump: paddr/pfn: debc2f000/debc2f -> cache physical page: debc2f000 <read_diskdump: addr: ffffffff89c30000 paddr: debc30000 cnt: 4096> read_diskdump: paddr/pfn: debc30000/debc30 -> cache physical page: debc30000 <read_diskdump: addr: ffffffff89c31000 paddr: debc31000 cnt: 4096> read_diskdump: paddr/pfn: debc31000/debc31 -> cache physical page: debc31000 <read_diskdump: addr: ffffffff89c32000 paddr: debc32000 cnt: 736> read_diskdump: paddr/pfn: debc32000/debc32 -> cache physical page: debc32000 WARNING: could not find MAGIC_START! GETBUF(288 -> 0) FREEBUF(0) GETBUF(1024 -> 0) <readmem: ffffffff8a228ec0, KVADDR, "possible", 1024, (ROE), f86fa0> <read_diskdump: addr: ffffffff8a228ec0 paddr: dec228ec0 cnt: 320> read_diskdump: paddr/pfn: dec228ec0/dec228 -> cache physical page: dec228000 <read_diskdump: addr: ffffffff8a229000 paddr: dec229000 cnt: 704> read_diskdump: paddr/pfn: dec229000/dec229 -> cache physical page: dec229000 cpu_possible_mask: cpus: (none) <readmem: ffffffff8a2286c0, KVADDR, "present", 1024, (ROE), f86fa0> <read_diskdump: addr: ffffffff8a2286c0 paddr: dec2286c0 cnt: 1024> read_diskdump: paddr/pfn: dec2286c0/dec228 -> physical page is cached: dec228000 cpu_present_mask: cpus: (none) <readmem: ffffffff8a228ac0, KVADDR, "online", 1024, (ROE), f86fa0> <read_diskdump: addr: ffffffff8a228ac0 paddr: dec228ac0 cnt: 1024> read_diskdump: paddr/pfn: dec228ac0/dec228 -> physical page is cached: dec228000 cpu_online_mask: cpus: (none) <readmem: ffffffff8a2282c0, KVADDR, "active", 1024, (ROE), f86fa0> <read_diskdump: addr: ffffffff8a2282c0 paddr: dec2282c0 cnt: 1024> read_diskdump: paddr/pfn: dec2282c0/dec228 -> physical page is cached: dec228000 cpu_active_mask: cpus: (none) FREEBUF(0) <readmem: ffffffff8a03dbc0, KVADDR, "pv_init_ops", 8, (ROE), 7ffcee0bed08> <read_diskdump: addr: ffffffff8a03dbc0 paddr: dec03dbc0 cnt: 8> read_diskdump: paddr/pfn: dec03dbc0/dec03d -> cache physical page: dec03d000 GETBUF(288 -> 0) FREEBUF(0) GETBUF(288 -> 0) FREEBUF(0) <readmem: ffffffff8a8b1360, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE), 7ffcee0becb8> <read_diskdump: addr: ffffffff8a8b1360 paddr: dec8b1360 cnt: 8> read_diskdump: paddr/pfn: dec8b1360/dec8b1 -> cache physical page: dec8b1000 xtime timespec.tv_sec: 0: Thu Jan 1 05:30:00 1970 <readmem: ffffffff8a0132a4, KVADDR, "init_uts_ns", 390, (ROE), d2bc7c> <read_diskdump: addr: ffffffff8a0132a4 paddr: dec0132a4 cnt: 390> read_diskdump: paddr/pfn: dec0132a4/dec013 -> cache physical page: dec013000 utsname: sysname: nodename: release: version: machine: domainname: base kernel version: 0.12.14 <readmem: ffffffff89c000c0, KVADDR, "accessible check", 8, (ROE|Q), 7ffcee0bc018> <read_diskdump: addr: ffffffff89c000c0 paddr: debc000c0 cnt: 8> read_diskdump: paddr/pfn: debc000c0/debc00 -> cache physical page: debc00000 <readmem: ffffffff89c000c0, KVADDR, "read_string characters", 1499, (ROE|Q), 7ffcee0bc370> <read_diskdump: addr: ffffffff89c000c0 paddr: debc000c0 cnt: 1499> read_diskdump: paddr/pfn: debc000c0/debc00 -> physical page is cached: debc00000 WARNING: cannot read linux_banner string linux_banner: crash: vmlinux-4.12.14-122.144-default.debug and vmcore do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details.
The kernel vmcore is from SLES-12-SP5. However, the SUSE bug filing does not allow to select SLES-12-SP5 as the product. How can this vmcore be opened using crash? I tried with SLES-12-SP5 with kernel-default-4.12.14-122.144.1.x86_64.rpm. But no success. Please help me to open the vmcore using crash.
(In reply to Ganapathi CH from comment #1) > The kernel vmcore is from SLES-12-SP5. However, the SUSE bug filing does not > allow to select SLES-12-SP5 as the product. > > How can this vmcore be opened using crash? > > I tried with SLES-12-SP5 with kernel-default-4.12.14-122.144.1.x86_64.rpm. > But no success. > > Please help me to open the vmcore using crash. The best first choice solution to this is to use a later version of SLES (or openSUSE) as the place you perform the use of crash, e.g. SLES15-SP5 (or the most recent openSUSE version). The process of performing debug of a coredump using a given release of crash on a SUSE product is supported for debug of coredumps from previous versions of SUSE products and the latest crash release on _any_ product is crash 8.0.4, which is substantially more useful and with broader kernel support than the very old crash 7.2.1 However, while that might give you immediate functionality, the same statement combined with your report implies that the kernel on SLES12-SP5 has been updated such that a crash update is required on SLES12-SP5 for normal kernel debugging. My preferred approach would be generalizing the use of crash 8 versions on supported products when this type of thing happens and, internally, the way crash is managed as source code for build was recently separated from source code management for general packages in individual products. However, whether crash 8 can be released for SLES12-SP5 requires verification before consideration and cannot result in a quick solution for you, whereas the approach suggested above can.
The same result with crash 8.0.4. Looks like the crash is unable to find the string 'linux_banner' in vmcore. But why? # crash -d vmcore -f vmlinux-4.12.14-122.144-default.gz usr/lib/debug/boot/vmlinux-4.12.14-122.144-default.debug crash 8.0.4 Copyright (C) 2002-2022 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2022 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... WARNING: could not find MAGIC_START! crash: usr/lib/debug/boot/vmlinux-4.12.14-122.144-default.debug and /proc/kcore do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details.
Thank you for testing and the feedback. I've previously worked bug where crash was not possible to find the banner. It was a long time ago but, as I recall it was a bit strange. I'll start researching it in my Tuesday (2024-04-30) morning.
A way to get crash to start under failure circumstances is to use the --minimal argument on the crash command-line. A large amount of the startup verification is not performed, but the result is a very limited set of functionality in crash. Going back to crash on SLES12-SP5, you should be able to start it in --minimal mode then use something like: crash> set debug 8 crash> rd linux_banner 20 crash> rd -a linux_banner It should demonstrate the path through memory (and the coredump) at debug 8 level. If linux_banner is discoverable the command's should dump the memory containing it and it should identify the kernel and version number, e.g. Linux version 4.12.14-122.144. The failure case I was referring to in comment #4 was that the "type" of the symbol linux_banner caused the banner string itself to be referenced as a pointer and since it's text, treating it as an address means it would be unlikely to represent any address in system memory. The address of linux banner would appear to be something like: 0x4C6C6E7578205665 (or 0x65562078756E6C4C), The first 8 bytes of linux_banner ASCII text treated as pointer to actual linux_banner content. That doesn't appear to be the case from your debug launch of crash but it's worth noting, output from the following might help in minimal mode: crash> info linux_banner You could always expand the compressed kernel to limit working through it's content to direct access rather than de-compress and access. If crash will not start when using --minimal on the command-line then it is likely the coredump (vmcore) is damaged or faulty in some way. That can happen if it is obtained using a method other than kdump, e.g. a xen memory image or vmware's equivalent. There have been some compatibility issues between such memory images and a kdump memory image. If it is a kdump memory image then if storage space permits then it couldn't hurt to use the dump level that includes the most memory in the vmcore image, e.g. dump level of zero (no pages filtered). You can set that in YAST or directly in /etc/sysconfig/kdump with the option KDUMP_DUMPLEVEL="<value>". Even on a system with a huge amount of memory it is a reasonable elimination case for inability to find objects by using KDUMP_DUMPLEVEL="0" As a last resort with the kdump model ensuring dump file format by using KDUMP_DUMPFORMAT="ELF" in /etc/sysconfig/kdump is a worthwhile elimination combined with the use of KDUMP_DUMPLEVEL="0" Beyond that, for debugging it I'd need access to a coredump exhibiting the problem (it would save time to have it and the matching kernel/debuginfo files as well) but let's take that up after your experience with all in the previous paragraphs in this post.
crash --minimal can open vmcore. However, with very minimal commands supported it is not useful for diagnosing the problem. Here is the output of linux_banner from crash --minimal: crash> rd -a linux_banner <addr: ffffffff89c000c0 count: 9223372036854775807 flag: 2080 (KVADDR)> <readmem: ffffffff89c000c0, KVADDR, "ascii", 1, (FOE), 7ffc758f2910> <read_diskdump: addr: ffffffff89c000c0 paddr: debc000c0 cnt: 1> read_diskdump: paddr/pfn: debc000c0/debc00 -> physical page is cached: debc00000 So, the vmcore seems to be good or else the 'minimal' also wouldn't have worked, I guess. But the limited scope of commands is hindering any further progress.
I have also attached crash folder from /var/crash/ so you can check the actual vmcore. I have not attached the vmlinux-4...debug file as it is too big (even after compressing with tar jcf).
I know --minimal mode has very limited functionality, I said it in comment #5. I was asking you to try it only to get the diagnostic information when reading linux_banner. Thank you, Plainly linux_banner is accessible as an identity, though that doesn't of itself demonstrate the coredump has no faults. In fact, I'm not sure I agree the rd -a command worked, I don't see the ASCII dump of memory from the address in linux_banner in comment #6. The command should have ended by outputting something like this: crash> rd -a linux_banner ...the details about what's being parsed due to the debug level, then... c082a020: Linux version 4.8.23-144.232.x86_64 (suseckbuild@hs20-bc2-4.buil c082a05c: dab.suse.com) (gcc version 4.4.4 20100726 (SUSE LLC 4.8.2-09) c082a098: (GCC) ) #1 SMP Tue May 9 20:25:37 CET 2021 That is a mock-up and obviously the address dumped would not be c082a020 in your case, the kernel version, builder and compiler version would be different as well. But, it would be very similar and your output appears to have showed nothing of the content of linux_banner. The identifier linux_banner is known, and it's address, but reading from the location referenced apears to have shown no output. I see the material you attached when you created the bug. I'll investigate using crash in gdb and aim to diagnose it then report back. It may take me two or three days, Debugging crash as it debugs the kernel requires careful consideraion of whether what is being observed is runtime activity in crash and/or interpretation of the core dump as a runtime image of the kernel.
Here's the end of the debug output I get with crash 8.0.4 on openSUSE Tumbleweed using your attached files, de-compressing the kernel binary and using the rpm version of vmlinux-4.12.14-122.144-default.debug that I excepted to match the dumped vmlinux-4.12.14-122.144-default: <readmem: ffffffff8a04ede8, KVADDR, "page_offset_base", 8, (FOE|Q), 55ad639e24c8> <read_diskdump: addr: ffffffff8a04ede8 paddr: dec04ede8 cnt: 8> read_diskdump: paddr/pfn: dec04ede8/dec04e -> cache physical page: dec04e000 read_diskdump/cache_page: descriptor with zero offset found at paddr/pfn/pos: dec04e000/dec04e/4262ba read_diskdump: READ_ERROR: cannot cache page: dec04e000 crash: page incomplete: kernel virtual address: ffffffff8a04ede8 type: "page_offset_base" Crucially, the value of page_offset_base (as I understand it) would be required to read kernel text, including the content of a symbol like linux_banner. From the point of view of diagnosis only: * The readmem fails * The attempt to read the value is from kernel virtual address ffffffff8a04ede8 * 8 bytes from physical address dec04ede8 * dec04ede8 is read into a cached physical page located at dec04e000 * caching the page fails de to a zero offset found at the physical address * Therefore, page_offset_base cannot be read from the cached page I can't even start crash in --minimal mode using this coredump. This does NOT match the behavior you observe, e.g. your debug output for page_offset_base is: <readmem: ffffffff8a04ede8, KVADDR, "page_offset_base", 8, (FOE), d45a28> <read_diskdump: addr: ffffffff8a04ede8 paddr: dec04ede8 cnt: 8> read_diskdump: paddr/pfn: dec04ede8/dec04e -> cache physical page: dec04e000 * The kernel virtual address is the same * The physical address is the same * The read to cache succeeds and subsequent steps of the initialization take place for your use On the basis of this I'm not convinced the attached coredump is complete. It would help to have a checksum of it (e.g. sha256 or sha512) so that I can confirm I am working on the same file as you. Nevertheless, the behavior I see is not entirely different from yours, I see it unable to read a basic memory reference point, you see it unable to read a slightly less significant basic memory reference point but both cases are failures to read a symbol's value. The fact that you were able to start crash in --minimal mode proves nothing about the integrity of the coredump, --minimal excludes a lot of normal initialization tests and tries to permit basic access to the coredump as a memory image. IOW, if the initialization required for crash to debug the coredump as a runtime image is not performed but crash loads (--minimal) it only proves that addressing some objects works in the coredump, not that the coredump has enough integrity for utility with crash. I strongly recommend retrying with a coredump created using KDUMP_DUMPLEVEL="0". It's not suggested as a solution, but rather a way to exclude some potential problem cases that might cause the observed symptoms. Given that I can't reproduce the problem you see but I see a similar problem (failure to read a more significant reference than your observation) I have little to go on other than the appearance that the coredump does not have the required integrity for normal use with crash. That might have happened when it was created, possibly due to the way it was filtered when created, or possibly in the manner it was made available to you. For any of these cases I would always want to start with a coredump created at kdump level zero then work down from that. I would also like to know what happens on a coredump triggered deliberately on an equivalent SLES12-SP5 system that is operating normally.
crash --minimal mode didn't open for me too on openSUSE Tumbleweed. However, it can be opened in --minimal mode on a SLES12-SP5 server.
(In reply to Ganapathi CH from comment #10) > crash --minimal mode didn't open for me too on openSUSE Tumbleweed. However, > it can be opened in --minimal mode on a SLES12-SP5 server. Thanks for the verification. It is still as I said, --minimal demonstrates little about the integrity of the coredump while allowing limited access to it's content. The evidence still suggests to me that the coredump could be bad. For progress as a crash bug it has to be verified that the problem exists in a coredump taken from a system that is not in the process of failing/aborting/oopsing/etc, i.e. a triggered coredump from idle server running normally, and the starting point has to be that said coredump has no content filtered (kdump level zero). If that coredump can be opened then it is likely the experience you are having with the attached coredump is not a crash bug and alterative configuration of kdump might help in the real problem case.
I'm working on a test VM to try to experiment with kdump/crash. I recommend you perform your own test case as described in comment #11 but I'll post my results if they are needed.
I also need to make note that the current kernel version for SLES12-SP5 is 4.12.14-122.201.1 and that the problem is reported for kernel 4.12.14-122.144 which is from December 2022 as I understand it and substantially outdated.
(In reply to David Mair from comment #12) > I'm working on a test VM to try to experiment with kdump/crash. I recommend > you perform your own test case as described in comment #11 but I'll post my > results if they are needed. OK. FOR THE RECORD: I cannot reproduce the reported problem using crash 7.2.1 on SLES12-SP5 patched to-date, using a triggered kdump coredump from an idle but operational SLES12-SP5 server with the current maintenance kernel version of 4.12.14-122.201. Crash loads the coredump just fine and reaches the crash > console prompt without error. The kdump level used was the install default of 23, which I understand excludes only the following: Pages Filled with Zeros Cache Pages Cache Private Pages Free Pages The dump format was LZO Compressed which makes the only difference between the format of my coredump and the supplied one is that I included User Data Pages. Which, as I mentioned, is the default filtering. My kdump on a 6GB host was configured with the following startup configuration: Low Memory: 72MiB High Memory: 165MiB The coredump triggered from the idle, and not failing server with 6GiB of memory has a compressed size of 111MiB using the above detailed enabled filters and compression.
I repeated the test with all page type filters enabled (which is KDUMP_DUMP_LEVEL of 31). It's still a patched to-date system, but now using the same KDUMP_DUMP_LEVEL as the reported problem coredump. I still had no problem opening the newly created coredump using crash. The coredump this time was 56MiB. It's an extremely crude value but, the ratio of physical memory to coredump size I see for a LZO compressed coredump is as-follows by KDUMP_DUMP_LEVEL: KDUMP_DUMP_LEVEL 23: 55.6 to 1 KDUMP_DUMP_LEVEL 23: 110.5 to 1 If this was a credibly consistent ratio across SLES12-SP5 systems (which is NOT really a fact but is a reasonable assumption for guidance) then, I would estimate the physical memory is between 24GiB and 32GiB on the system the supplied problem 233MiB coredump was from. I will now transfer the problem coredump to my test VM to see if there is any difference trying crash with it on a patched to date SLES12-SP5 (though with the problem system's kernel binary and debuginfo).
...and using the supplied problem coredump on a patched to-date (May 2024) SLES12-SP5 where the coredump is from a December 2022 kernel (version 4.12.14-122.144) using the matching kernel 4.12.14-122.144 binary and kernel 4.12.14-122.144 debuginfo for the dump of the kernel 4.12.14-122.144 system I observe the reported problem, the value of the linux_banner symbol is output as a 64-bit zero integer: fffffffff89c00c0: 0000000000000000 This is not valid, the page was read from the coredump and the value at the location in the coredump representing the location of the linux_banner symbol is zero and not either the text of the linux_banner or a pointer address to the linux_banner text. Going further, using: crash> rd 0xfffffffff89c00c0 4096 to browse the page supposedly containing linux_banner. Every byte in the page has zero value, i.e. it is a zero page. The kdump dump level was configured to exclude the byte content of zero pages from the coredump, therefore the real content of the page containing linux_banner was either damaged when inserted ito the coredump by kdump or improperly resolved by kdump such that a zero page was that was not the page contianing linux_banner was treated as that page and flagged in the coredump as a zero page. None of these scenarios suggest a problem in crash and further support my opinion that the coredump is invalid and should not be used with crash. Combine these with the fact that I can dump the core on a stable sytem with dump level 23 and level 31 then use it with crash using a patched to date SLES12-SP5. * I recommend the supplied coredump be considered invalid based on the above analysis * For progress I strongly recommend patching the problem system to-date and attempt to reproduce the problem underlying the creation of the coredump * If the underlying problem that caused the core to be dumped is reproduced on a patched to-date SLES12-SP5 and the coredump still cannot be opened by crash then it is likely the underlying cause corrupts memory sufficiently that the a dumped core is of little use * If that is observed then I strongly recommend that the kdump dump level be set to zero (no filtered pages, all memory in the coredump), the underlying problem be reproduced again and the unfiltered cordump be considered * If crash still fails to load that unfiltered coredump then I strongly recommend that a coredump be triggered on an idle instance of the system exhibiting the underlying problem (i.e. get a coredump of a working system that the underlying problem can be reproduced on). This should be done as soon as possible after linux boot is complete (i.e. a system that's just up and done almost nothing but boot to idle state) * If crash does not load that coredump we should continue talking but I would suspect that the coredump of a just booted, idle, patched to-date SLES12-SP5 will not be damaged, matching my observations on a test system and if you can use that coredump with crash then it is improbable that crash has a bug causing the reported problem but something causes the coredump to be invalid when created.
Thanks for the analysis and the suggestions. Yes, I agree that is unlikely to be a crash issue; maybe kdump or some other issue. That coredump was from a customer. I have pinged the customer. Haven't heard back. If the customer responds I will suggest getting the coredump with kdump level 0 and see if I can open that. If not, I will ask for a triggered coredump on the idle server and verify. You can close this bug for now. If the customer responds and if I get a coredump as suggested but still cannot open I will reach out again. Hope that is fine.