|
Bugzilla – Full Text Bug Listing |
| Summary: | [SLES15SP4][151.1][Kernel] VM panic on VM '-cpu Skylake-Client' | ||
|---|---|---|---|
| Product: | [openSUSE] PUBLIC SUSE Linux Enterprise Server 15 SP4 | Reporter: | Richard Fan <richard.fan> |
| Component: | Virtualization: Other | Assignee: | E-mail List <kvm-bugs> |
| Status: | NEW --- | QA Contact: | Antoine Ginies <aginies> |
| Severity: | Normal | ||
| Priority: | P2 - High | CC: | bchou, bpetkov, claudio.fontana, dfaggioli, fweisbecker, jfehlig, llzhao, mawerner, meissner, richard.fan, roy.cai, tiwai, xiaojing.liu, ysun |
| Version: | PublicRC-202204 | Flags: | dfaggioli:
needinfo?
(richard.fan) |
| Target Milestone: | --- | ||
| Hardware: | x86-64 | ||
| OS: | SLES 15 | ||
| Whiteboard: | |||
| Found By: | --- | Services Priority: | |
| Business Priority: | Blocker: | --- | |
| Marketing QA Status: | --- | IT Deployment: | --- |
Guest panic after booting up [in 2-3 minutes]
susetest login: r[ 74.858731][ T0] BUG: kernel NULL pointer dereference, address: 0000000000000068
[ 74.865003][ T0] #PF: supervisor read access in kernel mode
[ 74.869164][ T0] #PF: error_code(0x0000) - not-present page
[ 74.873730][ T0] PGD 0 P4D 0
[ 74.876171][ T0] Oops: 0000 [#1] PREEMPT SMP PTI
[ 74.879726][ T0] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.14.21-150400.22-default #1 SLE15-SP4 0b6a6578ade2de5c4a0b916095dff44f76ef1704
[ 74.888674][ T0] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
[ 74.897157][ T0] RIP: 0010:clockevents_program_min_delta+0x1a/0xf0
[ 74.901774][ T0] Code: 0f 1f 44 00 00 31 d2 e9 74 ff ff ff 0f 1f 40 00 0f 1f 44 00 00 41 55 41 54 49 c7 c4 ad 83 99 90 55 53 48 89 fb bd 03 00 00 00 <4c> 8b 6b 28 e8 ed 45 ff ff 4c 01 e8 83 7b 38 01 48 89 43 18 0f 84
[ 74.915432][ T0] RSP: 0018:ffffbcbb80097e38 EFLAGS: 00010086
[ 74.919645][ T0] RAX: ffffa038bdd60000 RBX: 0000000000000040 RCX: 0000000000000000
[ 74.925170][ T0] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000040
[ 74.930722][ T0] RBP: 0000000000000003 R08: 0000000000000343 R09: 0000000000017475
[ 74.936243][ T0] R10: ffffbcbb80143d58 R11: 0000000000012ff7 R12: ffffffff909983ad
[ 74.941788][ T0] R13: ffffa038bdd21e00 R14: ffffffff8f75338e R15: ffffffff8f6833d4
[ 74.947309][ T0] FS: 0000000000000000(0000) GS:ffffa038bdd00000(0000) knlGS:0000000000000000
[ 74.953517][ T0] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 74.958214][ T0] CR2: 0000000000000068 CR3: 0000000013aec000 CR4: 0000000000150ee0
[ 74.963896][ T0] Call Trace:
[ 74.966309][ T0] <TASK>
[ 74.968400][ T0] hrtimer_start_range_ns+0x136/0x300
[ 74.972257][ T0] ? rcu_eqs_enter.constprop.78+0xc0/0xe0
[ 74.976257][ T0] ? default_idle+0xa/0x10
[ 74.979331][ T0] ? default_idle_call+0x2d/0xe0
[ 74.982840][ T0] ? do_idle+0x1ec/0x2d0
[ 74.985905][ T0] ? cpu_startup_entry+0x19/0x20
[ 74.989451][ T0] ? start_secondary+0x11c/0x160
[ 74.992964][ T0] ? secondary_startup_64_no_verify+0xc2/0xcb
[ 74.997327][ T0] </TASK>
[ 74.999455][ T0] Modules linked in: af_packet nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security iscsi_ibft iscsi_boot_sysfs ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter bpfilter intel_rapl_msr intel_rapl_common intel_pmc_core kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt intel_pmc_bxt iTCO_vendor_support snd_hda_codec_generic ledtrig_audio qxl snd_hda_intel drm_ttm_helper snd_intel_dspcfg snd_intel_sdw_acpi ttm snd_hda_codec snd_hda_core snd_hwdep snd_pcm aesni_intel crypto_simd cryptd snd_timer drm_kms_helper i2c_i801 i2c_smbus pcspkr snd lpc_ich joydev soundcore mfd_core virtio_balloon cec virtio_net rc_core net_failover syscopyarea
[ 75.000163][ T0] failover sysfillrect sysimgblt fb_sys_fops button drm fuse configfs ip_tables x_tables hid_generic usbhid xhci_pci xhci_pci_renesas virtio_blk xhci_hcd ahci libahci libata usbcore serio_raw virtio_pci virtio_pci_modern_dev qemu_fw_cfg btrfs blake2b_generic libcrc32c crc32c_intel xor raid6_pq sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod virtio_rng
[ 75.085402][ T0] Supported: Yes
[ 75.087859][ T0] CR2: 0000000000000068
[ 75.090750][ T0] ---[ end trace 67c52ba9e71e569d ]---
[ 75.094542][ T0] RIP: 0010:clockevents_program_min_delta+0x1a/0xf0
[ 75.099136][ T0] Code: 0f 1f 44 00 00 31 d2 e9 74 ff ff ff 0f 1f 40 00 0f 1f 44 00 00 41 55 41 54 49 c7 c4 ad 83 99 90 55 53 48 89 fb bd 03 00 00 00 <4c> 8b 6b 28 e8 ed 45 ff ff 4c 01 e8 83 7b 38 01 48 89 43 18 0f 84
[ 75.112768][ T0] RSP: 0018:ffffbcbb80097e38 EFLAGS: 00010086
[ 75.117148][ T0] RAX: ffffa038bdd60000 RBX: 0000000000000040 RCX: 0000000000000000
[ 75.122804][ T0] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000040
[ 75.128338][ T0] RBP: 0000000000000003 R08: 0000000000000343 R09: 0000000000017475
[ 75.134072][ T0] R10: ffffbcbb80143d58 R11: 0000000000012ff7 R12: ffffffff909983ad
[ 75.139607][ T0] R13: ffffa038bdd21e00 R14: ffffffff8f75338e R15: ffffffff8f6833d4
[ 75.145145][ T0] FS: 0000000000000000(0000) GS:ffffa038bdd00000(0000) knlGS:0000000000000000
[ 75.151348][ T0] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 75.155914][ T0] CR2: 0000000000000068 CR3: 0000000013aec000 CR4: 0000000000150ee0
[ 75.161473][ T0] Kernel panic - not syncing: Attempted to kill the idle task!
[ 75.167479][ T0] Kernel Offset: 0xe600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 75.175563][ T0] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
HOST CPU info:
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 288
On-line CPU(s) list: 0-287
Thread(s) per core: 4
Core(s) per socket: 72
Socket(s): 1
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 133
Model name: Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz
Stepping: 0
CPU MHz: 1498.916
CPU max MHz: 1600.0000
CPU min MHz: 1000.0000
BogoMIPS: 2992.90
Virtualization: VT-x
L1d cache: 2.3 MiB
L1i cache: 2.3 MiB
L2 cache: 36 MiB
NUMA node0 CPU(s): 0-17,72-89,144-161,216-233
NUMA node1 CPU(s): 18-35,90-107,162-179,234-251
NUMA node2 CPU(s): 36-53,108-125,180-197,252-269
NUMA node3 CPU(s): 54-71,126-143,198-215,270-287
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT mitigated
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe sysca
ll nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq
dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave av
x f16c rdrand lahf_lm abm 3dnowprefetch ring3mwait cpuid_fault epb pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adj
ust bmi1 avx2 smep bmi2 erms avx512f rdseed adx avx512pf avx512er avx512cd xsaveopt dtherm ida arat pln pts avx512_vpopcntdq
avx512_4vnniw avx512_4fmaps
(In reply to Richard Fan from comment #0) > The issue can only be seen on my test setup, seems it has something to do > with CPU model. > > KVM Host:SUSE Linux Enterprise Server 15 SP3 > > qemu:5.2.0 > Libvirt: 7.1.0 > > Guest:SLES15SP4 151.1 textmode > Is this with the latest updates (for SP3) and with the latest available image (for SP4)? Do you happen to know if other combinations of hosts and guests work (e.g., SP4 on SP4, SP3 on SP3)? What's the guest doing when crashing? Just sitting idle? Do you have a libvirt xml config file for this VM? (In reply to Dario Faggioli from comment #2) > (In reply to Richard Fan from comment #0) > > The issue can only be seen on my test setup, seems it has something to do > > with CPU model. > > > > KVM Host:SUSE Linux Enterprise Server 15 SP3 > > > > qemu:5.2.0 > > Libvirt: 7.1.0 > > > > Guest:SLES15SP4 151.1 textmode > > > Is this with the latest updates (for SP3) and with the latest available > image (for SP4)? > I don't think so, the kvm host has only SP3 GM version, however, bad news is that, the test machine is not accessible due to some AC failures in the lab > Do you happen to know if other combinations of hosts and guests work (e.g., > SP4 on SP4, SP3 on SP3)? > I have another host with SP3 installed as well, and I can't reproduce the panic issue with the same VM image > What's the guest doing when crashing? Just sitting idle? Yes, nothing to do to reproduce the issue, just wait for a few minutes, it can panic soon > > Do you have a libvirt xml config file for this VM? I will try to give you the access information once the kvm host is back (In reply to Richard Fan from comment #3) > (In reply to Dario Faggioli from comment #2) > > Is this with the latest updates (for SP3) and with the latest available > > image (for SP4)? > > > I don't think so, the kvm host has only SP3 GM version, > Ok, next question then: is that on purpose, and does it have to stay GM? Or can it be updated (when it'll work again, of course)? Usually, updating to latest MU is required, as the problem might already have been fixed between GA and now... > however, bad news is > that, the test machine is not accessible due to some AC failures in the lab > Oh My... :-/ > > Do you happen to know if other combinations of hosts and guests work (e.g., > > SP4 on SP4, SP3 on SP3)? > > > I have another host with SP3 installed as well, and I can't reproduce the > panic issue with the same VM image > Mmm... ok, an host-hardware specific guest kernel crash. Interesting! :-O > > Do you have a libvirt xml config file for this VM? > I will try to give you the access information once the kvm host is back > Right. I'll set this to needinfo. Ping the bug when the box is on again, and more information can be retrieved from it. (In reply to Richard Fan from comment #3) > I don't think so, the kvm host has only SP3 GM version We've fixed a lot of bugs in the virt stack since SP3 was released. Please consider updating tests hosts to latest available packages to avoid encountering issues already reported/fixed. I'm not saying this bug has already been reported and fixed, but in general we'd prefer if bugs are reported against the latest available updates. Thanks! (In reply to James Fehlig from comment #5) > (In reply to Richard Fan from comment #3) > > I don't think so, the kvm host has only SP3 GM version > > We've fixed a lot of bugs in the virt stack since SP3 was released. Please > consider updating tests hosts to latest available packages to avoid > encountering issues already reported/fixed. I'm not saying this bug has > already been reported and fixed, but in general we'd prefer if bugs are > reported against the latest available updates. Thanks! Thanks James, I will keep this in mind. Any update here? Are you able to access the host again? If so, can the issue still be reproduced with the latest SLE15 SP3 updates applied to the host? (In reply to James Fehlig from comment #7) > Any update here? Are you able to access the host again? If so, can the issue > still be reproduced with the latest SLE15 SP3 updates applied to the host? Sorry, the server is still not accessible due to AC failure |
The issue can only be seen on my test setup, seems it has something to do with CPU model. KVM Host:SUSE Linux Enterprise Server 15 SP3 qemu:5.2.0 Libvirt: 7.1.0 Guest:SLES15SP4 151.1 textmode qemu backend process: /usr/bin/qemu-system-x86_64 -name guest=sle15sp4-fips-rfan,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-16-sle15sp4-fips-rfan/master-key.aes -machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -cpu Skylake-Client,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,avx512f=on,avx512pf=on,avx512er=on,avx512cd=on,umip=on,avx512-vpopcntdq=on,avx512-4vnniw=on,avx512-4fmaps=on,arch-capabilities=on,pdpe1gb=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,pcid=off,hle=off,invpcid=off,rtm=off,mpx=off,smap=off,xsavec=off,xgetbv1=off -m 2048 -object memory-backend-ram,id=pc.ram,size=2147483648 -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 7886ab28-6da2-4b5a-8459-254bc0daa4cb -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=42,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/sle-15-SP4-x86_64-151.1-textmode@64bit.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a3:70:0f,bus=pci.1,addr=0x0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=37,server=on,wait=off -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5903,addr=127.0.0.1,disable-ticketing=on,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.6,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on