|
Bugzilla – Full Text Bug Listing |
Created attachment 872774 [details]
VM booting CentOS 7.5 displays this text on boot, then hangs.
Screenshot of VM display after trying to boot CentOS 7.5 - seems to hang while probing the storage device.
Created attachment 872775 [details]
Attempting to boot the System Rescue CD, VM stops for a long time while displaying this text
Attempting to boot the VM using the System Rescue CD starts out normally but hangs with this error message text.
Created attachment 872776 [details]
After booting the VM from the System Rescue CD, the main storage device is not found in /dev/
In this screenshot you can see that the System Rescue CD, after it manages to boot, does not show the main storage device in the /dev/ folder.
Created attachment 872777 [details]
dmesg from boot of System Rescue CD
(Turned out that the VirtIO driver for the Ethernet device was also not working. Added an e1000 device so that I could scp the dmesg text to my desktop.)
This provides more info about the startup errors that were logged as the System Rescue booted.
Created attachment 872778 [details]
Screenshot showing systemd-udevd error message "Waiting for Processes" after attempted poweroff of VM.
After attempting to poweroff the VM the system says it's ready to Power Off but doesn't.
After a few seconds a message appears showing that some systemd-udevd processes are not exiting.
Created attachment 872779 [details]
Screenshot showing systemd-udevd error message "Waiting for Processes" after attempted poweroff of VM.
After attempting to poweroff the VM the system says it's ready to Power Off but doesn't.
After a few seconds a message appears showing that some systemd-udevd processes are not exiting.
Sometime later there are similar messages that appear (the VM has not powered-off as I write this.)
I have the same problem, VM doesn't start after qemu 8.2.0 update I was forced to downgrade qemu to version 8.1.3 to have a working VM again. I suspect the problem may be related to https://bugzilla.opensuse.org/show_bug.cgi?id=1219766#c20 Mmm... weird. I have a Tumbleweed testbox, updated to latest snapshot and everything, and I can run VMs there. I have the same versions of QEMU and udev packages shown in comment 0. Sam Azer, What about the kernel? I have: > # uname -a > Linux Zhaman.fritz.box 6.7.4-1-default #1 SMP PREEMPT_DYNAMIC Tue Feb 6 05:32:37 UTC 2024 (01735a3) x86_64 x86_64 x86_64 GNU/Linux Since you mentioned Leap15.5, I'm installing a VM with that OS right now, and it seems to be working fine. And, yes, I'm using VirtIO and QCOW2 images for disks. Can you share the XML of one of the failing VMs (`virsh dumpxml <domainname>`)? (In reply to Michele Cherici from comment #7) > I have the same problem, VM doesn't start after qemu 8.2.0 update > I was forced to downgrade qemu to version 8.1.3 to have a working VM again. > > I suspect the problem may be related to > https://bugzilla.opensuse.org/show_bug.cgi?id=1219766#c20 > I'm not sure I see how... That seems to be about a Virtualbox shell script failing to create a device file (needed for USB passthrough) with the right permissions. I guess it is possible that the "incompatibility" between glibc and systemd v254 mentioned there can potentially cause other problems (than the one described in that bug, I mean), but I'm not sure this could be one. (In reply to Dario Faggioli from comment #8) Thanks very much for your assistance - much appreciated > Mmm... weird. I have a Tumbleweed testbox, updated to latest snapshot and > everything, and I can run VMs there. > > I have the same versions of QEMU and udev packages shown in comment 0. Sam > Azer, What about the kernel? > > I have: > > # uname -a > > Linux Zhaman.fritz.box 6.7.4-1-default #1 SMP PREEMPT_DYNAMIC Tue Feb 6 05:32:37 UTC 2024 (01735a3) x86_64 x86_64 x86_64 GNU/Linux > Same kernel: Linux m68.local 6.7.4-1-default #1 SMP PREEMPT_DYNAMIC Tue Feb 6 05:32:37 UTC 2024 (01735a3) x86_64 x86_64 x86_64 GNU/Linux I will update the package list in the attachments as I've added a Virtualization repo (in the hope that I might get an update that helps.) The repo url is: https://download.opensuse.org/repositories/Virtualization/openSUSE_Tumbleweed/ > Since you mentioned Leap15.5, I'm installing a VM with that OS right now, > and it seems to be working fine. And, yes, I'm using VirtIO and QCOW2 images > for disks. Yes, I did try that, too. The first one that I tried was shortly after everything stopped working. I was able to build an OpenSUSE LEAP 15.5 system (a small server that I was intending to use to run Apache2 with PHP8.) The system is called PHP8, it installed fine, it rebooted after the installation and I installed some software into it. Then I shut it down and, next day, it had the same problems that I've described above. Funny but true: After installing the packages from the above mentioned Virtualization repo I decided to try again. Same idea exactly but this next try was called test155. It installed correctly, rebooted, worked fine as I installed some packages, shutdown and rebooted fine - no problems. This is now the only VM on my desktop that runs fine. Another thing: I realize now that I wasn't thinking clearly when I created this report. I've been thinking a bit more clearly now so I noticed that VirtIO ethernet ports were also not working on the failed VM's. I started adding e1000 devices to the VM's (for the php8 vm I just changed the device type to e1000,) so that I could get the dmesg boot text and send it to the desktop. I should have tested for the Spice displays and other VirtIO devices (Windows, for example, complains about memory access. When I change the device type for the storage from VirtIO to SATA... Windows complains that the storage has changed and, again, it dies with memory errors. Anyway I now realize that this new QEMU 8.2.0 hypervisor software is not correctly setting up the VirtIO devices (again, there may be many devices that don't work anymore but so far I only tinkered with the Ethernet and the Storage.) The Hypervisor should setup the devices so that the VM OS can detect them. It seems to me that there's something wrong with the current software so the VirtIO device drivers on the VM's are not detecting the devices properly. (I'm sure this is all very obvious for people who know more about QEMU. Sorry if this is not helpful.) > > Can you share the XML of one of the failing VMs (`virsh dumpxml > <domainname>`)? I've prepared a few attachments that I'll upload in a minute: (1) I've updated the package list for my Tumbleweed Desktop system. (2) CentOS75. This is a VM that I built in 2021 - it's been running pretty much 24/7 since then - it died after the upgrade to QEMU 8.2.0. It was then moved to a LEAP 15.5 system where it's working fine again. I will attach the XML file for the VM (ie: sudo virsh dumpxml centos75,) the dmesg boot log from the system rescue CD which does somehow boot off of the CDROM device and (some of the) packages from the System Rescue CD package list. I filtered the package list this way: $ grep -i "virt\|kvm\|qemu\|kernel\|linux\|udev" | sort (3) As above for the PHP8 OpenSUSE LEAP 15.5 mini server VM that I built using QEMU 8.2.0 which worked a bit and then stopped working. Again the dmesg boot text is collected by booting the same System Rescue CD and, therefore, the package list will be the same as for the CentOS75 VM. (4) As above for the test155 OpenSUSE LEAP 15.5 mini server VM that I built using qemu-8.2.0-Virt.1699.963.3.x86_64 from the Virtualization repo. I'm guessing that this VM works because the changes to the hypervisor code are now compatible with the OpenSUSE LEAP 15.5 VirtIO drivers. So I guess the only remaining problem is that the Hypervisor is no longer compatible with the drivers on the older VM's. I did not try yet to create a new Windows VM but I really do need the older Windows and CentOS 7.5 VM's that I have. My best wishes and deepest thanks to all the wonderful people who make OpenSUSE and Tumbleweed possible. You guys are great - Thank you all. Will upload the attachments now. Thanks again. Created attachment 872826 [details]
Package names and version numbers from the desktop Tumbleweed system
I've included the old package names and version numbers at the bottom of the file.
Created attachment 872827 [details]
CentOS 7.5 - dmesg boot text from System Rescue CD
This is a VM that was created in 2021. It stopped working after the upgrade to QEMU 8.2.0. The same qcow2 file was copied to an OpenSUSE LEAP 15.5 system where it runs fine.
As it's not possible to boot this VM directly anymore I tried the System Rescue CD and that does boot. This attachment contains the dmesg boot text from the System Rescue CD. Note that the VirtIO storage, vda, does not appear in /dev/.
There may be problems with other VirtIO devices. Sorry I haven't tested yet.
Created attachment 872828 [details]
CentOS 7.5 - filtered package list from the System Rescue CD
As it's not possible to boot this CentOS 7.5 VM directly anymore I tried the System Rescue CD and that does boot. This attachment contains the filtered package list from the System Rescue CD. The filter was:
$ grep -i "virt\|kvm\|qemu\|kernel\|linux\|udev" packages-list.txt | sort
Created attachment 872829 [details]
CentOS 7.5 - the output from virsh dumpxml centos75
This is the virsh dumpxml centos75 output. Please remember that this VM was created some time ago. I may have tinkered with the ethernet device to be able to scp the System Rescue dmesg boot text and package list (sorry, I can't remember just now.)
Created attachment 872831 [details]
php8 - dmesg boot text from the System Rescue CD
php8 was a VM that I tried to make after the upgrade to QEMU 8.2.0 that broke all the other VM's. I installed a small openSUSE LEAP 15.5 server. After the installation the VM was able to reboot and I was able to install some additional software before shutting it down. Next day, when I tried to run it again, it did not boot like the others.
It did boot the System Rescue CD so I was able to see that the VirtIO storage, usually vda, does not appear in /dev/. This attachment contains the dmesg boot text from the System Rescue CD.
Once again I don't remember about the Ethernet port? Sorry - I might have changed the original port or added a second port using the e1000 driver. I apologize also for not testing other VirtIO devices - perhaps I'll be able to get to that soon.
Created attachment 872832 [details]
php8 - filtered package list from the System Rescue CD (pro forma.)
This attachment contains the filtered package list from the System Rescue CD after booting from the CDROM device of the PHP8 VM.
This file is identical to the CentOS 7.5 package list (as it's the same System Rescue CD.)
Created attachment 872833 [details]
php8 - output from virsh dumpxml php8
The php8 VM was created after the upgrade to QEMU 8.2.0. I was able to install a small server, reboot into the server after the installation and install additional software and then shut it down. Next day it would no longer boot as with the other VM's on the system.
Created attachment 872834 [details] test155 - dmesg boot text Test155 was created after installing qemu-8.2.0-Virt.169???.x86_64 from the Virtualization repo (https://download.opensuse.org/repositories/Virtualization/openSUSE_Tumbleweed/) I was able to install a small OpenSUSE LEAP 15.5 server, reboot, install additional software and shutdown. Since then it continues to boot as before the QEMU update that stopped all the other VM's from booting. (Big mistake: As I write this I realize that I may have created test155 on Friday night or Saturday? I can't remember. I say this because I realize now that there have been many updates from the Virtualization repo over the past few days. Once again I apologize - Please let me know if you want me to make another one? Thanks.) This attachment contains the dmesg boot log from the VM which continues to boot normally. Created attachment 872835 [details]
test155 - Package list
This attachments contains a filtered package list from the test155 OpenSUSE LEAP 15.5 VM which was created recently (ie: After the install of QEMU 8.2.0 and after the switch to the Virtualization Repo.) The packages were filtered using:
sudo rpm -qa | grep -i "virt\|kvm\|qemu\|kernel\|linux\|udev" | sort
Created attachment 872836 [details]
test155 - output from virsh dumpxml test155
The virsh XML definition of this test VM
Ok, I'm testing with the php8's xml you provided. As of now, I have verified that if I lower the amount of RAM of the VM to 2815MB, things seems to work. It also depends on the CPU model used, and on the firmware. Like: - host-passthough: works - !host-passthrough, UEFI: works - !host-passthrough, BIOS, up to 2815MB RAM: works - !host-passthrough, BIOS, more than 2815MB RAM: fails! This is really quite werid... I'll keep looking. (In reply to Dario Faggioli from comment #21) > Ok, I'm testing with the php8's xml you provided. As of now, I have verified > that if I lower the amount of RAM of the VM to 2815MB, things seems to work. > > It also depends on the CPU model used, and on the firmware. > > Like: > - host-passthough: works > - !host-passthrough, UEFI: works > - !host-passthrough, BIOS, up to 2815MB RAM: works > - !host-passthrough, BIOS, more than 2815MB RAM: fails! > > This is really quite werid... I'll keep looking. > Ok, from a quick bisection, it seems like it could be this (upstream) SeaBIOS commit: https://gitlab.com/qemu-project/seabios/-/commit/96a8d130a8c2e908e357ce62cd713f2cc0b0a2eb Sam Azer, can you test modifying the configuration of one of your VMs like follows? So, for example, in php8's xml, I see this: > <cpu mode='custom' match='exact' check='full'> > <model fallback='forbid'>kvm64</model> > <feature policy='require' name='vme'/> > <feature policy='require' name='x2apic'/> > <feature policy='require' name='hypervisor'/> > </cpu> Can you change it to: > <cpu mode="custom" match="exact" check="full"> > <model fallback="forbid">kvm64</model> > <maxphysaddr mode="passthrough"/> > <feature policy="require" name="vme"/> > <feature policy="require" name="x2apic"/> > <feature policy="require" name="hypervisor"/> > </cpu> And tell me if it works? (FTR, it does, for me) It's a rather annoying issue... I think we need to discuss it upstream. I'll try to think if there is also something that I can do in the meanwhile (here in the downstream package, I mean) (In reply to Dario Faggioli from comment #25) > Reverting only that change is not really doable, IMO, as it's tangled with > the other one you're referring to (i.e., > a6ed6b701f0a57db0569ab98b0661c12a6ec3ff8 "limit address space used for pci > devices."). > > I can, temporarily, revert both of them. Or I can downgrade SeaBIOS from > 1.16.3 to 1.16.2 (i.e., what we had for QEMU 8.1.0). > And, yes, I can confirm that reverting these two commits: - https://gitlab.com/qemu-project/seabios/-/commit/a6ed6b701f0a57db0569ab98b0661c12a6ec3ff8 - https://gitlab.com/qemu-project/seabios/-/commit/96a8d130a8c2e908e357ce62cd713f2cc0b0a2eb Can be done, and the resulting firmware seems to work (even without the need to explicitly use host-phys-bits=on / <maxphysaddr mode="passthrough"/>. I'm probably going to do that in the next few hours, unless other ideas come up... (In reply to Dario Faggioli from comment #24) > Sam Azer, can you test modifying the configuration of one of your VMs like > follows? > > So, for example, in php8's xml ... Can you change it to: > > > <cpu mode="custom" match="exact" check="full"> > > <model fallback="forbid">kvm64</model> > > <maxphysaddr mode="passthrough"/> > > <feature policy="require" name="vme"/> > > <feature policy="require" name="x2apic"/> > > <feature policy="require" name="hypervisor"/> > > </cpu> > > And tell me if it works? (FTR, it does, for me) Yes - the PHP8 VM is working now :) Thank you - I don't know what that change means or how I can apply it to the CentOS or Windows VM's but I'll take a look. Thanks again The CentOS and Windows 10 VM's are now working :) Using the XML editor in the Virt-Manager, I've added the line you suggested to the <cpu> section: <maxphysaddr mode="passthrough"/> This seems to be an effective work-around for me, it seems to have solved all my problems. Thanks again for your most valuable efforts. Hi Folks, Thanks again for this workaround. I've removed the Virtualization repo and switched all the packages back to the main Tumbleweed repos. I now have these main packages installed: qemu-8.2.0-2.1.x86_64 kernel-default-6.7.5-1.1.x86_64 The workaround has been applied to two Windows VM's. They now boot normally, no problems. Ditto for the CentOS 7.5 and PHP8 VM's. Interestingly, I noticed that I forgot to set the CPU for test155 to KVM64. When I did this the test155 VM stopped working and I had to add the <maxphysaddr mode="passthrough"/> to the <cpu> definition to get it to work again. Thanks again - I would never have figured this out on my own - Much appreciated. Have a wonderful day, Take care (In reply to Sam Azer from comment #29) > Hi Folks, > > Thanks again for this workaround. > No problem. Actually, thanks to you for the report. None of what you had to do (i.e., adding the line in the XML as a workaround) should have been necessary. This is quite an annoying issue, and we'll try to report it upstream and see if it needs to/can be fixed there as well. For now, as soon as https://build.opensuse.org/request/show/1149630 is accepted and released, you won't need the workaround in place any longer (although, it does not hurt to leave it there). Something unrelated, you probably have your reasons for using 'kvm64' as the CPU Model but, AFAIK, it's one of the worst. In fact, if you manage to move to something else (e.g., host-model or host-passthrough) you should have (at least potentially) higher performance and a better experience overall (although, the entity of the actual benefits are dependent on a number of things). But again, that's independent from the bug, and it shouldn't have been necessary for fixing it. Just FTR, I've reported this upstream: https://lore.kernel.org/qemu-devel/88e2bebdbfa4a9e29cedcfdb08537f2eb9654ce3.camel@suse.com/T/#u This is an autogenerated message for OBS integration: This bug (1219977) was mentioned in https://build.opensuse.org/request/show/1153541 Factory / qemu (In reply to Dario Faggioli from comment #33) > Just FTR, I've reported this upstream: > > https://lore.kernel.org/qemu-devel/88e2bebdbfa4a9e29cedcfdb08537f2eb9654ce3. > camel@suse.com/T/#u > Ok, it turned out that the issue is actually caused by a downstream patch that we carry (and that has already caused a bunch of other nasty issues... but we can't get rid of it, for compatibility reason). Anyway, I've hacked it in a way that it should be less of (not any longer?) a problem. For sure, it fixes this bug and lets us re-introduce all the proper SeaBIOS commits. This is an autogenerated message for OBS integration: This bug (1219977) was mentioned in https://build.opensuse.org/request/show/1154847 Factory / qemu |
Created attachment 872773 [details] Package names and version numbers via RPM -QA udevd in a KVM/QEMU VM no longer displaying VirtIO storage device (backed by QCOW2 file) * Was running a few VM's on my desktop Tumbleweed system - no problems * Installed some QEMU / KVM updates one night (a few nights ago) * Next morning all VM's unusable - nothing boots anymore. * Microsoft VM tries to boot but goes immediately into the blue screen with a memory error * Linux VM's (CentOS, Leap 15.5) crash quickly on boot The same VM's run fine after being copied to a computer running LEAP 15.5. * Booting System Rescue CD off the CD-ROM device for any VM, I noticed that the main storage device, usually /dev/vda, is no longer available (it's the default storage device: a QCOW2 file using the VirtIO driver.) * The System Rescue CD takes a long time to boot (seems like some software is crashing during boot.) When I try to shutdown the VM, the last message appearing on the screen is talking about a number of SystemD-Udevd tasks that are taking a long time to stop. The system hangs at that point. * At the moment the above behavior is absolutely repeatable. Thanks again for all your most excellent efforts.