Bug 1188105

Summary: virtualbox after last updates of tumbleweed did not run vbox machines
Product: [openSUSE] openSUSE Tumbleweed Reporter: Igor Kuznetsov <virex>
Component: Virtualization:OtherAssignee: Larry Finger <Larry.Finger>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P5 - None CC: Andreas.Stieger, dimstar, ingo.goeppert+suse, jimoe, Larry.Finger, oliver, Sauerlandlinux, sebix+novell.com, suse.junky, timothyadams, viksvv, wb_jerry
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: Vbox.log

Description Igor Kuznetsov 2021-07-08 07:53:33 UTC
Created attachment 850847 [details]
Vbox.log

virtualbox after last updates of tumbleweed did not run vbox machines

try last version from virtaulization repo, from repo tumbleweed, from leap 15.2 and from 15.3 same error
Comment 1 Andreas Stieger 2021-07-08 08:48:53 UTC
Last change was 2 months ago:
https://build.opensuse.org/request/show/894688
So maybe something else broke.
Comment 2 Andreas Stieger 2021-07-08 08:49:07 UTC
*** Bug 1188108 has been marked as a duplicate of this bug. ***
Comment 3 Larry Finger 2021-07-08 19:31:36 UTC
This problem is due to a change in kernel 5.13, which turned on split-lock detection. Whenever an executing program attempts to lock a memory object that crosses a cache-line boundary, an exception is triggered. See https://lwn.net/Articles/816918/ for a further explanation. The article covers VMX, but it also applies to VirtualBox.

A workaround is to supply the option 'split_lock_detect=none' as a boot-time parameter. This can be set using YaST under the System->BootLoader option. Using the 'Kernel Parameters' tab, add the option there.
Comment 4 Sebastian Wagner 2021-07-09 08:17:16 UTC
> A workaround is to supply the option 'split_lock_detect=none' as a boot-time parameter. This can be set using YaST under the System->BootLoader option. Using the 'Kernel Parameters' tab, add the option there.

This workaround does not work for me, with kernel 5.13.0-1-default. Using the older kernel version 5.12.13-1-default works for me.

Possibly this bug is a duplicate of https://bugzilla.opensuse.org/show_bug.cgi?id=1188089
Comment 5 Larry Finger 2021-07-09 15:00:49 UTC
*** Bug 1188089 has been marked as a duplicate of this bug. ***
Comment 6 Bit Juggler 2021-07-11 09:11:11 UTC
(In reply to Larry Finger from comment #3)

> A workaround is to supply the option 'split_lock_detect=none' as a boot-time
> parameter. This can be set using YaST under the System->BootLoader option.
> Using the 'Kernel Parameters' tab, add the option there.

This workaround did not work for me.

My system:

Operating System: openSUSE Tumbleweed 20210708

KDE Plasma Version: 5.22.2
KDE Frameworks Version: 5.83.0
Qt Version: 5.15.2
Kernel Version: 5.13.0-1-default (64-bit)

Graphics Platform: X11
Processors: 8 × Intel® Core™ i7-7820HK CPU @ 2.90GHz
Memory: 62.7 GiB of RAM
Graphics Processor: Mesa DRI Intel® HD Graphics 630 

virtualbox: 6.1.22-3.3

> cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.13.0-1-default root=UUID=45f155f5-1765-4136-06bb-34b31c4314c8 acpi_backlight=vendor split_lock_detect=none mitigations=auto
>
Comment 7 Larry Finger 2021-07-13 14:57:59 UTC
This problem is not due to split lock detection as I thought at first.

The real cause is that openSUSE turned on 5-level paging in the 5.13.0 kernel, but VirtualBox cannot handle this feature. 5-level paging enables access to larger address space: up to 128 PiB of virtual address space and 4 PiB of                                                                                                                                      physical address space. My largest system has only 12 GiB of physical address space (RAM), thus this change makes no difference to me, but there are larger systems in the world.

The latest Intel processors implement this paging level - as far as I know, no AMD processors do; however, when this feature is implemented, all code must adhere to this standard.

Unfortunately, updating the VB code for this change is beyond my abilities and will need to be done by Oracle. I created a bug report on their site at https://www.virtualbox.org/ticket/20456, but no response yet. I have asked the openSUSE kernel developers if it would be possible for them to generate an alternative kernel for TW that would have 4-level paging, but I have not heard back about that either. For the moment, keep your 5.12.X kernel safe.
Comment 8 Larry Finger 2021-07-13 15:00:42 UTC
*** Bug 1188189 has been marked as a duplicate of this bug. ***
Comment 9 Stephan Hemeier 2021-07-13 15:33:50 UTC
Adding to grub
randomize_kstack_offset=off
has worked for me:
https://www.virtualbox.org/ticket/20452
https://forums.opensuse.org/showthread.php/556271-VirtualBox-crash-(guru-meditation)-with-new-kernel-5-13-0

Stephan
Comment 10 Andreas Stieger 2021-07-13 15:59:11 UTC
What if you boot with no5lvl?
Comment 11 Larry Finger 2021-07-13 16:28:30 UTC
For me, no5lvl does not work; however, randomize_kstack_offset=off as a grub command-line addition does work.

It seems that I may have come to the wrong conclusion from my testing. The randomized stack offset should be easier to fix.
Comment 12 Bit Juggler 2021-07-13 18:44:18 UTC
(In reply to Larry Finger from comment #11)
> For me, no5lvl does not work; however, randomize_kstack_offset=off as a grub
> command-line addition does work.
> 
> It seems that I may have come to the wrong conclusion from my testing. The
> randomized stack offset should be easier to fix.

Same here.

Kernel-boot-Parameter no5lvl does not work

Kernel-boot-Parameter randomize_kstack_offset=0 works

However

randomize_kstack_offset=
			[KNL] Enable or disable kernel stack offset
			randomization, which provides roughly 5 bits of
			entropy, frustrating memory corruption attacks
			that depend on stack address determinism or
			cross-syscall address exposures. This is only
			available on architectures that have defined
			CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET.
			Format: <bool>  (1/Y/y=enable, 0/N/n=disable)
			Default is CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT.

sounds like it setting it to 0 is disabling a security feature ???
Comment 13 Larry Finger 2021-07-13 19:37:26 UTC
(In reply to Bit Juggler from comment #12)

> Kernel-boot-Parameter randomize_kstack_offset=0 works
> sounds like it setting it to 0 is disabling a security feature ???

Yes, but running kernel 5.12.x does not have that security feature. You are susceptible to the same kind of attack as with turning off the randomize feature in 5.13.x.
Comment 14 Larry Finger 2021-07-15 01:20:14 UTC
Today a fix was provided by Oracle and a new version of VirtualBox that solves the problem without requiring any special kernel command line modifications has been submitted to OBS. When it actually reaches the TW repos, I will close this issue.
Comment 15 Andreas Stieger 2021-07-15 08:16:08 UTC
https://build.opensuse.org/request/show/906361
Comment 16 Larry Finger 2021-07-19 15:11:52 UTC
The Tumbleweed snapshot 20210717 contains the fix for this issue. With it, you can remove all special boot-line parameters.

The issue is now closed.
Comment 17 OBSbugzilla Bot 2021-07-21 19:40:10 UTC
This is an autogenerated message for OBS integration:
This bug (1188105) was mentioned in
https://build.opensuse.org/request/show/907595 15.3 / virtualbox
Comment 18 OBSbugzilla Bot 2021-07-22 03:00:10 UTC
This is an autogenerated message for OBS integration:
This bug (1188105) was mentioned in
https://build.opensuse.org/request/show/907614 15.2 / virtualbox
Comment 19 OBSbugzilla Bot 2021-07-30 06:20:10 UTC
This is an autogenerated message for OBS integration:
This bug (1188105) was mentioned in
https://build.opensuse.org/request/show/909278 15.2 / virtualbox
https://build.opensuse.org/request/show/909279 15.3 / virtualbox
Comment 20 Swamp Workflow Management 2021-08-05 01:58:03 UTC
openSUSE-SU-2021:1092-1: An update that solves four vulnerabilities and has two fixes is now available.

Category: security (important)
Bug References: 1188045,1188105,1188535,1188536,1188537,1188538
CVE References: CVE-2021-2409,CVE-2021-2442,CVE-2021-2443,CVE-2021-2454
JIRA References: 
Sources used:
openSUSE Leap 15.3 (src):    virtualbox-6.1.24-lp153.2.6.1, virtualbox-kmp-6.1.24-lp153.2.6.1
Comment 21 Swamp Workflow Management 2021-08-10 04:17:42 UTC
openSUSE-SU-2021:1114-1: An update that solves four vulnerabilities and has two fixes is now available.

Category: security (important)
Bug References: 1188045,1188105,1188535,1188536,1188537,1188538
CVE References: CVE-2021-2409,CVE-2021-2442,CVE-2021-2443,CVE-2021-2454
JIRA References: 
Sources used:
openSUSE Leap 15.2 (src):    virtualbox-6.1.26-lp152.2.35.1, virtualbox-kmp-6.1.26-lp152.2.35.1