Bug 1213779

Summary: Boot hang on Lenovo laptops due to tpm_tis regression on 6.4.x.
Product: [openSUSE] openSUSE Tumbleweed Reporter: Cosmin Tanczel <cosmin.tanczel>
Component: KernelAssignee: openSUSE Kernel Bugs <kernel-bugs>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P2 - High CC: bootloader-maintainers, glin, jlee, mchang, rickscafe.casablanca, tiwai
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE Tumbleweed   
Whiteboard:
Found By: Customer Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: dmidecode
dmidecode - Lenovo ThinkBook 13s-IML
Fix patch

Description Cosmin Tanczel 2023-07-29 18:55:41 UTC
I did the daily TW update and after reboot I got a big delay after "Loading initial ramdisk"
Tried acpi=off, got the same result, so I decided to do a clean install since there is no important data on the disk. 

Strange was that even if I was booting from the USB (for the reinstall) I got the same behavior. The installation finished fine, everything works, except the big delay.

The machine it's a Lenovo ThinkPad E14 Gen2 and I think this is related to some BIOS/drivers issues. 

I have another laptop (HP ProBook 430 G8 Notebook PC) also updated today => no issues.
Comment 1 Cosmin Tanczel 2023-07-29 19:13:22 UTC
I also have a ThinkPad T490s on which I just performed the update. On this machine everything looks good.

To exclude HW issues I tried installing openSUSE Leap 15.5 and booting from a openSUSE Leap 15.5 USB stick works fine. There is no delay and it's working quite fast.

Bottom line, I've tried on 3 different machines:

openSUSE TW:
1. ThinkPad E14 Gen2 - boot delay after Loading initial ramdisk
2. ThinkPad T490s - no boot delay
3. HP ProBook 430 - no boot delay

openSUSE LEAP 15.5:
1. ThinkPad E14 Gen2 - no boot delay
2. ThinkPad T490s - no boot delay
3. HP ProBook 430 - no boot delay

Let me know if I need to share any logs.
Comment 2 Cosmin Tanczel 2023-07-29 19:24:00 UTC
Startup finished in 8.023s (firmware) + 5.172s (loader) + 2min 28.393s (kernel) + 1.690s (initrd) + 3.158s (userspace) = 2min 46.438s 
multi-user.target reached after 3.158s in userspace.
Comment 3 Cosmin Tanczel 2023-07-31 18:54:19 UTC
Adding noapic to bootloader solves the issue.
Any updates?
Comment 4 Cosmin Tanczel 2023-08-02 13:09:29 UTC
Without ACPI enabled I loose features like wake from sleep when pressing any key, so I am forced to hard reset the laptop.
Comment 5 Gary Ching-Pang Lin 2023-08-03 06:17:17 UTC
Sounds like a kernel issue.

Joey, any idea?
Comment 6 Cosmin Tanczel 2023-08-04 10:46:03 UTC
It's almost one week since the report and it took 5 days only for the triage. Can we speed this up please? Do you need more info?

Thanks,
Comment 7 Mark Draheim 2023-08-04 16:14:25 UTC
Have a Lenovo Thinkbook 13s-IML showing the same behaviour. It's more than two minutes before the disk decrypt pw prompt shows. Just wanted to mention that on Tumbleweed the previous kernel 6.4.4 does not have this problem.
Comment 8 Cosmin Tanczel 2023-08-05 10:45:21 UTC
@Mark: I'm not using an encrypted drive at all. The behavior started one week ago after an update, but again, no encryption used.

I even reinstalled tumbleweed from scratch without keeping any data (formatted all positions) and I get the same behavior.
Basically even booting from a tumbleweed usb stick takes a lot of time initializing ram disk.

Using Leap works, but for sure I cannot switch to Leap.
Comment 9 Cosmin Tanczel 2023-08-05 10:47:47 UTC
Ah.. Just re-read your comment. So you have the delay BEFORE the decrypt password promptrompty appears.

Then it's related to this bug.

Thanks
Comment 10 Michael Chang 2023-08-07 03:13:09 UTC
Reassign to kernel component given the 'noapic' option has to be used to circumvent the delay problem.
Comment 11 Takashi Iwai 2023-08-07 07:03:38 UTC
It might be tpm_tis issue a few other machines have hit.
Try to pass tpm_tis.interrupts=0 boot option.
Comment 12 Mark Draheim 2023-08-07 08:08:22 UTC
(In reply to Takashi Iwai from comment #11)
> It might be tpm_tis issue a few other machines have hit.
> Try to pass tpm_tis.interrupts=0 boot option.

adding that works on my ThinkBook 13s-IML. Though Tumbleweed kernel 6.4.8 took "only" 30 secs or so without it. But with that opt the crypt pw prompt shows immediately.
Comment 13 Takashi Iwai 2023-08-07 08:21:05 UTC
There was a bit of improvement in 6.4.8 to cut down the stall time and proceed somehow after that.  You must see some relevant error message from kernel.

If tpm_tis boot option works, it's basically a BIOS firmware bug, and we can add your device to the quirk list.  Please give the output of dmidecode.
Comment 14 Cosmin Tanczel 2023-08-07 09:34:39 UTC
Created attachment 868648 [details]
dmidecode
Comment 15 Cosmin Tanczel 2023-08-07 09:38:54 UTC
Comment on attachment 868648 [details]
dmidecode

Adding tpm_tis.interrupts=0 boot option works for me too.

Without tpm_tis.interrupts=0 boot option:

Startup finished in 8.023s (firmware) + 5.172s (loader) + 2min 28.393s (kernel) + 1.690s (initrd) + 3.158s (userspace) = 2min 46.438s 
multi-user.target reached after 3.158s in userspace.


With tpm_tis.interrupts=0 boot option:

Startup finished in 6.154s (firmware) + 1min 24.869s (loader) + 1.186s (kernel) + 1.735s (initrd) + 3.843s (userspace) = 1min 37.790s 
graphical.target reached after 3.843s in userspace.



Short version: 2min 28.393s vs 1.186s

Maybe we should do what Linus suggested these days (disable TPM)
Comment 16 Mark Draheim 2023-08-07 09:55:15 UTC
Created attachment 868649 [details]
dmidecode - Lenovo ThinkBook 13s-IML
Comment 17 Takashi Iwai 2023-08-07 10:04:22 UTC
Thanks.  I'm building now the test kernel package with a fix patch (for both TP E14 and 13s-MIL).  It's being built in OBS home:tiwai:bsc1213779 repo.  Once after the build finishes (takes an hour or so), the package will be available at
  http://download.opensuse.org/repositories/home:/tiwai:/bsc1213779/standard/
Please give it a try later without the boot option workaround.

Note that it's an unofficial build, hence you'd need to turn off Secure Boot.

Once after confirming the patch working, I'll submit it to the upstream and merge to TW kernel.
Comment 18 Takashi Iwai 2023-08-07 10:19:41 UTC
Created attachment 868652 [details]
Fix patch
Comment 19 Mark Draheim 2023-08-07 13:44:33 UTC
(In reply to Takashi Iwai from comment #17)

>   http://download.opensuse.org/repositories/home:/tiwai:/bsc1213779/standard/
> Please give it a try later without the boot option workaround.

works for the ThinkBook IML. 

Installed this one http://download.opensuse.org/repositories/home:/tiwai:/bsc1213779/standard/x86_64/kernel-default-6.4.8-1.1.ga777411.x86_64.rpm , removed the tpm_tis boot opt, checked that 6.4.4 was taking forever, and then again that your 6.4.8 works.

Disabling TPM in Bios is also possible on this laptop but I keep that enabled for now.

Thanks
Comment 20 Takashi Iwai 2023-08-11 07:25:15 UTC
The upstream dev took a larger hammer to apply the workaround to all Lenovo machines, as there are many other Lenovo machines hitting the very same problem.
  https://lore.kernel.org/r/20230810182433.518523-1-jarkko@kernel.org

I'm building a TW kernel with the backport in OBS home:tiwai:bsc1213779-2 repo.
It'd be appreciated if anyone can test it quickly once after the build finishes.
Comment 21 Mark Draheim 2023-08-11 15:17:21 UTC
(In reply to Takashi Iwai from comment #20)
> I'm building a TW kernel with the backport in OBS home:tiwai:bsc1213779-2
> repo.

your 6.4.9-1.gefae9ac-default works for my Thinkbook
Comment 23 Takashi Iwai 2023-08-14 13:03:20 UTC
The upstream decided to disable tpm_tis interrupts as default, after all: the fix commit 6aaf663ee04a80b445f8f5abff53cb92cb583c88 landed in 6.5-rc6.

I backported it to TW kernel now.  It'll be included after 6.4.10.