Bug 1198527

Summary: System crashes during boot
Product: [openSUSE] openSUSE Distribution Reporter: Axel Keller <axel.keller>
Component: KernelAssignee: Ivan Ivanov <ivan.ivanov>
Status: RESOLVED DUPLICATE QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P5 - None CC: axel.keller, ivan.ivanov, mbrugger, patrik.jakobsson, stefan.wahren, tiwai
Version: Leap 15.3   
Target Milestone: ---   
Hardware: aarch64   
OS: openSUSE Leap 15.3   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: Update list
Log with monitor
Log without monitor

Description Axel Keller 2022-04-15 19:39:00 UTC
Created attachment 858183 [details]
Update list

After updating Leap 15.3 on a Raspberry PI4 the system cannot be booted again (black screen)

Also after a fresh installation of
     openSUSE-Leap-15.3-ARM-KDE-raspberrypi.aarch64-2022.03.04-Build9.449
and updating 18 packages directly (second update), the following boot will crash.

The problem is the 'Security update for the Linux kernel' !!! (see attachment)


Updating the kernel of Leap 15.3 on a raspberry PI4 will destroy any system!!!
Comment 1 Ivan Ivanov 2022-04-19 14:03:12 UTC
Maybe same as bsc#1198484 "Regression in kernel update 5.3.18-150300.59.60.4"
Comment 2 Ivan Ivanov 2022-04-21 13:15:53 UTC
Hi, please, could you boot without monitor attached to device.
Wait, let say 30 seconds, ssh to it and run dmesg -w and then 
plug the monitor. Then attach the whole log here. Thanks!
Comment 3 Patrik Jakobsson 2022-04-27 09:30:30 UTC
Possibly a duplicate of bsc#1198534
Comment 4 Axel Keller 2022-04-27 18:30:36 UTC
I don't think that it is a duplicate of bsc#1198534, because the boot doesn't reach the login page. Therefore a ssh command cannot work.
Comment 5 Patrik Jakobsson 2022-04-27 20:36:47 UTC
(In reply to Axel Keller from comment #4)
> I don't think that it is a duplicate of bsc#1198534, because the boot
> doesn't reach the login page. Therefore a ssh command cannot work.

Yes, perhaps you're right.

You can try to add the following to the kernel command line: modprobe.blacklist=vc4

If that works you might be hitting the bug in bsc#1198942
Comment 6 Ivan Ivanov 2022-04-28 14:07:59 UTC
There are few VC4 related issues. I believe better match is bsc#(In reply to Axel Keller from comment #4)
> I don't think that it is a duplicate of bsc#1198534, because the boot
> doesn't reach the login page. Therefore a ssh command cannot work.

Hm, have you checked? Login page and SSHd are different processes.

Please, try what was described in comment#2. If this doesn't work 
try disabling vc4 driver as described by Patrik.
Comment 7 Axel Keller 2022-04-29 12:05:56 UTC
(In reply to Patrik Jakobsson from comment #5)
> (In reply to Axel Keller from comment #4)
> > I don't think that it is a duplicate of bsc#1198534, because the boot
> > doesn't reach the login page. Therefore a ssh command cannot work.
> 
> Yes, perhaps you're right.
> 
> You can try to add the following to the kernel command line:
> modprobe.blacklist=vc4
> 
> If that works you might be hitting the bug in bsc#1198942

I tried this with the last published version 
openSUSE-Leap-15.3-ARM-KDE-raspberrypi.aarch64-2022.03.04-Build9.449.raw

(It was not easy for me to figure out where and how the command line has to be extended during boot!)

With this extension the system comes up, the login page works ...

I'm wondering that a system is published which wasn't tested obviously.
Comment 8 Axel Keller 2022-04-29 12:15:50 UTC
Created attachment 858534 [details]
Log with monitor
Comment 9 Axel Keller 2022-04-29 12:16:33 UTC
Created attachment 858535 [details]
Log without monitor
Comment 10 Axel Keller 2022-04-29 12:20:37 UTC
(In reply to Ivan Ivanov from comment #6)
> There are few VC4 related issues. I believe better match is bsc#(In reply to
> Axel Keller from comment #4)
> > I don't think that it is a duplicate of bsc#1198534, because the boot
> > doesn't reach the login page. Therefore a ssh command cannot work.
> 
> Hm, have you checked? Login page and SSHd are different processes.
> 
> Please, try what was described in comment#2. If this doesn't work 
> try disabling vc4 driver as described by Patrik.

This I can only test with the previous version
openSUSE-Leap-15.3-ARM-KDE-raspberrypi.aarch64-2021.05.31-Build9.201.raw
because after an update the boot hangs totally.

I'm not quite sure how to boot and to login without a monitor!? A ssh command doesn't work without login. So I managed it blind. See attachments.
Comment 11 Ivan Ivanov 2022-04-29 12:29:00 UTC
(In reply to Axel Keller from comment #10)

> 
> I'm not quite sure how to boot and to login without a monitor!? A ssh
> command doesn't work without login. So I managed it blind. See attachments.

The idea is to use another computer from which you could ssh to RPi.
So even without monitor you can login to device and watch dmesg output.
Comment 12 Stefan Wahren 2022-04-29 16:06:50 UTC
Hi,
yesterday i noticed a regression in current kernel mainline tree and it seems that the offending patch was already backported to openSuSE:

https://lore.kernel.org/all/20220222140732.253819-1-maxime@cerno.tech/

This patch is broken and should be dropped.

I already send a bugfix:

https://marc.info/?l=linux-clk&m=165117066630375&w=2
Comment 13 Ivan Ivanov 2022-04-29 19:05:36 UTC
Thanks, I noticed your fix, but it didn't help.
Comment 14 Axel Keller 2022-04-29 19:43:16 UTC
(In reply to Ivan Ivanov from comment #13)
> Thanks, I noticed your fix, but it didn't help.

It comes as little surprise...
Look at the vc4 problem
Comment 15 Axel Keller 2022-04-30 09:41:08 UTC
WORKAROUND

On the ssd-card (ROOT partition) add to /etc/modprobe.d/50-blacklist.conf:
  
  blacklist vc4

This enables correct booting.
Comment 16 Ivan Ivanov 2022-05-03 07:07:49 UTC
(In reply to Axel Keller from comment #14)
> (In reply to Ivan Ivanov from comment #13)
> > Thanks, I noticed your fix, but it didn't help.
> 
> It comes as little surprise...
> Look at the vc4 problem

My reply was about Stefan Wahren comment#12.

Anyway, it looks like you have hit bsc#1198534.

Please, see my comment#10 in bsc#1198534. I am closing
this as duplicate. Thanks!

*** This bug has been marked as a duplicate of bug 1198534 ***