Bugzilla – Bug 1190469
Boot hangs on Raspberry Pi 3
Last modified: 2023-01-18 17:11:25 UTC
Created attachment 852489 [details] Serial log on RPi3 On openQA, snapshot 20210912 hangs on Raspberry Pi 3, whereas it boots properly on Raspberry Pi 4. Logs in attachment. From the logs, the following lines may be relevant: [ 23.749199] dracut-pre-mount[769]: warning: /dev/mmcblk0: partition 3 is not defined yet [ 24.031592] dracut-pre-mount[769]: Re-reading the partition table failed.: Device or resource busy It also hangs on Raspberry Pi 2 (armv7), but I do not have the logs yet.
Created attachment 852490 [details] Main changes from last green snapshot
Let's assume kernel for now. @Guillaume, is this 100% reproducible also on yours or just once in openQA?
Created attachment 852493 [details] Serial log on RPi2 (armv7)
(In reply to Andreas Färber from comment #2) > Let's assume kernel for now. > > @Guillaume, is this 100% reproducible also on yours or just once in openQA? The RPi{2,3,4} boards in openQA are mine and I do not have spare RPi2 nor RPi3 boards, outside my openQA setup. But this 100% reproducible and older images are still booted properly in openQA, so this is not a hardware failure.
(In reply to Guillaume GARDET from comment #0) > > On openQA, snapshot 20210912 hangs on Raspberry Pi 3, whereas it boots > properly on Raspberry Pi 4. Logs in attachment. Could you please point me to snapshot location. I was unable to find aarch64 port of it.
(In reply to Ivan Ivanov from comment #5) > > Could you please point me to snapshot location. > I was unable to find aarch64 port of it. You can download the image from https://openqa.opensuse.org/tests/1914974#downloads
Do we see anything on the serial console while U-Boot is running? Maybe it's related to: https://github.com/raspberrypi/firmware/issues/1619 With patches that claims to fix the issue: https://patchwork.ozlabs.org/project/uboot/list/?series=262375
(In reply to Matthias Brugger from comment #7) > Do we see anything on the serial console while U-Boot is running? > > Maybe it's related to: > https://github.com/raspberrypi/firmware/issues/1619 > > With patches that claims to fix the issue: > https://patchwork.ozlabs.org/project/uboot/list/?series=262375 This snapshot still uses 2021.03.10-2.2 version of the firmware. I think the issue is around VC4 driver update and DDC usage. Once i saw: [ 33.190642][ T858] i2c-bcm2835 3f805000.i2c: i2c transfer timed out which is this device node, I believe: i2c2: i2c@7e805000 { compatible = "brcm,bcm2835-i2c"; which is DDC channel for this node: hdmi: hdmi@7e902000 { compatible = "brcm,bcm2835-hdmi"; ddc = <&i2c2>; and if I connect RPi3 HDMI to monitor device boots fine ;-). I am looking through VC4 driver update to find which commit caused this regression ...
Most probably the issue is mitigated by this patch [1], see description for details. Patch it already part of TW kernel branch, I think. I am struggling to build kernel package at the moment. Do we have more recent snapshot? [1] https://lore.kernel.org/all/20210525091059.234116-4-maxime@cerno.tech/
(In reply to Ivan Ivanov from comment #9) > Most probably the issue is mitigated by this patch [1], see description for > details. Patch it already part of TW kernel branch, I think. I am struggling > to build kernel package at the moment. Do we have more recent snapshot? > > [1] https://lore.kernel.org/all/20210525091059.234116-4-maxime@cerno.tech/ Latest unreleased snapshot in openQA is 20210914 and it still shows the problem. https://openqa.opensuse.org/tests/overview?distri=microos&distri=opensuse&version=Tumbleweed&build=20210914&groupid=3 This patch seems to be included in upstream 5.14 and was even in upstream 5.13, so I doubt this the solution.
I tried to update the raspberrypi-firmware and raspberrypi-firmware-dt packages, but the boot still hangs. I increased the priority since it blocks Tumbleweed.
Ivan, please don't forget to update SLE15-SP4 branch; it's also 5.14-based tree.
(In reply to Guillaume GARDET from comment #11) > I tried to update the raspberrypi-firmware and raspberrypi-firmware-dt > packages, but the boot still hangs. > > I increased the priority since it blocks Tumbleweed. Fix "drm/vc4: hdmi: Fix HPD GPIO detection" was merged to stable.
I would need an approval of the update request for Factory, from a kernel maintainer, please: https://build.opensuse.org/request/show/920247
(In reply to Guillaume GARDET from comment #14) > I would need an approval of the update request for Factory, from a kernel > maintainer, please: > https://build.opensuse.org/request/show/920247 Sorry, I should have said, it was merged into stable branch from which Tumbleweed is build. (In reply to Takashi Iwai from comment #12) > Ivan, please don't forget to update SLE15-SP4 branch; it's also 5.14-based > tree. Pushed to SLE15-SP4/for-next. Thanks Takashi!
Guillaume, I have spent 1/2 hour searching for newish snapshot and then give up. Could you confirm that issue is fixed for you?
There is no (Arm) Tumbleweed snapshot with the fix yet. The request to monitor is https://build.opensuse.org/request/show/920271 I will try to prepare an image with latest kernel to check if the fix is ok.
(In reply to Guillaume GARDET from comment #17) > There is no (Arm) Tumbleweed snapshot with the fix yet. > The request to monitor is https://build.opensuse.org/request/show/920271 > > I will try to prepare an image with latest kernel to check if the fix is ok. Unfortunately, kernel 5.14.5-1.gfdb6afd-default from Kernel:stable repo also hangs.
(In reply to Guillaume GARDET from comment #18) > > Unfortunately, kernel 5.14.5-1.gfdb6afd-default from Kernel:stable repo also > hangs. Where I could find this? I am finding only x86_64 builds ...
(In reply to Ivan Ivanov from comment #19) > (In reply to Guillaume GARDET from comment #18) > > > > > Unfortunately, kernel 5.14.5-1.gfdb6afd-default from Kernel:stable repo also > > hangs. > > Where I could find this? I am finding only x86_64 builds ... You can find them for download at https://download.opensuse.org/repositories/Kernel:/stable/ARM/ and the OBS project is https://build.opensuse.org/project/show/Kernel:stable With kernel 5.14.6-1.g6131a3c-default it works! So, we just need to wait for https://build.opensuse.org/request/show/920271
(In reply to Guillaume GARDET from comment #20) > (In reply to Ivan Ivanov from comment #19) > > (In reply to Guillaume GARDET from comment #18) > > > > > > > > Unfortunately, kernel 5.14.5-1.gfdb6afd-default from Kernel:stable repo also > > > hangs. > > > > Where I could find this? I am finding only x86_64 builds ... > > You can find them for download at > https://download.opensuse.org/repositories/Kernel:/stable/ARM/ Thanks, will bookmark this now :-) > > and the OBS project is https://build.opensuse.org/project/show/Kernel:stable > > With kernel 5.14.6-1.g6131a3c-default it works! Yey! > > So, we just need to wait for https://build.opensuse.org/request/show/920271 I will close this bug then.
This landed to stable via 5.14.9: commit ef8eac0c61d7a7b80acd5683b9494c2b014ffa8c Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Sun Sep 19 10:06:46 2021 -0700 Revert drm/vc4 hdmi runtime PM changes [ Upstream commit b1044a9b8100a0cc5c9d2e1e2f9ca4bb8e32b23a ] This reverts commits 9984d6664ce9 ("drm/vc4: hdmi: Make sure the controller is powered in detect") 411efa18e4b0 ("drm/vc4: hdmi: Move the HSM clock enable to runtime_pm") as Michael Stapelberg reports that the new runtime PM changes cause his Raspberry Pi 3 to hang on boot, probably due to interactions with other changes in the DRM tree (because a bisect points to the merge in commit e058a84bfddc: "Merge tag 'drm-next-2021-07-01' of git://.../drm"). Revert these two commits until it's been resolved. ================= So is this fix still needed: patches.suse/drm-vc4-hdmi-Fix-HPD-GPIO-detection.patch ? It conflicts with the above patch and is still not upstream. I resolved the conflict for the time being, but should it still be kept?
(In reply to Jiri Slaby from comment #25) > So is this fix still needed: > patches.suse/drm-vc4-hdmi-Fix-HPD-GPIO-detection.patch Just tested on RPi3 without this patch and no HDMI cable connected. It boots fine. So this is no longer required for original bug reported. Should I send pull request with this patch dropped?
(In reply to Ivan Ivanov from comment #26) > (In reply to Jiri Slaby from comment #25) > > So is this fix still needed: > > patches.suse/drm-vc4-hdmi-Fix-HPD-GPIO-detection.patch > > Just tested on RPi3 without this patch and no HDMI cable connected. > It boots fine. So this is no longer required for original bug reported. > > Should I send pull request with this patch dropped? If this fixes the issue without resurrecting bug 1190469 sure :)
(In reply to Matthias Brugger from comment #27) > (In reply to Ivan Ivanov from comment #26) > > (In reply to Jiri Slaby from comment #25) > > > So is this fix still needed: > > > patches.suse/drm-vc4-hdmi-Fix-HPD-GPIO-detection.patch > > > > Just tested on RPi3 without this patch and no HDMI cable connected. > > It boots fine. So this is no longer required for original bug reported. > > > > Should I send pull request with this patch dropped? > > If this fixes the issue without resurrecting bug 1190469 sure :) Hahaha, just realized that the fix was for the very same bug. So yes, please drop it.
(In reply to Matthias Brugger from comment #28) > (In reply to Matthias Brugger from comment #27) > > (In reply to Ivan Ivanov from comment #26) > > > (In reply to Jiri Slaby from comment #25) > > > > So is this fix still needed: > > > > patches.suse/drm-vc4-hdmi-Fix-HPD-GPIO-detection.patch > > > > > > Just tested on RPi3 without this patch and no HDMI cable connected. > > > It boots fine. So this is no longer required for original bug reported. > > > > > > Should I send pull request with this patch dropped? > > > > If this fixes the issue without resurrecting bug 1190469 sure :) > > Hahaha, just realized that the fix was for the very same bug. So yes, please > drop it. Dropped from stable -- no need to do anything there, unless the result is broken ;).
Takashi has merged upstream revert into SP4 already. Closing this for now.