Bug 1180336

Summary: [RPi4] No USB in rpi4b and rpi400 since v5.10
Product: [openSUSE] openSUSE Tumbleweed Reporter: Nicolas Patricio Saenz Julienne <nsaenzjulienne>
Component: KernelAssignee: Ivan Ivanov <ivan.ivanov>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P2 - High CC: afaerber, amajer, cristian_anita, David, dillon.ang.h, fvogt, guillaume.gardet, hello, jslaby, mbrugger, nsaenzjulienne, ptesarik, rexbinary
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Nicolas Patricio Saenz Julienne 2020-12-24 13:14:18 UTC
RPi4's firmware doesn't support receiving two USB reset signals in a row, even though it's supposed to do so (funnily enough 3 resets in a row work fine). This leaves the xHC in a broken state for TW as both u-boot and Linux perform this operation.

I'm in contact with the RPi engineers and a firmware fix should show up eventually. That said we're in the middle of the holiday season, so we'll have to be patient. We could advise TW users to kexec into the same Linux image to trigger a third USB reset, which will fix the issue, but that seems overkill. Otherwise, I prepared a small kernel fix to bypass the reset signal if USB is already up. This fix will never make it upstream, as it's somewhat hacky. But I figure it'll be nice to port it to TW while we wait for the RPi foundation to publish a new firmware.
Comment 1 Guillaume GARDET 2021-01-04 07:59:12 UTC
Or we could revert the firmware if we know which commit broke it.
Comment 2 Fabian Vogt 2021-01-04 15:28:51 UTC
What's the current state of this? Did you submit the kernel workaround? Did anything happen on the FW side?
Comment 3 Nicolas Patricio Saenz Julienne 2021-01-04 15:44:39 UTC
A kernel fix was queued today in kernel:stable.
Comment 4 Nicolas Patricio Saenz Julienne 2021-01-14 09:40:18 UTC
*** Bug 1180799 has been marked as a duplicate of this bug. ***
Comment 5 Nicolas Patricio Saenz Julienne 2021-01-14 09:41:14 UTC
fix merged.
Comment 6 Nicolas Patricio Saenz Julienne 2021-01-14 16:48:28 UTC
Closed the issue too soon. A temporary fix was merged in the kernel, but the proper fix has to be introduced in u-boot.
Comment 7 Nicolas Patricio Saenz Julienne 2021-01-14 16:57:27 UTC
It turns out it's not something firmware can fix, as per RPi's engineer's comment: the USB firmware load sequence can only be performed *once* per controller reset. Subsequent firmware load operations will leave the controller in a bad state. This means in practice that every time a handover happens during the boot process the PCIe bus has to be properly reset, in particular the #PERST line.

I'll soon post here a u-boot series fixing this issue.
Comment 8 Ang 2021-01-25 19:09:20 UTC
Just an update that the issue with USB (in both the installer and the installed system in general) persists with u-boot-rpiarm64-2020.10-10.1 and kernel-default 5.10.9-1.1 which were both released with snapshot 20210122.
Currently I'm running on 20210122 with all the updates except for the kernel. I'm using kernel-default 5.9.14-1.1 instead of 5.10.x and everything works as expected.
Comment 9 Anthony Rabbito 2021-01-25 22:01:22 UTC
Thanks for your update, and confirmation Ang.

I was waiting out for snapshot 20210122 and glad I didn't get around to installing it yet. Is it possible to install 5.10.9-1.1 on a fresh image flash? Are there any documentation around to do that?
Comment 10 Ang 2021-01-25 22:50:33 UTC
I tested this by flashing one of the prebuilt Tumbleweed 20210122 images for the Raspberry Pi 4 to a USB and tried booting it. I ended up with the same result where there was no USB after it got past the Grub 2 screen. I don't have a spare SD card at the moment since all the ones I have are being used. I might have to get more the next time they go on sale.
Comment 11 Nicolas Patricio Saenz Julienne 2021-01-26 12:56:33 UTC
Just tested yesterday's Factory build with:
 - kernel-default: 5.10.9-1.1
 - u-boot-rpi4: 2020.10-10.1

Everything is back to normal, hopefully, the next TW snapshot will contain the fixed packages.

If you want to install the latest kernel you can add the kernel:stable repository:
https://download.opensuse.org/repositories/Kernel:/stable/ARM/
Comment 12 Ang 2021-01-26 13:53:27 UTC
Out of curiosity, has the package u-boot-rpiarm64 been patched? I was under the impression that the rpi3 and rpi4 u-boot packages were replaced by the single u-boot-rpiarm64 package. The rpiarm64 package is the one the installer and the prebuilt images are using when they boot. When you use the installer, by default it installs u-boot-rpiarm64 instead of either the rpi3 or rpi4 versions.
Comment 17 Ang 2021-02-03 04:14:58 UTC
I tested Tumbleweed aarch64 snapshot 20210128 on my Raspberry Pi 4 1.1 4 GB running the latest stable eeprom firmware. There were no new packages in this snapshot for both kernel-default or u-boot-rpiarm64 (not u-boot-rpi4) in this snapshot and the problem still persists in this snapshot. To get around this issue, I am using kernel-default 5.9.14-1.1 instead of kernel-default 5.10.9-1.1. I am running u-boot-rpiarm64-2020.10-10.1. I chose to run u-boot-rpiarm64 instead of u-boot-rpi4 since this is the default u-boot used in the installation DVD, the prebuilt Raspberry Pi 4 images, and is installed by default when installing on a Raspberry Pi 4 using the installation DVD.

All of the packages (u-boot etc.) installed on my Raspberry Pi 4 systems are identical to what is in snapshot 20210128 with the exception of the kernel since I'm running a 5.9 kernel.
Comment 18 Adam Majer 2021-02-04 20:59:09 UTC
Current in TW, the kernel seems patched, 

* Thu Dec 24 2020 nsaenzjulienne@suse.de
- reset: raspberrypi: Don't reset USB if already up (bsc#1180336).
- commit cbfc03c

unfortunately this seems to remain? The issue is that in initrd, there is no USB while it works just fine after normal boot proceeds. This breaks MicroOS configuration stage.
Comment 19 Ang 2021-02-05 15:34:47 UTC
Just tested in snapshot 20210203 with the kernel-default 5.10.12 on my Raspberry Pi 4 4GB HW Revision 1.1. I am using the latest eeprom in the Raspberry Pi 4 stable branch. When booting the installation DVD for Tumbleweed Aarch64 20210203, one loses all USB devices after the Grub2 menu where you can select Installation..., Boot from hard disk, etc. It will progress to the graphical license screen, but one cannot select anything or input anything using USB input devices in order to proceed with installation.

I also tested the prebuilt snapshot 20210203 images for Raspberry Pi 4, and it will not be able to proceed to mounting and setting up filesystems (will time out) if installed on a USB key. If booted from an SD card, it will boot, but one has no USB after the Grub2 menu.
Comment 20 Nicolas Patricio Saenz Julienne 2021-02-09 16:18:19 UTC
(In reply to Adam Majer from comment #18)
> Current in TW, the kernel seems patched, 
> 
> * Thu Dec 24 2020 nsaenzjulienne@suse.de
> - reset: raspberrypi: Don't reset USB if already up (bsc#1180336).
> - commit cbfc03c
> 
> unfortunately this seems to remain? The issue is that in initrd, there is no
> USB while it works just fine after normal boot proceeds. This breaks MicroOS
> configuration stage.

I sent fix for this: https://github.com/dracutdevs/dracut/pull/1079

USB3 in rpi4 depends on kernel module 'reset-raspberrypi.ko' which isn't being picked up by dracut ATM.
Comment 21 Anthony Rabbito 2021-02-09 16:44:19 UTC
Nicholas thanks for your work.
Comment 22 Guillaume GARDET 2021-02-15 09:17:27 UTC
(In reply to Nicolas Patricio Saenz Julienne from comment #20)
> (In reply to Adam Majer from comment #18)
> > Current in TW, the kernel seems patched, 
> > 
> > * Thu Dec 24 2020 nsaenzjulienne@suse.de
> > - reset: raspberrypi: Don't reset USB if already up (bsc#1180336).
> > - commit cbfc03c
> > 
> > unfortunately this seems to remain? The issue is that in initrd, there is no
> > USB while it works just fine after normal boot proceeds. This breaks MicroOS
> > configuration stage.
> 
> I sent fix for this: https://github.com/dracutdevs/dracut/pull/1079
> 
> USB3 in rpi4 depends on kernel module 'reset-raspberrypi.ko' which isn't
> being picked up by dracut ATM.

Any backports planned/required for Tumbleweed (or Leap)?
Comment 23 Nicolas Patricio Saenz Julienne 2021-02-15 09:39:10 UTC
(In reply to Guillaume GARDET from comment #22)
> Any backports planned/required for Tumbleweed (or Leap)?

Yes, I sent some pull requests into openSUSE/dracut.

On top of all that I have to finish (hopefully today) a series doing pretty much the same in the TW installation-image package.
Comment 34 Swamp Workflow Management 2021-02-24 14:19:18 UTC
SUSE-RU-2021:0573-1: An update that has two recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1176171,1180336
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Module for Basesystem 15-SP2 (src):    dracut-049.1+suse.185.g9324648a-3.21.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 36 Swamp Workflow Management 2021-02-27 17:17:06 UTC
openSUSE-RU-2021:0351-1: An update that has two recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1176171,1180336
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.2 (src):    dracut-049.1+suse.185.g9324648a-lp152.2.18.1
Comment 40 Nicolas Patricio Saenz Julienne 2021-03-04 09:50:16 UTC
All fixes channeled into their projects.
Comment 44 Jiri Slaby 2022-02-08 07:34:06 UTC
(In reply to Adam Majer from comment #18)
> Current in TW, the kernel seems patched, 
> 
> * Thu Dec 24 2020 nsaenzjulienne@suse.de
> - reset: raspberrypi: Don't reset USB if already up (bsc#1180336).
> - commit cbfc03c

There is a note in the patch:
NOTE: This is a temporary fix while we wait for a firmware update. It's pretty
hacky due to the call to pci_get_device(). It modifies an RPi specific driver,
so it's unlikely to affect other platforms, while making life easier to RPi TW
users.

So can we drop the patch from master+stable now?
Comment 45 Guillaume GARDET 2022-02-08 08:01:13 UTC
(In reply to Jiri Slaby from comment #44)
> So can we drop the patch from master+stable now?

No idea.

Could you provide a kernel to test, please?
Comment 46 Ivan Ivanov 2022-02-21 09:12:48 UTC
Package with reverted commit cbfc03cfbd8e6be5f6809aeabf8267ef26e09a7e 
is now building here[1].

[1] https://build.opensuse.org/project/show/home:iivanov:bsc1180336
Comment 47 Ivan Ivanov 2022-02-21 10:21:42 UTC
Briefly tested kernel with above kernel workaround
reverted, using USB keyboard. It was detected during boot
and plugging it in and out seems to properly detected and 
handled. 

raspberrypi-firmware          - 2022.01.24-1.1
raspberrypi-eeprom[-firmware] - 2021.04.29-2.1
u-boot-rpiarm64               - 2022.01-2.1
Comment 48 Ivan Ivanov 2022-03-08 18:49:03 UTC
Workaround patch removed and change is merged in Kernel:stable.
Closing this bug.