Bugzilla – Bug 1212877
[RPi4] cannot wake up after suspend triggered
Last modified: 2024-01-16 15:13:48 UTC
Created attachment 867900 [details] yast2 logs ssh to my RPi 4 Model B, later I see following -- p200300ea3707b36b055205817bcf9853:~ # Broadcast message from root@localhost (Fri 2023-06-30 06:03:37 UTC): The system will suspend now! -- then I try to wake it up by hitting keyboard and click on mouse, nothing happened on screen. attache yast2logs.
Is it a regression from earlier releases? Also, could you check the behavior with the latest 6.4.x kernel in OBS Kernel:stable repo?
(In reply to Takashi Iwai from comment #1) > Is it a regression from earlier releases? > > Also, could you check the behavior with the latest 6.4.x kernel in OBS > Kernel:stable repo? sorry I cannot tell about this behavior before because I take over RPi testing recently. And I cannot get Tumbleweed with kernel 6.4.x at moment. if you want me to test it, please provide download link for RPi, I can check this of course.
if I trigger Suspend manually I got error: Failed to suspend. I use Tumbleweed 20230707 with kernel 6.3.9
Created attachment 868130 [details] yast logs
I found out there is also an issue with HDMI for display, I switch to another HDMI and triggered suspend again, always same issue, failed to suspend the session.
You can get the kernel from OBS kernel:stable repo, http://download.opensuse.org/repositories/Kernel:/stable/standard/ then just install it.
I updated kernel and some kernel firmware. localhost:~ # uname -a Linux localhost.localdomain 6.4.2-2.gb97b894-default #1 SMP PREEMPT_DYNAMIC Mon Jul 10 10:03:25 UTC 2023 (b97b894) aarch64 aarch64 aarch64 GNU/Linux Triggering suspend failed as before.
Ivan, does the suspend/resume work with your RPi4?
(In reply to Zaoliang Luo from comment #3) > if I trigger Suspend manually I got error: Failed to suspend. > > I use Tumbleweed 20230707 with kernel 6.3.9 How did you 'manually' trigger suspend? I am trying on XFCE log-out -> suspend and I am getting "Failed to lock the screen". Invoking suspend from the serial console is refused because it is login shell.
(In reply to Ivan Ivanov from comment #9) > (In reply to Zaoliang Luo from comment #3) > > if I trigger Suspend manually I got error: Failed to suspend. > > > > I use Tumbleweed 20230707 with kernel 6.3.9 > > How did you 'manually' trigger suspend? I am trying on XFCE > log-out -> suspend and I am getting "Failed to lock the screen". > Invoking suspend from the serial console is refused because it > is login shell. yes, see comment #3 there is menu entry to select Suspend. This is very easy to find ;)
(In reply to Zaoliang Luo from comment #10) > yes, see comment #3 > > there is menu entry to select Suspend. This is very easy to find ;) Well, my question was more like are you using XFCE, Gnome ... to suspend from GUI or something else? Whihc is not clear from comment #3
(In reply to Ivan Ivanov from comment #11) > (In reply to Zaoliang Luo from comment #10) > > > yes, see comment #3 > > > > there is menu entry to select Suspend. This is very easy to find ;) > > Well, my question was more like are you using XFCE, Gnome ... > to suspend from GUI or something else? Whihc is not clear from > comment #3 XFCE
In general, the suspend/resume can be triggered from the command systemctl suspend It can be issued from both user and root. And S4 is "systemctl hibernate". A brute-force suspend(-to-memory) in the kernel level is to run echo mem > /sys/power/state as root. It's sometimes useful to identify whether the issue is related with systemd suspend hooks.
Thanks! I don't have physical access to device right now, but perhaps following back trace is related: [ 151.205740][ T2306] Freezing user space processes [ 151.212063][ T2306] Freezing user space processes completed (elapsed 0.001 seconds) [ 151.219866][ T2306] OOM killer disabled. [ 151.223842][ T2306] Freezing remaining freezable tasks [ 151.230303][ T2306] Freezing remaining freezable tasks completed (elapsed 0.001 seconds) [ 152.316698][ T2340] brcmfmac_wcc: brcmf_wcc_detach: executing [ 152.395768][ T2306] brcm-pcie fd500000.pcie: failed to enter low-power link state [ 152.403408][ T2306] xhci_hcd 0000:01:00.0: Possible wake-up device; regulators will not be disabled [ 155.206392][ T958] ------------[ cut here ]------------ [ 155.211785][ T958] Firmware transaction timeout [ 155.211832][ T958] WARNING: CPU: 1 PID: 958 at drivers/firmware/raspberrypi.c:63 rpi_firmware_property_list+0x24c/0x270 [ 155.227546][ T958] Modules linked in: af_packet qrtr algif_hash aes_neon_bs aes_neon_blk algif_skcipher af_alg bnep bcm2835_v4l2(C) brcmfmac_wcc bcm2835_mmal_vchiq(C) btsdio videobuf2_vmalloc brcmfmac videobuf2_memops vc4 videobuf2_v4l2 hci_uart brcmutil snd_soc_hdmi_codec btqca videodev btrtl btbcm drm_display_helper btintel videobuf2_common cec rc_core bluetooth drm_dma_helper snd_bcm2835(C) mc cfg80211 snd_soc_core cpufreq_dt raspberrypi_cpufreq ecdh_generic snd_compress rfkill bcm2711_thermal ac97_bus nls_iso8859_1 snd_pcm_dmaengine vchiq(C) snd_pcm iproc_rng200 nls_cp437 snd_timer snd leds_gpio soundcore vfat fat uio_pdrv_genirq uio fuse efi_pstore dmi_sysfs ip_tables x_tables ext4 mbcache jbd2 uas usb_storage hid_generic usbhid xhci_pci xhci_pci_renesas xhci_hcd usbcore usb_common mmc_block broadcom bcm_phy_lib gpio_raspberrypi_exp sdhci_iproc clk_raspberrypi raspberrypi_hwmn crct10dif_ce pcie_brcmstb genet sdhci_pltfm mdio_bcm_unimac bcm2835_wdt sdhci mmc_core clk_bcm2711_dvp i2c_brcmstb bcm2835_dma phy_generic [ 155.227753][ T958] virt_dma gpio_regulator nvmemOrmem fixed btrfs blake2b_generic xor xor_neon raid6_pq libcrc32c zlib_deflate sunrpc dmOmirror dm_region_hash dm_log dm_mod sg efivarfs [ 155.336323][ T958] CPU: 1 PID: 958 Comm: kworker/1:3 Tainted: G C 6.3.9-1-default #1 openSUSE Tumbleweed c3ad57c2b5cc05dfd4e9e9e570ead848417362e1 [ 155.351059][ T958] Hardware name: raspberrypi rpi/rpi, BIOS 2023.04 04/01/2023 [ 155.358473][ T958] Workqueue: events get_values_poll [raspberrypi_hwmon] [ 155.365369][ T958] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 155.373137][ T958] pc : rpi_firmware_property_list+0x24c/0x270 [ 155.379141][ T958] lr : rpi_firmware_prperty_list+0x24c/0x270 [ 155.385145][ T958] sp : ffff800008513cc0 [ 155.389207][ T958] x29: ffff800008513cc0 x28: 0000000000000000 x27: ffff58ae0c5196c0 [ 155.397156][ T958] x26: 0000000000000005 x25: ffff8000081de008 x24: 0000000000001000 [ 155.405104][ T958] x23: ffff58ae05200480 x22: ffffbc0b3178d2d0 x21: 0000000000000010 [ 155.413052][ T958] x20: ffff58ae0c519680 x19: ffff8000081de000 x18: ffffffffffffffff [ 155.420999][ T958] x17: 0000000000000000 x16: ffffbc0b2f8a7d30 x15: ffffffffffffffff [ 155.428946][ T958] x14: 0000000000000000 x13: 74756f656d697420 x12: 6e6f69746361736e [ 155.436893][ T958] x11: 00000000ffffdfff x10: ffffbc0b3158a660 x9 : ffffbc0b2ee701a4 [ 155.444840][ T958] x8 : 000000000002ffe8 x7 : c0000000ffffdfff x6 : 00000000000affa8 [ 155.452787][ T958] x5 : ffff58aebb75cd80 x4 : 0000000000000000 x3 : 0000000000000027 [ 155.460733][ T958] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff58ae0c340000 [ 155.468681][ T958] Call trace: [ 155.471862][ T958] rpi_firmware_property_list+0x24c/0x270 [ 155.477514][ T958] rpi_firmware_property+0x8c/0xe8 [ 155.482548][ T958] get_values_poll+0x54/0x120 [raspberrypi_hwmon e5afffe6f5fe47bb345ef677c56ab0c28324fa89] [ 155.492524][ T958] process_one_work+0x204/0x4a8 [ 155.497297][ T958] worker_thread+0x70/0x438 [ 155.501714][ T958] kthread+0x10c/0x118 [ 155.505691][ T958] ret_from_fork+0x10/0x20 [ 155.510021][ T958] ---[ end trace 0000000000000000 ]--- [ 155.515427][ T958] hwmon hwmon0: Failed to get throttled (-110)
Zaoliang Luo, do you have serial cable connected to device by any chance? If yes, please, could you add no_console_suspend to kernel command line and watch/capture serial output during suspend.
(In reply to Ivan Ivanov from comment #15) > Zaoliang Luo, do you have serial cable connected to device by any chance? > If yes, please, could you add no_console_suspend to kernel command line > and watch/capture serial output during suspend. okay, I have no idea about how to do this, but I'll try to figure out and update with information later. serial cable connect to keyboard, right?
(In reply to Zaoliang Luo from comment #16) > (In reply to Ivan Ivanov from comment #15) > > Zaoliang Luo, do you have serial cable connected to device by any chance? > > If yes, please, could you add no_console_suspend to kernel command line > > and watch/capture serial output during suspend. > > serial cable connect to keyboard, right? Well, no, I think. Something like this [1]. [1] https://en.opensuse.org/HCL:Raspberry_Pi3_Serial_Console
(In reply to Ivan Ivanov from comment #17) > (In reply to Zaoliang Luo from comment #16) > > (In reply to Ivan Ivanov from comment #15) > > > Zaoliang Luo, do you have serial cable connected to device by any chance? > > > If yes, please, could you add no_console_suspend to kernel command line > > > and watch/capture serial output during suspend. > > > > > > serial cable connect to keyboard, right? > > Well, no, I think. Something like this [1]. > > [1] https://en.opensuse.org/HCL:Raspberry_Pi3_Serial_Console I got it. Now I could use serial cable connected to my Tumbleweed and use picocom in the way described in your provided information. I added no_console_suspend to linux kernel command line at GRUB stage and then boot. After the system is up and I trigger systemctl suspend. I see in the terminal of picocom there is no output at, just repeat login prompt. Please attached screen capture from booting to suspend.
Created attachment 868377 [details] screen logs
I tried in both terminals for suspend, but same results.
please let me know how to test further, I would like provide any information which I can get.
I think there is similar issue with my Tumbleweed for suspend. I installed the Tumbleweed on Samsung Flash USB 3.1, if I close my laptop and open it later again. The system is gone and the screen shows a lot of BTRFS error. https://live.staticflickr.com/65535/53060875037_24e089c5c0_h.jpg If this is not related to RPi suspend, I will report it seperatly.
Documenting here the new findings. As a test, I managed to wakeup from gpio to prove that WFI implementation on the core is fine and that we can wakeup even from cascaded interrupt. Coming back on track, the USB (a Via VL805 chipset in this case) generates two kinds of interrupts (that should be able to wakeup the board): MSI or Legacy INT (providing pci=nomsi on kernel options). The first one is cascaded and the second one goes directly to GIC and is shared with PCIe PME. The other wakeup source is for the USB controller (that acts as a bridge to PCIe bus) to generate a PME event. Keep in mind that in what follows, the USB registers (included PCI config related ones) can be read/written till xhci_suspend() exits, then the USB controller is stopped and any access on them will freeze the system. HARDWARE SIDE ----------------- PMIC voltages are always turned on. Hooking a scope on USB wires, I can see the suspended USB keyboard generating J-state that should wake the controller. Test point TP34 is providing the VL805 core with VDD (1.05V) voltage at all staged, even during suspend. For any case and since the USB device can only provide D3cold, auxiliary (VSUS33) power should be in place but must be confirmed. SOFTWARE SIDE ----------------- WAKING BY INTERRUPTS The conditions under which MSI/Legacy Int would be triggered are, from XHCI specs (see ref [1], page 322): -HC.Halted=0 -PLC=1 -PSCEG=0 -Event Ring not full That seems to be fully met except for PSCEG=0. In fact, since supending set Run/Stop bit in USBCMD register to 0, the USB controller is disabled and it will eventually set HC.Halted to 1. Since the wakeup signal (internal register PSCEG) is HC.Halted ANDed together with the wakeup signal from USB Device (i.e. PLC or Port Link Changed bit), it won't trigger the wakeup interrupt. This seems to suggest that USB Controller should remain operative for the wakeup to work *or* there should be some other means (i.e. custom hw, ACPI, etc.) responsible for the wakeup. WAKING BY PME The conditions under which PME wakeup message is sent are, from XHCI specs (see ref [1], page 322): -PLS=15 (Resume) -PMCSR.PME_En=1 Forcing the PME_Enable flag in PMCSR, the PME capabilities are acknowledged by the VL805 PCI configuration registers, so it should be feasible to suspend to D3cold (only). On wakeup, a PME inbound message should be generated on PCIe link. During the suspend phase the kernel will poll PME flag (in PCI config register) since some hw is defective or some PCIe hub do not forward them correctly, using a freezable workqueue. This of course is not feasible after the suspend is completed since the USB controller is completely stopped. After that, the USB device should be able to generate PME wakeup. RESULTS In all cases, no wakeup interrupt nor PME message are generated. STILL TOBE CHECKED: - VL805 EEPROM fw version (even though release notes has nothing about suspend related fixes) - VSUS33 voltage, mandatory to wakeup from D3cold (pads are tiny, difficult to solder any wire without some equipment) - double check if GIC interrupts (MSI/PME/Legacy) are indeed enabled - check whether the PCI chain really forward PME wakeup messages REFERENCES: [1] https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf