Bug 1221603

Summary: rk3588: install from SD-card fails to detect own medium
Product: [openSUSE] openSUSE Distribution Reporter: Torsten Duwe <duwe>
Component: InstallationAssignee: Steffen Winterfeldt <snwint>
Status: RESOLVED FIXED QA Contact: Jiri Srain <jsrain>
Severity: Major    
Priority: P5 - None CC: duwe, felix.niederwanger
Version: Leap 15.6   
Target Milestone: ---   
Hardware: aarch64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: lsmod from TW kernel 6.7.7-1-default (1ff84c5)

Description Torsten Duwe 2024-03-18 12:09:00 UTC
Created attachment 873595 [details]
lsmod from TW kernel 6.7.7-1-default (1ff84c5)

On a Radxa Rock 5B with a recent U-Boot in the SPI flash, the installation ISO starts from SD-card (the usual way to install SBCs).

The installation initrd however then fails to locate the medium -- no partitions present at all.

The TumbleWeed live image for e.g. the rockpro64 runs nicely, so I can collect some data:

udevadm info -a /dev/mmcblk1 | grep DRIVERS yields:
    DRIVERS=="mmcblk"
    DRIVERS=="dwmmc_rockchip"

so presumably adding dw_mmc_rockchip (plus dependencies) to the initrd  should fix the issue.
Comment 1 Steffen Winterfeldt 2024-03-18 17:19:32 UTC
dw_mmc-rockchip is already part of the initrd.

When it fails: there's a shell on console 9 running all the time.
Or, if you don't have that - go to 'Expert -> Start shell' in the menu.

Sorry, I have no way to find out what's missing if it's a kernel module.

If you have an idea what it could be, you can build a new install iso:

mksusecd --create new.iso --nano --modules foo,bar \
  --kernel kernel-default.rpm -- tumbleweed.iso

This would add modules foo and bar. kernel-default.rpm should be taken
from the iso. Note the isolated '--' in the argument list.

Module dependencies are automatically resolved (IIRC).
Comment 2 Steffen Winterfeldt 2024-03-18 17:23:44 UTC
Uhm, scrap the '--nano' (this would only produce a network iso):

mksusecd --create new.iso --modules foo,bar \
  --kernel kernel-default.rpm -- tumbleweed.iso
Comment 3 Torsten Duwe 2024-03-19 09:02:53 UTC
(In reply to Steffen Winterfeldt from comment #1)
> dw_mmc-rockchip is already part of the initrd.

Hm, okay. something else then.

> Sorry, I have no way to find out what's missing if it's a kernel module.

Sure, understood.

> mksusecd --create new.iso --nano --modules foo,bar \
>   --kernel kernel-default.rpm -- tumbleweed.iso

Will try. Network ISO fails as well, because I assume the LAN drivers
are not in the initrd either ;-)

> Module dependencies are automatically resolved (IIRC).

Will check. There's cases when there is no explicit code dependency, but e.g. some subsystem "register_foo()" and later someone checks all the "foo"s
Comment 4 Steffen Winterfeldt 2024-03-19 09:13:04 UTC
> Network ISO fails as well, because I assume the LAN drivers
> are not in the initrd either ;-)

It's not strictly a net iso but an iso without all the fluff. :-)

It's enough for checking (with manual=1 or waiting for the error message)
whether the modules have been loaded.
Comment 5 Torsten Duwe 2024-03-19 15:06:21 UTC
FTR:
mksusecd --create /tmp/new.iso --micro --modules dw_mmc-rockchip --kernel kernel-default.rpm -- openSUSE-Leap-15.6-NET-aarch64-Media.iso

does not work.

| assuming repo-md sources
| transient signing key created, keyid = C3D864298DAE1C1B
| kernel-default.rpm: don't know how to unpack this
| cannot make directory read+writeable for /var/tmp/mksusecd.gSerC_u2/mnt_0001: | Read-only file system at /usr/lib/perl5/5.26.1/File/Temp.pm line 784.

and many more RO errors. anyway...
The dw_mmc_rockchip seems to be there indeed, but

<6>[   16.079922][   T73] mmc0: SDHCI controller on fe2e0000.mmc [fe2e0000.mmc] using ADMA
<6>[   16.103696][   T54] mmc0: Failed to initialize a non-removable card
<6>[   30.447880][  T164] platform fe2c0000.mmc: deferred probe pending
<6>[   30.447884][  T164] platform fe2d0000.mmc: deferred probe pending

investigating further...
Comment 6 Torsten Duwe 2024-03-19 15:15:09 UTC
When I boot the functioning TW and insert the installation SD later, dmesg says:

[   51.505760][  T506] mmc_host mmc1: Bus speed (slot 0) = 198000000Hz (slot req 200000000Hz, actual 198000000HZ div = 0)
[   51.928763][  T506] dwmmc_rockchip fe2c0000.mmc: Successfully tuned phase to 106
[   51.929468][  T506] mmc1: new ultra high speed SDR104 SDHC card at address 1234
[   51.965420][ T1321] mmcblk1: mmc1:1234 SA16G 14.4 GiB
[   51.968178][ T1321]  mmcblk1: p1 p2
Comment 7 Steffen Winterfeldt 2024-03-19 15:54:28 UTC
Does it help to load the module manually?

> kernel-default.rpm: don't know how to unpack this

Which mksusecd version?

> cannot make directory read+writeable for /var/tmp/mksusecd.gSerC_u2/mnt_0001:

Are you running a microos variant?
Comment 8 Torsten Duwe 2024-03-19 17:17:15 UTC
(In reply to Steffen Winterfeldt from comment #7)
> Does it help to load the module manually?

The module gets loaded, by an OF match. It's what prints the "platform fe2c0000.mmc: deferred probe pending" in c#5.

> > kernel-default.rpm: don't know how to unpack this
> 
> Which mksusecd version?

I was so optimistic to use mksusecd-2.10 on Leap 15.5/amd64 to work on 15.6/aarch64. Would it support cross-platform?

> > cannot make directory read+writeable for /var/tmp/mksusecd.gSerC_u2/mnt_0001:
> 
> Are you running a microos variant?

Nope. Leap 15.5/amd64 physical. But the installation medium is an ISO 9660 FS, no? It's mounted loopback and the error path points there.

I'm now comparing TW install (same problem) with TW from disk, working.
Comment 9 Steffen Winterfeldt 2024-03-19 17:26:59 UTC
> I was so optimistic to use mksusecd-2.10 on Leap 15.5/amd64 to work on
> 15.6/aarch64. Would it support cross-platform?

Cross-platform is ok but it's better to use the latest 2.14

https://software.opensuse.org/download/package?project=home:snwint:ports&package=mksusecd

as TW or newer SLE versions have changes the old mksusecd does not know about.

> It's mounted loopback 

You don't need to, you can pass the ISO image file.
Comment 10 Torsten Duwe 2024-03-19 18:04:46 UTC
spi_rockchip makes a difference!

When it is loaded, mmc partitions show up (with mmc-block loaded ;-)

Without it, behaviour on TW is as buggy as described here.

I'll check with the schematics whether this is only for the rock pi5 or rk3588 in general, and will test mksusecd-2.14

Then we can see whether support is feasible for Leap 15.6
Comment 11 Torsten Duwe 2024-03-20 10:58:16 UTC
Sorry, mksusecd-2.14 does not work for me either.

mksusecd --create /tmp/new.iso --micro --modules spi_rockchip,rk8xx_spi,rk808_regulator --kernel kernel-default.rpm -- openSUSE-Leap-15.6-NET-aarch64-Media.iso

(lots of RO errors, as with 2.10)

# mount
[...] openSUSE-Leap-15.6-NET-aarch64-Media.iso on /var/tmp/mksusecd.HQHl_Hq2/mnt_0001 type iso9660 (ro,relatime, ...

This is what mksusecd obviously did. How is this supposed to work?

Besides that, DTSes suggest it is very common that rk3588 boards talk to their rk806 PMIC over SPI2, so this problem has a broader impact than just the rock5.

Please try to include above modules for aarch64
Comment 12 Torsten Duwe 2024-03-20 11:20:42 UTC
(In reply to Torsten Duwe from comment #11)
> Sorry, mksusecd-2.14 does not work for me either.
> 
> mksusecd --create /tmp/new.iso --micro --modules
> spi_rockchip,rk8xx_spi,rk808_regulator --kernel kernel-default.rpm --
> openSUSE-Leap-15.6-NET-aarch64-Media.iso

FTR, I'm trying to work on images from here:
https://download.opensuse.org/ports/aarch64/tumbleweed/iso/openSUSE-Tumbleweed-NET-aarch64-Current.iso

> (lots of RO errors, as with 2.10)
Comment 13 Steffen Winterfeldt 2024-03-20 11:25:59 UTC
Interesting. I'll check what's going on.
Comment 14 Steffen Winterfeldt 2024-03-20 11:45:52 UTC
Based on openSUSE-Tumbleweed-DVD-aarch64-Snapshot20240318-Media.iso, there are

~snwint/Export/n-net.iso
~snwint/Export/n-full.iso

Note that the module name is for example spi-rockchip.ko ('-' not '_' as in
the driver name). Also, rk8xx-spi and rk808-regulator are already there.

Please give them a try.
Comment 15 Torsten Duwe 2024-03-20 12:33:13 UTC
Confirmed, requested modules are e.g. in n-net.iso.

But I had to load them manually, and there is still something missing.
Comment 16 Steffen Winterfeldt 2024-03-20 12:36:02 UTC
Well, at least udev should load them. Is there a udev rule hidden in some package
that is missing?
Comment 17 Torsten Duwe 2024-03-20 15:02:25 UTC
No, the rest runs automatically once I manually load the "pl330" DMA driver, that's the missing link!
Comment 18 Steffen Winterfeldt 2024-03-20 15:34:41 UTC
Ok, I've updated ~snwint/Export/n-net.iso.

I've found an issue in mksusecd which made it fail working with
pre-TW images:

https://github.com/openSUSE/mksusecd/pull/71

Would be cool if you could try this. (You just need the new mksusecd script
from github.)

Apart from that I've no idea what went wrong on your machine. Two things:

- you must run it as root if you pass ISO files directly (but it should tell you
  that if you forget)
- Leap kernels are spread over 3 packages: kernel-default{,-extra,-optional}.rpm
  You'll need all three.
Comment 19 Torsten Duwe 2024-03-20 16:16:28 UTC
(In reply to Steffen Winterfeldt from comment #18)
> Ok, I've updated ~snwint/Export/n-net.iso.

Yes! That image smoothly starts a network install on my rock5!

Shall we also go for Leap-15.6, with kernel 6.4?
The pl330 is handled by generic code there (no module), the PMIC is a MFD ("rk808" ?), dw_mmc-rockchip is the same, and I'm now looking how spi-rockchip is handled...
Comment 20 Steffen Winterfeldt 2024-03-20 16:40:58 UTC
TW:

- https://github.com/openSUSE/installation-images/pull/700
Comment 21 Steffen Winterfeldt 2024-03-20 16:48:07 UTC
I've added all spi drivers, just in case.

We can basically use the same patch for sle15-sp6.
Comment 22 Torsten Duwe 2024-03-20 16:56:41 UTC
Off-Topic:

The RO errors here are due to my /tmp -> var/tmp symlink, which confuses
mksusecd line 87:
    if((split)[1] eq $mp) {
unmount fails, and the tmp remover tries to rm -rf the still RO-mounted ISO.

I'll simply use the save-temp option next time ;)
Comment 23 Steffen Winterfeldt 2024-03-20 17:17:34 UTC
Could you test this one?

- https://github.com/openSUSE/mksusecd/pull/72
Comment 24 Steffen Winterfeldt 2024-03-20 17:24:32 UTC
SLE15-SP6:

- https://github.com/openSUSE/installation-images/pull/701
Comment 27 Maintenance Automation 2024-06-18 16:30:09 UTC
SUSE-RU-2024:2071-1: An update that has six fixes can now be installed.

Category: recommended (moderate)
Bug References: 1213185, 1213606, 1214789, 1218706, 1221603, 1226047
Maintenance Incident: [SUSE:Maintenance:34213](https://smelt.suse.de/incident/34213/)
Sources used:
openSUSE Leap 15.6 (src):
 mksusecd-2.18-150600.3.3.1
Development Tools Module 15-SP6 (src):
 mksusecd-2.18-150600.3.3.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 28 Maintenance Automation 2024-06-18 16:30:12 UTC
SUSE-RU-2024:2070-1: An update that has six fixes can now be installed.

Category: recommended (moderate)
Bug References: 1213185, 1213606, 1214789, 1218706, 1221603, 1226047
Maintenance Incident: [SUSE:Maintenance:34214](https://smelt.suse.de/incident/34214/)
Sources used:
openSUSE Leap 15.5 (src):
 mksusecd-2.18-150500.3.3.1
Development Tools Module 15-SP5 (src):
 mksusecd-2.18-150500.3.3.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 29 Maintenance Automation 2024-06-18 16:30:14 UTC
SUSE-RU-2024:2069-1: An update that has six fixes can now be installed.

Category: recommended (moderate)
Bug References: 1213185, 1213606, 1214789, 1218706, 1221603, 1226047
Maintenance Incident: [SUSE:Maintenance:34215](https://smelt.suse.de/incident/34215/)
Sources used:
openSUSE Leap 15.4 (src):
 mksusecd-2.18-150400.3.18.2
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP4 (src):
 mksusecd-2.18-150400.3.18.2
SUSE Linux Enterprise High Performance Computing LTSS 15 SP4 (src):
 mksusecd-2.18-150400.3.18.2
SUSE Linux Enterprise Desktop 15 SP4 LTSS 15-SP4 (src):
 mksusecd-2.18-150400.3.18.2
SUSE Linux Enterprise Server 15 SP4 LTSS 15-SP4 (src):
 mksusecd-2.18-150400.3.18.2
SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src):
 mksusecd-2.18-150400.3.18.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.