Bug 1215335

Summary: Enable CONFIG_REGULATOR_MT63XX=y by default on aarch64 (arm64) for MediaTek boards
Product: [openSUSE] openSUSE Tumbleweed Reporter: Macpaul Lin <macpaul.lin>
Component: KernelAssignee: Chester Lin <chester.lin>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P3 - Medium CC: 95kreaninw95, chester.lin, dmueller, forgotten_0079121656, ivan.ivanov, m4ng4n, macpaul.lin, mgorman, mhocko, qa-bugs, tiwai, vbabka
Version: CurrentFlags: tiwai: needinfo? (macpaul.lin)
Target Milestone: Current   
Hardware: aarch64   
OS: openSUSE Tumbleweed   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 1222818    

Description Macpaul Lin 2023-09-14 09:21:35 UTC
Reference:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2035591

Enabling CONFIG_REGULATOR_MT63XX=y (built-in) for mt8195-demo (and other MediaTek boards).
mt8195-demo: Fix regulator kernel setting are missing: causes peripheral probe failure for linux (include installer)

1. u-boot: [2023-04-26 14:34:28.294] run distro_bootcmd

2. disable silent mode in grub menu [2023-04-26 14:34:42.196] GNU GRUB version 2.06

3. boot to Linux
[2023-04-26 14:35:09.115] [ 0.000000] Linux version 6.2.0-18-generic (buildd@bos02-arm64-039) (aarch64-linux-gnu-gcc-12 (Ubuntu 12.2.0-17ubuntu1) 12.2.0, GNU ld (GNU Binutils for Ubuntu) 2.40) #18-Ubuntu SMP PREEMPT_DYNAMIC Sun Mar 19 19:49:50 UTC 2023 (Ubuntu 6.2.0-18.18-generic 6.2.6)
[2023-04-26 14:35:09.115] [ 0.000000] Machine model: MediaTek MT8195 demo board

4. lots of error and driver probing defered. USB, Ethernet, MMC and multimedia interfaces.
Ex: [2023-04-26 14:35:22.624] [ 13.808645] platform 1c014000.merge0: deferred probe pending

5. No ethernet and U-disk could be found
[2023-04-26 14:36:15.548] Unable to find a medium containing a live file system
[2023-04-26 14:36:15.548] Attempt interactive netboot from a URL?
[2023-04-26 14:36:15.548] yes no (default yes): no
[2023-04-26 14:36:21.432]
[2023-04-26 14:36:21.432]
[2023-04-26 14:36:21.432] BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1) built-in shell (ash)
[2023-04-26 14:36:21.432] Enter &#39;help&#39; for a list of built-in commands.
[2023-04-26 14:36:21.432]
[2023-04-26 14:36:21.432] (initramfs) Unable to find a medium containing a live file system

6. solution for regulator part:
Compare to generic kernel config for arm64
file: arch/arm64/configs/defconfig
The following settings should be enabled "built-in" in generic kernel settings.

CONFIG_REGULATOR_MT6357=y
CONFIG_REGULATOR_MT6358=y
CONFIG_REGULATOR_MT6359=y
CONFIG_REGULATOR_MT6360=y
CONFIG_REGULATOR_MT6397=y

7. effected kernel (5.19~latest)
Comment 1 Takashi Iwai 2023-09-14 09:37:59 UTC
What's the reason that those regulator driver *must* be built-in?
IOW, if those modules are included in the installation image, wouldn't it work?
If the driver probe ordering is the problem, it should be worked around in a different way than enforcing built-in, usually.
Comment 2 Michal Hocko 2023-09-14 11:28:40 UTC
And why has bug 1206937 been set as blocker for this?
Comment 3 Chester Lin 2023-09-14 13:04:25 UTC
(In reply to Takashi Iwai from comment #1)
> What's the reason that those regulator driver *must* be built-in?
> IOW, if those modules are included in the installation image, wouldn't it
> work?
> If the driver probe ordering is the problem, it should be worked around in a
> different way than enforcing built-in, usually.


CONFIG_MFD_MT6397(In reply to Takashi Iwai from comment #1)
> What's the reason that those regulator driver *must* be built-in?
> IOW, if those modules are included in the installation image, wouldn't it
> work?
> If the driver probe ordering is the problem, it should be worked around in a
> different way than enforcing built-in, usually.

I agree with Takashi. Maybe we should try having these regulator drivers as kernel modules in the initrd. Based on the current kernel tree for Tumbleweed, the CONFIG_MFD_MT6397 is not set so REGULATOR configs of MT6357, MT6358, MT6359, and MT6397 are missing. Should we enable them first?
Comment 4 Macpaul Lin 2023-09-15 02:28:00 UTC
According to the DTS, it seems these regulator are necessary for mmc and usb for installation process. For Ubuntu's case, I've tested built as module on an ubuntu ready board which leads boot fail (for example: no response when probing on board mmc).
However if SUSE could help to enable them as module in initrd for Tumbleweed, I could test new released image. That will be great for the debugging process. Thanks a lot. :)
Comment 5 Takashi Iwai 2023-09-15 09:25:04 UTC
If the problem is the ordering of the driver load, a workaround with softdep might work.  You can define the ordering there.  See "man modprobe.d"
Comment 6 Chester Lin 2023-09-18 08:31:06 UTC
I have created a PR to enable CONFIG_MFD_MT6397 and relevant arm64 regulator modules in the stable tree.
Comment 7 Macpaul Lin 2023-09-19 08:16:38 UTC
Thanks for the suggestion and the PR for enabling these regulators.

I've attempted to add these modules into /etc/initramfs-tools/modules. Most of them seem to work without the need for adding softdep. However, I still have some questions regarding this hardware enablement issue.

Q1. How can I upstream the other required modules list to SUSE to ensure these modules are built into the default initrd files for each binary release?

Q2. I've discovered that the module mtk_wdt.ko (CONFIG_MEDIATEK_WATCHDOG) must be set as built-in to prevent the system from encountering a hardware reset during the boot process. I've tried building it as a module and installing it into initrd, and I've also attempted to set softdep for it. However, I still can't find it being probed in the initcall debug dump. The system also reboots around 10 seconds into the boot time. After setting CONFIG_MEDIATEK_WATCHDOG=y, the hardware reset issue was resolved and the system was able to boot into the console. 

Should I modify it in config/arm64/default from https://github.com/openSUSE/kernel-source and then upload the patch for watchdog issue?
Comment 8 Chester Lin 2023-09-22 09:03:52 UTC
(In reply to Macpaul Lin from comment #7)
> Thanks for the suggestion and the PR for enabling these regulators.
> 
> I've attempted to add these modules into /etc/initramfs-tools/modules. Most
> of them seem to work without the need for adding softdep. However, I still
> have some questions regarding this hardware enablement issue.
> 

I assume that this should be in other distros, such as Ubuntu or Debian?

> Q1. How can I upstream the other required modules list to SUSE to ensure
> these modules are built into the default initrd files for each binary
> release?

I think you could create a PR here:

https://github.com/openSUSE/kernel-source

That's the public repo we synced from SUSE inside, but you may also have to contact branch maintainers since they might not always keep eyes on PRs from GitHub.

> 
> Q2. I've discovered that the module mtk_wdt.ko (CONFIG_MEDIATEK_WATCHDOG)
> must be set as built-in to prevent the system from encountering a hardware
> reset during the boot process. I've tried building it as a module and
> installing it into initrd, and I've also attempted to set softdep for it.
> However, I still can't find it being probed in the initcall debug dump. The
> system also reboots around 10 seconds into the boot time. After setting
> CONFIG_MEDIATEK_WATCHDOG=y, the hardware reset issue was resolved and the
> system was able to boot into the console. 

What's the expected timeout to kick the dog? 10 seconds? When did the watchdog device get probed and linked with a driver?

> 
> Should I modify it in config/arm64/default from
> https://github.com/openSUSE/kernel-source and then upload the patch for
> watchdog issue?

You could but please read the README file before doing it :-)

https://github.com/SUSE/kernel-source/blob/master/README
Comment 9 Takashi Iwai 2023-09-22 09:19:20 UTC
(In reply to Chester Lin from comment #8)
> > Q1. How can I upstream the other required modules list to SUSE to ensure
> > these modules are built into the default initrd files for each binary
> > release?
> 
> I think you could create a PR here:
> 
> https://github.com/openSUSE/kernel-source
> 
> That's the public repo we synced from SUSE inside, but you may also have to
> contact branch maintainers since they might not always keep eyes on PRs from
> GitHub.

In the case of SLE/openSUSE, the best would be just to report to Bugzilla (like this one).  By specifying the branches, we'll take the change / fix as much as possible.

But, as you see in this bug entry, the needed changes aren't necessarily in kernel package but rather in the other places such as dracut or installation-images.

In 99% cases, changing modules into built-in is no ideal workaround.  If there is a dependency, it has to be fixed in the driver side, and the issue for the upstream.
Comment 10 Michal Hocko 2023-09-22 10:46:45 UTC
(In reply to Michal Hocko from comment #2)
> And why has bug 1206937 been set as blocker for this?

I haven't heard any argument for this so I am dropping the dependency on the bug.
Comment 11 Macpaul Lin 2023-10-06 06:45:41 UTC
(In reply to Chester Lin from comment #8)
> (In reply to Macpaul Lin from comment #7)
> > Thanks for the suggestion and the PR for enabling these regulators.
> > 
> > I've attempted to add these modules into /etc/initramfs-tools/modules. Most
> > of them seem to work without the need for adding softdep. However, I still
> > have some questions regarding this hardware enablement issue.
> > 
> 
> I assume that this should be in other distros, such as Ubuntu or Debian?

Yes, initramfs-tools is used by Ubuntu and Debian.
I've found there are some problem when update driver's dependencies through update-initramfs (includes in package initramfs-tools).
I've found dracut is much easier to use for probing drivers and dependencies!
With dracut, most of the module dependencies could be detected automatically.
Then I could take the dependencies result of 'lsmod' with the 'initrd.img' generated by 'dracut' as a reference for 'initramfs-tools'

> > 
> > Q2. I've discovered that the module mtk_wdt.ko (CONFIG_MEDIATEK_WATCHDOG)
> > must be set as built-in to prevent the system from encountering a hardware
> > reset during the boot process. I've tried building it as a module and
> > installing it into initrd, and I've also attempted to set softdep for it.
> > However, I still can't find it being probed in the initcall debug dump. The
> > system also reboots around 10 seconds into the boot time. After setting
> > CONFIG_MEDIATEK_WATCHDOG=y, the hardware reset issue was resolved and the
> > system was able to boot into the console. 
> 
> What's the expected timeout to kick the dog? 10 seconds? When did the
> watchdog device get probed and linked with a driver?

This issue has been clarified with firmware (TF-A) update. 
No need to built-in watchdog driver anymore.
I'll create a new issue for updating recent debug result.

> > 
> > Should I modify it in config/arm64/default from
> > https://github.com/openSUSE/kernel-source and then upload the patch for
> > watchdog issue?
> 
> You could but please read the README file before doing it :-)
> 
> https://github.com/SUSE/kernel-source/blob/master/README

Got it!
Comment 12 Macpaul Lin 2023-10-06 06:51:31 UTC
I've tried openSUSE-Tumbleweed-DVD-aarch64-Snapshot20230922-Media.iso and confirmed these MediaTek's REGUATLORS has been included in initrd.img.
However it seems some power and clock driver still not probed correctly.

Fortunately one of my colleague suggested I to use dracut to replace initramfs-tools for help debug. Now the board with more peripheral function is working with dracut's help. I'll submit another issue for enabling more drivers in initrd.img.

This issue for enabling MediaTek's regulators could be closed. Thanks!