Bug 1215995 - [Mediatek] mt8195-demo: please help to include these MediaTek drivers in initrd.img in CD/DVD release image
Summary: [Mediatek] mt8195-demo: please help to include these MediaTek drivers in init...
Status: RESOLVED FIXED
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Kernel (show other bugs)
Version: Current
Hardware: aarch64 openSUSE Tumbleweed
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: openSUSE Kernel Bugs
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 1216767
  Show dependency treegraph
 
Reported: 2023-10-06 10:09 UTC by Macpaul Lin
Modified: 2023-11-01 03:17 UTC (History)
10 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
/etc/modprobe.d/mediatek.conf (version 20231006) (953 bytes, text/plain)
2023-10-07 03:18 UTC, Macpaul Lin
Details
/etc/modprobe.d/mediatek.conf (version 20231006) (936 bytes, text/plain)
2023-10-07 03:26 UTC, Macpaul Lin
Details
mediatek-drivers-for-mt8195-demo-bringup (version 20231006) (924 bytes, text/plain)
2023-10-07 03:30 UTC, Macpaul Lin
Details
lsmod log for i1200-demo on kernel 6.2 with dracut made initrd.img (version 20231006) (6.13 KB, text/plain)
2023-10-07 05:11 UTC, Macpaul Lin
Details
/var/log/boot.msg generated when the system is booted via USB disk with CD-Image (60.53 KB, text/plain)
2023-10-12 15:56 UTC, Macpaul Lin
Details
dmesg dump on a reboot system after installation has been complete (72.37 KB, text/plain)
2023-10-12 15:58 UTC, Macpaul Lin
Details
/var/log/boot.log generated on a Suse installed mt8195-demo board. (65.54 KB, text/plain)
2023-10-12 16:00 UTC, Macpaul Lin
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Macpaul Lin 2023-10-06 10:09:14 UTC
Reference:
[Ubuntu]
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2035591

[Impact]
Peripheral probe failure and boot to console failure for MediaTek boards 'mt8195-demo' and 'genio-1200-evk'.

[Kernel version]
6.4~latest
Note: 'COMMON_CLK_MT8195' has been refactored since kernel 6.4.
Hence many 'clk-mt8195-foo.ko' are required in initrd.img.

[MediaTek relate drivers]
file: mediatek-drivers-for-mt8195-demo-bringup.txt
(Not listed in probing sequence)

Clock drivers:
clk-mt8195-apusys_pll
clk-mt8195-cam
clk-mt8195-ccu
clk-mt8195-img
clk-mt8195-imp_iic_wrap
clk-mt8195-ipe
clk-mt8195-mfg
clk-mt8195-scp_adsp
clk-mt8195-vdec
clk-mt8195-vdo0
clk-mt8195-vdo1
clk-mt8195-venc
clk-mt8195-vpp0
clk-mt8195-vpp1
clk-mt8195-wpe
clk-mt8195-msdc

Other peripheral and power drivers:
i2c-mt65xx
spi-mt65xx
reset-ti-syscon
mt6397
rtc-mt6397
mtk-pmic-wrap
mt6315-regulator
spmi-mtk-pmif
mtk_scp
mtk_scp_ipi
mediatek-drm
mtk-vcodec-dec
mtk-vcodec-enc
mtk_jpeg
mtk-vcodec-common
mtk-jpeg-enc-hw
mtk-vpu
mtk-jpeg-dec-hw
mtk-cmdq-helper
mtk-cmdq-helper
mtk-cmdq-mailbox
mtk-mdp3
phy-mtk-mipi-dsi-drv
btmtk
leds-mt6360
tcpci_mt6360
mt6360_charger
mt6360-regulator
mt6360-core
mt6359-regulator
mt6360-adc
snd-soc-mt8195-afe
snd-soc-mtk-common
snd-soc-dmic
dwmac-mediatek
stmmac-platform
stmmac
mtk-rng
mtk_rpmsg
pwm-mediatek
pwm-mtk-disp
nvmem_mtk-efuse
mtk-sd
cqhci
phy-mtk-tphy
mtu3
xhci-mtk-hcd
mtk_iommu
mtk-smi
phy-mtk-pcie
pcie-mediatek-gen3

[file: mediatek-drivers-for-mt8195-demo-bringup.txt]
description: module drivers should be included in initrd.img

[file: lsmod-i1200-demo-kernel-6.2-dracut-initrd.txt]
description: lsmod-i1200-demo-kernel-6.2-dracut-initrd.txt

[file: /etc/modprobe.d/mediatek.conf]
description: module dependencies
(Not optimized yet, some dependencies between modules might need to be fixed.)

[Other info]
effected kernel (6.4~latest)

[Known issue]
1. USB port 1 failed (t-phy, xhci port 1, and hub port not working) port3 is working.
2. Ethernet driver loaded but not working.
3. PCIE not working.
Comment 1 Ivan Ivanov 2023-10-06 11:23:41 UTC
Well, I can't believe that all of these needs to be part of initrd :-).

I can accept that they are needed to have all peripheral devices work
properly, but just to be able to mount rootfs, really?
Comment 2 Macpaul Lin 2023-10-07 03:13:12 UTC
I think some of the multimedia parts (video/camera encoder/decoders) could be excluded. But I think display might be necessary for installation with panels when uart is not available on some boards? Since Debian supports accessibility speech synthesis installation, I'm not sure if audio part could also be excluded?

Video encode/decoders:
mtk-vcodec-dec
mtk-vcodec-enc
mtk_jpeg
mtk-vcodec-common
mtk-jpeg-enc-hw
mtk-vpu
mtk-jpeg-dec-hw

Audio:
snd-soc-mt8195-afe
snd-soc-mtk-common
snd-soc-dmic

Many of the clocks are share and they have some dependencies.
I need to check which clocks are not related to display systems.
Comment 3 Macpaul Lin 2023-10-07 03:18:04 UTC
Created attachment 869976 [details]
/etc/modprobe.d/mediatek.conf (version 20231006)

/etc/modprobe.d/mediatek.conf (version 20231006)
This is not the optimized version but the settings could help to bring up system to console with USB port 3 enabled, which could be used for U-disk installation.
Comment 4 Macpaul Lin 2023-10-07 03:22:24 UTC
(In reply to Macpaul Lin from comment #3)
> Created attachment 869976 [details]
> /etc/modprobe.d/mediatek.conf (version 20231006)
> 
> /etc/modprobe.d/mediatek.conf (version 20231006)
> This is not the optimized version but the settings could help to bring up
> system to console with USB port 3 enabled, which could be used for U-disk
> installation.

Oops.. How can I delete this attachment? I've found there are some return/line feed characters issue of this file.
Comment 5 Macpaul Lin 2023-10-07 03:26:24 UTC
Created attachment 869977 [details]
/etc/modprobe.d/mediatek.conf (version 20231006)

This is not the optimized version but the settings could help to bring up system to console with USB port 3 enabled, which could be used for U-disk installation.
Comment 6 Macpaul Lin 2023-10-07 03:30:57 UTC
Created attachment 869978 [details]
mediatek-drivers-for-mt8195-demo-bringup (version 20231006)

Maybe this file could be saved as /etc/modules-load.d/mediatek.conf

Some of the multimedia drivers could be removed if they are not related to display.
Comment 7 Macpaul Lin 2023-10-07 05:11:20 UTC
Created attachment 869979 [details]
lsmod log for i1200-demo on kernel 6.2 with dracut made initrd.img (version 20231006)

The clock were refactored since kernel 6.4.
Hence clock modules were all built-in in kernel 6.2 and not included in this 'lsmod' dump log.
Comment 8 Ivan Ivanov 2023-10-10 08:32:53 UTC
Hi, it will be of great help if you could narrow down minimal list of
required modules which makes you platform bootable. Here what you could do:

Download kernel-default package from here [1].

Click on "Grab binary packages directly" from section "Packages for ARM",
get the one which end with aarch64.rpm. Save downloaded package.

Download latest "minimal" Tumbleweed ISO from here [2].

Specify which modules you think are essential for device to boot.

For example:

MODULES="i2c-mt65xx spi-mt65xx pinctrl-mt8195 mt6315-regulator phy-mtk-tphy \
         mtk-pmic-wrap mediatek-drm mt6359-regulator mt6357-regulator"

Create driver upadate package (DUD) [3] from the downloaded kernel package:

mkdud --create mediatek.dud --dist TW --install repo kernel-default*

Rebuild installation ISO [4] init ramfs with your additional modules specified above.

sudo mksusecd --verbose --create mediatek.iso --rebuild-initrd \
              --initrd mediatek.dud --kernel kernel-default* \
              --modules ${MODULES} -- openSUSE-Tumbleweed-NET-aarch64-Current.iso

Test new mediatek.iso. Please try to narrow down required modules. Once you
are happy with the result share the lit of modules here.

Hope this helps.

[1] https://software.opensuse.org//download.html?project=Kernel%3Astable&package=kernel-default
[2] https://download.opensuse.org/ports/aarch64/tumbleweed/iso/openSUSE-Tumbleweed-NET-aarch64-Current.iso
[3] https://github.com/openSUSE/mkdud
[4] https://github.com/openSUSE/mksusecd
Comment 9 Macpaul Lin 2023-10-11 09:49:15 UTC
Hi Ivan,

Thank you for providing very clear steps. I must say that Suse's 'mkdud' and 'mksusecd' are very easy to get started and use. Although there are no dpkg packages, I can use alien to convert them into .deb files, so I can run these debug steps on Ubuntu.

When I put all the drivers (about 50+) into initrd.img, it's clear that most of the drivers can be detected. This also includes USB HOST and ethernet. After about half a day of cleaning up, it is now clear that to support the USB host driver (USB PORT 3) on the mt8195-demo board, the following drivers need to be added. This is base on the current iso images and kernel packages.

For enable USB HOST:
MODULES="clk-mt8195-imp_iic_wrap mt6360_charger mtk-pmic-wrap nvmem_mtk-efuse"

The following are the binaries I've used:
 - kernel: kernel-default-6.5.6-1.1.gc97c2df.aarch64.rpm
current-ISO:
 - openSUSE-Tumbleweed-NET-aarch64-Snapshot20231003-Media.iso

I'll do next for enabling the ethernet driver.
Comment 10 Ivan Ivanov 2023-10-12 07:12:38 UTC
Hi Steffen, 

Macpaul Lin is working on creating narrow list of modules which
are required in inirtd for Mediatek demo board to boot (comment#9).

Please could you help integrate required changes in
installation-images once Macpaul Lin is ready?

Thanks!
Comment 11 Macpaul Lin 2023-10-12 07:22:23 UTC
Hi Ivan and Steffen,

To enable USB HOST and Ethernet driver for mt8195-demo boards more clock drivers are required.

Solution:
MODULES="clk-mt8195-imp_iic_wrap mt6360_charger mtk-pmic-wrap nvmem_mtk-efuse clk-mt8195-vdec clk-mt8195-venc clk-mt8195-vdo0 clk-mt8195-vdo1 clk-mt8195-vpp0 clk-mt8195-vpp1 clk-mt8195-wpe clk-mt8195-ipe clk-mt8195-cam clk-mt8195-ccu clk-mt8195-img"

Compared previous found solution for enabling USB HOST.
MODULES="clk-mt8195-imp_iic_wrap mt6360_charger mtk-pmic-wrap nvmem_mtk-efuse"

There are more multi media clocks drivers are required, and I just could not to get rid any one from the following list to enable ethernet.
"clk-mt8195-vdec clk-mt8195-venc clk-mt8195-vdo0 clk-mt8195-vdo1 clk-mt8195-vpp0 clk-mt8195-vpp1 clk-mt8195-wpe clk-mt8195-ipe clk-mt8195-cam clk-mt8195-ccu clk-mt8195-img"

After checking the code, ethernet only depends on 'topckgen' and 'apmixsys' which should be a built-in clock driver already. Is it possible to create a cd-image with adding softdep to ethernet drivers? I'm not sure if re-order the probe sequence will help.

The following logs are the 'lsmod' log for used ethernet drivers.
realtek                36864  1
dwmac_mediatek         16384  0
stmmac_platform        32768  1 dwmac_mediatek
stmmac                282624  3 stmmac_platform,dwmac_mediatek
pcs_xpcs               20480  1 stmmac
phylink                69632  2 stmmac,pcs_xpcs

At last, I could use the modified 'openSUSE-Tumbleweed-DVD-aarch64-Snapshot20231010-Media.iso' with these driver to install Suse on to the mt8195-demo right now. However there still some issue blocking system boot after installation. I'll attach the boot log later.

But at this stage, please help to add the solution into pre-built CD/DVD image at least we could get installer to work on mt8195-demo boards. Thanks.
Comment 12 Macpaul Lin 2023-10-12 08:12:52 UTC
Hi,

By adding blasklist modules to kernel cmdline in grub I could boot the installation into console.

"module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" 

I'm not sure if this is because I've chosen 'server installation' hence I'm not sure if some display relate drivers were not included. Thank you for the supporting.
Comment 13 Chester Lin 2023-10-12 08:44:23 UTC
(In reply to Macpaul Lin from comment #12)
> Hi,
> 
> By adding blasklist modules to kernel cmdline in grub I could boot the
> installation into console.

Do you mean serial console? or tty console via any video port?

> 
> "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" 
> 
> I'm not sure if this is because I've chosen 'server installation' hence I'm
> not sure if some display relate drivers were not included. Thank you for the
> supporting.
Comment 14 Chester Lin 2023-10-12 08:58:36 UTC
(In reply to Chester Lin from comment #13)
> (In reply to Macpaul Lin from comment #12)
> > Hi,
> > 
> > By adding blasklist modules to kernel cmdline in grub I could boot the
> > installation into console.
> 
> Do you mean serial console? or tty console via any video port?
> 
> > 
> > "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" 
> > 
> > I'm not sure if this is because I've chosen 'server installation' hence I'm
> > not sure if some display relate drivers were not included. Thank you for the
> > supporting.

You may need to check if any DT dependencies [e.g. nodes and drivers they should match] required by mediatek-drm could be unavailable at that time since the module could be automatically deferred by kernel.
Comment 15 Steffen Winterfeldt 2023-10-12 12:00:08 UTC
> Please could you help integrate required changes in
> installation-images once Macpaul Lin is ready?

Sure. Just ping we when you have the list.
Comment 16 Macpaul Lin 2023-10-12 15:52:51 UTC
(In reply to Chester Lin from comment #14)
> (In reply to Chester Lin from comment #13)
> > (In reply to Macpaul Lin from comment #12)
> > > Hi,
> > > 
> > > By adding blasklist modules to kernel cmdline in grub I could boot the
> > > installation into console.
> > 
> > Do you mean serial console? or tty console via any video port?

Yes! Currently all the installation has been done via serial console UART0 of the board. 

> > > 
> > > "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" 
> > > 
> > > I'm not sure if this is because I've chosen 'server installation' hence I'm
> > > not sure if some display relate drivers were not included. Thank you for the
> > > supporting.
> 
> You may need to check if any DT dependencies [e.g. nodes and drivers they
> should match] required by mediatek-drm could be unavailable at that time
> since the module could be automatically deferred by kernel.

Unfortunate arm_dsu_pmu is not working on all MediaTek at this moment, we currently don't support it by default. An patch has been submitted to upstream via the maintainer. I guess there is some similar situation for the ffa-module.
https://patchwork.kernel.org/project/linux-mediatek/patch/20230720200753.322133-1-nfraprado@collabora.com/

For DRM, I think I could try to add drivers via dracut and rerun the dracut to generate new initrd.img on the installed system and see if it could help.
Comment 17 Macpaul Lin 2023-10-12 15:56:30 UTC
Created attachment 870132 [details]
/var/log/boot.msg generated when the system is booted via USB disk with CD-Image
Comment 18 Macpaul Lin 2023-10-12 15:58:00 UTC
Created attachment 870133 [details]
dmesg dump on a reboot system after installation has been complete
Comment 19 Macpaul Lin 2023-10-12 16:00:40 UTC
Created attachment 870134 [details]
/var/log/boot.log generated on a Suse installed mt8195-demo board.
Comment 20 Macpaul Lin 2023-10-12 16:03:09 UTC
(In reply to Macpaul Lin from comment #16)
> (In reply to Chester Lin from comment #14)
>
> > You may need to check if any DT dependencies [e.g. nodes and drivers they
> > should match] required by mediatek-drm could be unavailable at that time
> > since the module could be automatically deferred by kernel.
> 
> Unfortunate arm_dsu_pmu is not working on all MediaTek at this moment, we
> currently don't support it by default. An patch has been submitted to
> upstream via the maintainer. I guess there is some similar situation for the
> ffa-module.
> https://patchwork.kernel.org/project/linux-mediatek/patch/20230720200753.
> 322133-1-nfraprado@collabora.com/
> 
> For DRM, I think I could try to add drivers via dracut and rerun the dracut
> to generate new initrd.img on the installed system and see if it could help.

Ah. there seems a sub-system driver design flaw when a device node was missing. Which will cause drm-mediatek probe failed. I've already report this issue to the driver owner, but they require some time to fix it.
Comment 21 Macpaul Lin 2023-10-12 16:05:54 UTC
(In reply to Steffen Winterfeldt from comment #15)
> > Please could you help integrate required changes in
> > installation-images once Macpaul Lin is ready?
> 
> Sure. Just ping we when you have the list.

Hi Steffen,
To enable both USB HOST and Ethenet drivers we need to include the following drivers into the lists.

USB:
clk-mt8195-imp_iic_wrap
mt6360_charger
mtk-pmic-wrap
nvmem_mtk-efuse

Ethernet:
clk-mt8195-vdec
clk-mt8195-venc
clk-mt8195-vdo0
clk-mt8195-vdo1
clk-mt8195-vpp0
clk-mt8195-vpp1
clk-mt8195-wpe
clk-mt8195-ipe
clk-mt8195-cam
clk-mt8195-ccu
clk-mt8195-img

Thanks!
Comment 22 Steffen Winterfeldt 2023-10-12 17:06:59 UTC
Done:

https://github.com/openSUSE/installation-images/pull/665
Comment 23 Ivan Ivanov 2023-10-17 07:24:03 UTC
Thank you Seffen! Do happen to know if there a way that Bugzilla
is notified when new ISO with PR#665 is build and published?
Comment 24 Steffen Winterfeldt 2023-10-17 07:44:44 UTC
> Do happen to know if there a way that Bugzilla
> is notified when new ISO with PR#665 is build and published?

Not to my knowledge. You have to check regularly.
Comment 25 Macpaul Lin 2023-10-17 09:22:33 UTC
(In reply to Steffen Winterfeldt from comment #22)
> Done:
> 
> https://github.com/openSUSE/installation-images/pull/665

Thanks for you all.

I've tested 20231013's daily build.

openSUSE-Tumbleweed-DVD-aarch64-Snapshot20231013-Media.iso

The installation process on "mt8195-demo" board was very smooth.
After booting, user need to add kernel CMDLINE with
"module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" for avoiding unsupported drivers.
I've also found adding "loglevel=8" is required. It seems there are some timing  or dependency issues when probing the drivers.

Next MediaTek will continue on enabling PCIE and display systems.
Comment 26 Ivan Ivanov 2023-10-17 09:36:23 UTC
Thanks for the feedback Macpaul Lin. I would suggest to close this bug
and open new one for the rest of the peripheral device support.
Comment 27 Macpaul Lin 2023-10-18 02:58:39 UTC
Verified with openSUSE-Tumbleweed-DVD-aarch64-Snapshot20231013-Media.iso daily build. (heart)