Bugzilla – Bug 1215995
[Mediatek] mt8195-demo: please help to include these MediaTek drivers in initrd.img in CD/DVD release image
Last modified: 2023-11-01 03:17:48 UTC
Reference: [Ubuntu] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2035591 [Impact] Peripheral probe failure and boot to console failure for MediaTek boards 'mt8195-demo' and 'genio-1200-evk'. [Kernel version] 6.4~latest Note: 'COMMON_CLK_MT8195' has been refactored since kernel 6.4. Hence many 'clk-mt8195-foo.ko' are required in initrd.img. [MediaTek relate drivers] file: mediatek-drivers-for-mt8195-demo-bringup.txt (Not listed in probing sequence) Clock drivers: clk-mt8195-apusys_pll clk-mt8195-cam clk-mt8195-ccu clk-mt8195-img clk-mt8195-imp_iic_wrap clk-mt8195-ipe clk-mt8195-mfg clk-mt8195-scp_adsp clk-mt8195-vdec clk-mt8195-vdo0 clk-mt8195-vdo1 clk-mt8195-venc clk-mt8195-vpp0 clk-mt8195-vpp1 clk-mt8195-wpe clk-mt8195-msdc Other peripheral and power drivers: i2c-mt65xx spi-mt65xx reset-ti-syscon mt6397 rtc-mt6397 mtk-pmic-wrap mt6315-regulator spmi-mtk-pmif mtk_scp mtk_scp_ipi mediatek-drm mtk-vcodec-dec mtk-vcodec-enc mtk_jpeg mtk-vcodec-common mtk-jpeg-enc-hw mtk-vpu mtk-jpeg-dec-hw mtk-cmdq-helper mtk-cmdq-helper mtk-cmdq-mailbox mtk-mdp3 phy-mtk-mipi-dsi-drv btmtk leds-mt6360 tcpci_mt6360 mt6360_charger mt6360-regulator mt6360-core mt6359-regulator mt6360-adc snd-soc-mt8195-afe snd-soc-mtk-common snd-soc-dmic dwmac-mediatek stmmac-platform stmmac mtk-rng mtk_rpmsg pwm-mediatek pwm-mtk-disp nvmem_mtk-efuse mtk-sd cqhci phy-mtk-tphy mtu3 xhci-mtk-hcd mtk_iommu mtk-smi phy-mtk-pcie pcie-mediatek-gen3 [file: mediatek-drivers-for-mt8195-demo-bringup.txt] description: module drivers should be included in initrd.img [file: lsmod-i1200-demo-kernel-6.2-dracut-initrd.txt] description: lsmod-i1200-demo-kernel-6.2-dracut-initrd.txt [file: /etc/modprobe.d/mediatek.conf] description: module dependencies (Not optimized yet, some dependencies between modules might need to be fixed.) [Other info] effected kernel (6.4~latest) [Known issue] 1. USB port 1 failed (t-phy, xhci port 1, and hub port not working) port3 is working. 2. Ethernet driver loaded but not working. 3. PCIE not working.
Well, I can't believe that all of these needs to be part of initrd :-). I can accept that they are needed to have all peripheral devices work properly, but just to be able to mount rootfs, really?
I think some of the multimedia parts (video/camera encoder/decoders) could be excluded. But I think display might be necessary for installation with panels when uart is not available on some boards? Since Debian supports accessibility speech synthesis installation, I'm not sure if audio part could also be excluded? Video encode/decoders: mtk-vcodec-dec mtk-vcodec-enc mtk_jpeg mtk-vcodec-common mtk-jpeg-enc-hw mtk-vpu mtk-jpeg-dec-hw Audio: snd-soc-mt8195-afe snd-soc-mtk-common snd-soc-dmic Many of the clocks are share and they have some dependencies. I need to check which clocks are not related to display systems.
Created attachment 869976 [details] /etc/modprobe.d/mediatek.conf (version 20231006) /etc/modprobe.d/mediatek.conf (version 20231006) This is not the optimized version but the settings could help to bring up system to console with USB port 3 enabled, which could be used for U-disk installation.
(In reply to Macpaul Lin from comment #3) > Created attachment 869976 [details] > /etc/modprobe.d/mediatek.conf (version 20231006) > > /etc/modprobe.d/mediatek.conf (version 20231006) > This is not the optimized version but the settings could help to bring up > system to console with USB port 3 enabled, which could be used for U-disk > installation. Oops.. How can I delete this attachment? I've found there are some return/line feed characters issue of this file.
Created attachment 869977 [details] /etc/modprobe.d/mediatek.conf (version 20231006) This is not the optimized version but the settings could help to bring up system to console with USB port 3 enabled, which could be used for U-disk installation.
Created attachment 869978 [details] mediatek-drivers-for-mt8195-demo-bringup (version 20231006) Maybe this file could be saved as /etc/modules-load.d/mediatek.conf Some of the multimedia drivers could be removed if they are not related to display.
Created attachment 869979 [details] lsmod log for i1200-demo on kernel 6.2 with dracut made initrd.img (version 20231006) The clock were refactored since kernel 6.4. Hence clock modules were all built-in in kernel 6.2 and not included in this 'lsmod' dump log.
Hi, it will be of great help if you could narrow down minimal list of required modules which makes you platform bootable. Here what you could do: Download kernel-default package from here [1]. Click on "Grab binary packages directly" from section "Packages for ARM", get the one which end with aarch64.rpm. Save downloaded package. Download latest "minimal" Tumbleweed ISO from here [2]. Specify which modules you think are essential for device to boot. For example: MODULES="i2c-mt65xx spi-mt65xx pinctrl-mt8195 mt6315-regulator phy-mtk-tphy \ mtk-pmic-wrap mediatek-drm mt6359-regulator mt6357-regulator" Create driver upadate package (DUD) [3] from the downloaded kernel package: mkdud --create mediatek.dud --dist TW --install repo kernel-default* Rebuild installation ISO [4] init ramfs with your additional modules specified above. sudo mksusecd --verbose --create mediatek.iso --rebuild-initrd \ --initrd mediatek.dud --kernel kernel-default* \ --modules ${MODULES} -- openSUSE-Tumbleweed-NET-aarch64-Current.iso Test new mediatek.iso. Please try to narrow down required modules. Once you are happy with the result share the lit of modules here. Hope this helps. [1] https://software.opensuse.org//download.html?project=Kernel%3Astable&package=kernel-default [2] https://download.opensuse.org/ports/aarch64/tumbleweed/iso/openSUSE-Tumbleweed-NET-aarch64-Current.iso [3] https://github.com/openSUSE/mkdud [4] https://github.com/openSUSE/mksusecd
Hi Ivan, Thank you for providing very clear steps. I must say that Suse's 'mkdud' and 'mksusecd' are very easy to get started and use. Although there are no dpkg packages, I can use alien to convert them into .deb files, so I can run these debug steps on Ubuntu. When I put all the drivers (about 50+) into initrd.img, it's clear that most of the drivers can be detected. This also includes USB HOST and ethernet. After about half a day of cleaning up, it is now clear that to support the USB host driver (USB PORT 3) on the mt8195-demo board, the following drivers need to be added. This is base on the current iso images and kernel packages. For enable USB HOST: MODULES="clk-mt8195-imp_iic_wrap mt6360_charger mtk-pmic-wrap nvmem_mtk-efuse" The following are the binaries I've used: - kernel: kernel-default-6.5.6-1.1.gc97c2df.aarch64.rpm current-ISO: - openSUSE-Tumbleweed-NET-aarch64-Snapshot20231003-Media.iso I'll do next for enabling the ethernet driver.
Hi Steffen, Macpaul Lin is working on creating narrow list of modules which are required in inirtd for Mediatek demo board to boot (comment#9). Please could you help integrate required changes in installation-images once Macpaul Lin is ready? Thanks!
Hi Ivan and Steffen, To enable USB HOST and Ethernet driver for mt8195-demo boards more clock drivers are required. Solution: MODULES="clk-mt8195-imp_iic_wrap mt6360_charger mtk-pmic-wrap nvmem_mtk-efuse clk-mt8195-vdec clk-mt8195-venc clk-mt8195-vdo0 clk-mt8195-vdo1 clk-mt8195-vpp0 clk-mt8195-vpp1 clk-mt8195-wpe clk-mt8195-ipe clk-mt8195-cam clk-mt8195-ccu clk-mt8195-img" Compared previous found solution for enabling USB HOST. MODULES="clk-mt8195-imp_iic_wrap mt6360_charger mtk-pmic-wrap nvmem_mtk-efuse" There are more multi media clocks drivers are required, and I just could not to get rid any one from the following list to enable ethernet. "clk-mt8195-vdec clk-mt8195-venc clk-mt8195-vdo0 clk-mt8195-vdo1 clk-mt8195-vpp0 clk-mt8195-vpp1 clk-mt8195-wpe clk-mt8195-ipe clk-mt8195-cam clk-mt8195-ccu clk-mt8195-img" After checking the code, ethernet only depends on 'topckgen' and 'apmixsys' which should be a built-in clock driver already. Is it possible to create a cd-image with adding softdep to ethernet drivers? I'm not sure if re-order the probe sequence will help. The following logs are the 'lsmod' log for used ethernet drivers. realtek 36864 1 dwmac_mediatek 16384 0 stmmac_platform 32768 1 dwmac_mediatek stmmac 282624 3 stmmac_platform,dwmac_mediatek pcs_xpcs 20480 1 stmmac phylink 69632 2 stmmac,pcs_xpcs At last, I could use the modified 'openSUSE-Tumbleweed-DVD-aarch64-Snapshot20231010-Media.iso' with these driver to install Suse on to the mt8195-demo right now. However there still some issue blocking system boot after installation. I'll attach the boot log later. But at this stage, please help to add the solution into pre-built CD/DVD image at least we could get installer to work on mt8195-demo boards. Thanks.
Hi, By adding blasklist modules to kernel cmdline in grub I could boot the installation into console. "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" I'm not sure if this is because I've chosen 'server installation' hence I'm not sure if some display relate drivers were not included. Thank you for the supporting.
(In reply to Macpaul Lin from comment #12) > Hi, > > By adding blasklist modules to kernel cmdline in grub I could boot the > installation into console. Do you mean serial console? or tty console via any video port? > > "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" > > I'm not sure if this is because I've chosen 'server installation' hence I'm > not sure if some display relate drivers were not included. Thank you for the > supporting.
(In reply to Chester Lin from comment #13) > (In reply to Macpaul Lin from comment #12) > > Hi, > > > > By adding blasklist modules to kernel cmdline in grub I could boot the > > installation into console. > > Do you mean serial console? or tty console via any video port? > > > > > "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" > > > > I'm not sure if this is because I've chosen 'server installation' hence I'm > > not sure if some display relate drivers were not included. Thank you for the > > supporting. You may need to check if any DT dependencies [e.g. nodes and drivers they should match] required by mediatek-drm could be unavailable at that time since the module could be automatically deferred by kernel.
> Please could you help integrate required changes in > installation-images once Macpaul Lin is ready? Sure. Just ping we when you have the list.
(In reply to Chester Lin from comment #14) > (In reply to Chester Lin from comment #13) > > (In reply to Macpaul Lin from comment #12) > > > Hi, > > > > > > By adding blasklist modules to kernel cmdline in grub I could boot the > > > installation into console. > > > > Do you mean serial console? or tty console via any video port? Yes! Currently all the installation has been done via serial console UART0 of the board. > > > > > > "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" > > > > > > I'm not sure if this is because I've chosen 'server installation' hence I'm > > > not sure if some display relate drivers were not included. Thank you for the > > > supporting. > > You may need to check if any DT dependencies [e.g. nodes and drivers they > should match] required by mediatek-drm could be unavailable at that time > since the module could be automatically deferred by kernel. Unfortunate arm_dsu_pmu is not working on all MediaTek at this moment, we currently don't support it by default. An patch has been submitted to upstream via the maintainer. I guess there is some similar situation for the ffa-module. https://patchwork.kernel.org/project/linux-mediatek/patch/20230720200753.322133-1-nfraprado@collabora.com/ For DRM, I think I could try to add drivers via dracut and rerun the dracut to generate new initrd.img on the installed system and see if it could help.
Created attachment 870132 [details] /var/log/boot.msg generated when the system is booted via USB disk with CD-Image
Created attachment 870133 [details] dmesg dump on a reboot system after installation has been complete
Created attachment 870134 [details] /var/log/boot.log generated on a Suse installed mt8195-demo board.
(In reply to Macpaul Lin from comment #16) > (In reply to Chester Lin from comment #14) > > > You may need to check if any DT dependencies [e.g. nodes and drivers they > > should match] required by mediatek-drm could be unavailable at that time > > since the module could be automatically deferred by kernel. > > Unfortunate arm_dsu_pmu is not working on all MediaTek at this moment, we > currently don't support it by default. An patch has been submitted to > upstream via the maintainer. I guess there is some similar situation for the > ffa-module. > https://patchwork.kernel.org/project/linux-mediatek/patch/20230720200753. > 322133-1-nfraprado@collabora.com/ > > For DRM, I think I could try to add drivers via dracut and rerun the dracut > to generate new initrd.img on the installed system and see if it could help. Ah. there seems a sub-system driver design flaw when a device node was missing. Which will cause drm-mediatek probe failed. I've already report this issue to the driver owner, but they require some time to fix it.
(In reply to Steffen Winterfeldt from comment #15) > > Please could you help integrate required changes in > > installation-images once Macpaul Lin is ready? > > Sure. Just ping we when you have the list. Hi Steffen, To enable both USB HOST and Ethenet drivers we need to include the following drivers into the lists. USB: clk-mt8195-imp_iic_wrap mt6360_charger mtk-pmic-wrap nvmem_mtk-efuse Ethernet: clk-mt8195-vdec clk-mt8195-venc clk-mt8195-vdo0 clk-mt8195-vdo1 clk-mt8195-vpp0 clk-mt8195-vpp1 clk-mt8195-wpe clk-mt8195-ipe clk-mt8195-cam clk-mt8195-ccu clk-mt8195-img Thanks!
Done: https://github.com/openSUSE/installation-images/pull/665
Thank you Seffen! Do happen to know if there a way that Bugzilla is notified when new ISO with PR#665 is build and published?
> Do happen to know if there a way that Bugzilla > is notified when new ISO with PR#665 is build and published? Not to my knowledge. You have to check regularly.
(In reply to Steffen Winterfeldt from comment #22) > Done: > > https://github.com/openSUSE/installation-images/pull/665 Thanks for you all. I've tested 20231013's daily build. openSUSE-Tumbleweed-DVD-aarch64-Snapshot20231013-Media.iso The installation process on "mt8195-demo" board was very smooth. After booting, user need to add kernel CMDLINE with "module_blacklist=arm_dsu_pmu,ffa-module,drm,mediatek-drm" for avoiding unsupported drivers. I've also found adding "loglevel=8" is required. It seems there are some timing or dependency issues when probing the drivers. Next MediaTek will continue on enabling PCIE and display systems.
Thanks for the feedback Macpaul Lin. I would suggest to close this bug and open new one for the rest of the peripheral device support.
Verified with openSUSE-Tumbleweed-DVD-aarch64-Snapshot20231013-Media.iso daily build. (heart)