Bug 1172900

Summary: Videoconferencing apps on openSUSE --> much higher CPU and heat than Clear Linux
Product: [openSUSE] openSUSE Tumbleweed Reporter: S. B. <sb56637>
Component: KernelAssignee: openSUSE Kernel Bugs <kernel-bugs>
Status: RESOLVED WORKSFORME QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: mbenes, mkoutny, sb56637, tiwai
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: lots of screenshots and terminal outputs
Arch with Clear Linux kernel
openSUSE with Clear kernel vs stock openSUSE kernel
Clear kernel on openSUSE vs default from same repo
openSUSE TW with Mate desktop and stock kernel, long Zoom meeting
openSUSE TW with Mate desktop and Clear kernel, long Zoom meeting

Description S. B. 2020-06-14 20:12:19 UTC
Created attachment 838777 [details]
lots of screenshots and terminal outputs

Hi there, the pandemic has forced a lot of users to spend much more time in videoconferencing apps, which in turn has given more exposure to some of the inefficiencies in the Linux graphics stack. I've been fighting with Jitsi Meet via Firefox and Zoom making my laptop so hot that the fans go full speed, and it smells and feels hot. It's an all-Intel components Thinkpad T530. I also observe even higher CPU loads but slightly less palpable heat on my secondary Thinkpad T440s, also with all-Intel components. And of course, with the addtional CPU load comes much shorter battery life. I've also extensively tested other mainstream distros like the most recent Ubuntu, Fedora, and Manjaro releases with similar results, but since Tumbleweed is the one I actually want to use on this laptop I'm reporting this bug here.

As a last resort, I tried Clear Linux on the Thinkpad T530, since I heard Clear was optimized for Intel hardware. Surprisingly, it reported drastically lower CPU load and generally lower heat readings. And to the touch/smell the machine was obviously cooler and less stressed. The Firefox CPU load with Jitsi under Clear Linux was similar to what I have observed under Windows 8 on my Thinkpad T440s laptop. I retested several times in Clear under Gnome (Xorg session), and then several times more on my daily driver installed openSUSE Tumbleweed system with KDE Plasma (Xorg) and Openbox, and then I confirmed the excess CPU usage with a Tumbleweed live USB running Gnome on Xorg and Wayland.

I'm attaching a tarball of all my findings, since there are a lot of files. The filenames describe what OS and what applications are running. The Zoom test was with a simple 2 user meeting with first video and then screensharing. The Jitsi test was performed with just 2 users with video enabled on a fresh Firefox profile. I'm attaching Firefox's support output and Zoom's terminal output, as well as the loaded kernel modules. I'm also attaching the hwinfo for my Thinkpad T530, but I don't think it's particularly relevant because I have observed the same CPU loads with the same apps on all the other laptops I own running Linux. Please let me know if you need any other information, I'd be glad to help troubleshoot.

I'm not really an expert on the graphics stack in Linux, but I have tried to follow the Arch Wiki guidelines for enabling VA-API, and I've tried tons of options in about:config for Firefox on my installed openSUSE TW system. I'm sure it has packages installed that aren't present on the TW live Gnome system I booted, which is probably the reason for the differences in CPU load and temperature between the two systems. But both are drastically worse than Clear Linux straight out-of-the-box. Any ideas on what Clear Linux is doing differently? Is it just the result of thousands of little tweaks and optimizations, or is there some major configuration or compile flag in the graphics stack that Clear is using that could be applied to Tumbleweed?

Thanks a lot for the help.
Comment 1 Takashi Iwai 2020-06-15 08:05:05 UTC
You can try to build the kernel based on the config of Clear Linux manually on TW and see whether you get the given performance boost.  If not, it's a user-space problem.

Install linux git repo on the running Clear Linux system, set up the compiler and other tool chains, and run "make localmodconfig".  You might need to catch up more via "make oldconfig".  Change the CONFIG_LOCAL_VERSION string not to conflict with the existing one, make, and install the kernel, and confirm that the system works.

Then you can copy the .config file from there to TW, and build again there in a similar step.
Comment 2 S. B. 2020-06-15 13:06:25 UTC
Hi Takashi, thanks for the reply, would this be any easier?
https://github.com/clearlinux-pkgs/linux
Comment 3 Takashi Iwai 2020-06-15 13:23:58 UTC
Use whatever the config you can build the upstream vanilla kernel :)
The point is to build such a kernel on TW, and test whether you get the performance gain.
Comment 4 S. B. 2020-06-16 14:01:49 UTC
Created attachment 838855 [details]
Arch with Clear Linux kernel

Hi again, I found the Clear Linux kernel and kernel boot options available for Arch Linux:
https://software.opensuse.org/package/linux-clear
https://aur.archlinux.org/packages/linux-clear/#pinned-630000

This made it much easier for me to test it. I created an Arch installation on a live USB and installed the above linux-clear package and compared it with the stock Arch kernel. In summary, it does indeed appear that the Clear Linux kernel usually drastically reduces CPU load, whereas the stock Arch kernel behaves approximately the same as the stock openSUSE kernel. I found that Jitsi CPU loads were highly variable, so started testing with a similar service at http://appr.tc, which seems to produce more consistent results.

Please let me know if you need any more information. Thanks!
Comment 5 Takashi Iwai 2020-06-16 20:31:19 UTC
Thanks, that's an interesting finding.

Now I tried to build a kernel that contains Clear Linux downstream patches and the config aligned with it but slightly adjusted to be usable on TW.
It's being built in OBS home:tiwai:test:kernel-clear repo.  The new kernel package is "kernel-clear", not "kernel-default", and it'll be available at
  http://download.opensuse.org/repositories/home:/tiwai:/test:/kernel-clear/standard/x86_64/

Download kernel-clear-5.7.*.rpm from there and install it via
  zypper in kernel-clear-5.7.*.rpm
later.  And let me know if this brings anything better than kernel-default.
Comment 6 S. B. 2020-06-16 23:42:59 UTC
Created attachment 838878 [details]
openSUSE with Clear kernel vs stock openSUSE kernel

Thanks very much Takashi, you saved a ton of time with that.

It does look like the Clear kernel also makes a significant difference on openSUSE. I did the tests on my installed daily driver system, but with Openbox to eliminate some of the variables and to prevent my normal Plasma autostart apps from running. The only weird thing is that there doesn't seem to be a clear (pun not intended) advantage in terms of thermal performance, but that might just be a function of which kernel I tested first and how long the machine was running. Or it might be Clear being more aggressive with the processor. At any rate, I'll use this kernel in a "real" Zoom meeting that I have scheduled for this weekend, and I'll see how it feels, literally, in terms of heat and smell.

Please let me know if you'd like any more information. Thanks again!
Comment 7 Takashi Iwai 2020-06-17 10:28:16 UTC
Thanks, I'll wait for your feedback.

Meanwhile, I built kernel-default from the same source again.  It'll be published soon in the same URL above.  If the significant improvement is confirmed with kernel-clear, please try kernel-default from that repo and test whether you see the similar improvement.

If kernel-default in that repo gets improvements, it implies that it's due to some Clear Linux downstream patches.  OTOH, if kernel-default doesn't show the difference, it means that the key point is the kernel config, not the patches.
Comment 8 S. B. 2020-06-17 14:14:36 UTC
Created attachment 838896 [details]
Clear kernel on openSUSE vs default from same repo

Thanks a lot Takashi for helping me to test this.

So I tried the default kernel you compiled, and it behaves the same as the current Tumbleweed kernel and all the other major distros I've tested.

I also was wondering how much of a difference the kernel boot parameters make. From the AUR instructions I was using this with the Clear kernel:
> quiet console=tty0 console=ttyS0,115200n8 cryptomgr.notests initcall_debug intel_iommu=igfx_off kvm-intel.nested=1 no_timer_check noreplace-smp page_alloc.shuffle=1 rcupdate.rcu_expedited=1 rootfstype=ext4,btrfs,xfs,f2fs tsc=reliable rw 
So I tested the with just `quiet splash mitigations=off` with the Clear kernel, but apart from some boot warnings it performed about as well. Then I tried the recommended Clear parameter with your kernel-default, and it also threw some boot warnings and performed the same as before with the same kernel. So the performance differences don't seem to be due to the boot parameters.

I also realized that my thermal readings are probably being affected by the "ksgrd_network_helper" process that seems to be going a bit crazy, generating almost as much CPU load as Firefox itself. So I'll try to run some more tests later tonight with a lighter GUI system monitor.

Please let me know if you need any more info. Cheers.
Comment 9 S. B. 2020-06-23 17:51:26 UTC
So, a few updates:

I did a fresh install of Tumbleweed, now using the Mate desktop. Needed something more lightweight on this machine, and that alone has helped a lot with the thermal problems and excess CPU load.

Then I had an ~2 hour Zoom meeting with ~25 participants using the stock openSUSE 5.7.1 kernel, and then I had another similar Zoom meeting with the Clear kernel. There are indeed major differences in CPU load, although they don't seem to translate directly to a cooler system. However, I wonder if it's because Clear has set different temp thresholds for the fan speeds, because the fan seems to generally spin slower with the Clear kernel.
Comment 10 S. B. 2020-06-23 17:52:26 UTC
Created attachment 839043 [details]
openSUSE TW with Mate desktop and stock kernel, long Zoom meeting
Comment 11 S. B. 2020-06-23 17:52:47 UTC
Created attachment 839044 [details]
openSUSE TW with Mate desktop and Clear kernel, long Zoom meeting
Comment 14 Miroslav BeneŇ° 2022-01-14 14:37:18 UTC
S. B., just out of curiosity, has anything changed since June 2020? We have no resources to follow up on this, I am afraid.
Comment 15 S. B. 2022-01-14 15:03:15 UTC
(In reply to Miroslav BeneŇ° from comment #14)
> S. B., just out of curiosity, has anything changed since June 2020? We have
> no resources to follow up on this, I am afraid.

Hi there, thanks for the follow-up. I don't have any news either, I guess this bug can be closed. Cheers!