Bug 1212403 - [Build :29299:kernel-default] kernel is reserving more RAM
Summary: [Build :29299:kernel-default] kernel is reserving more RAM
Status: NEW
Alias: None
Product: PUBLIC SUSE Linux Enterprise Server 15 SP4
Classification: openSUSE
Component: Kernel (show other bugs)
Version: SLES15SP4-MaintUpd
Hardware: x86-64 Other
: P2 - High : Major
Target Milestone: ---
Assignee: Kernel Bugs
QA Contact:
URL: https://openqa.suse.de/tests/11286164...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-06-15 08:34 UTC by Jozef Pupava
Modified: 2023-06-22 12:56 UTC (History)
10 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jozef Pupava 2023-06-15 08:34:24 UTC
## Observation

After kernel update https://build.suse.de/request/show/300877 is kernel reserving more RAM.
With 2G RAM, has now system total RAM 1957Mi, before it was 1969Mi.
This is causing few kernel parameters having different values.
If it's expected, we have to inform SAP about this change.

Sysctl 	Sysctl_net_ipv4_tcp_mem
expected: 21800:23000 29500:31000 44800:46000
returned: 22326	29769	44652

Sysctl 	Sysctl_net_ipv4_udp_mem
expected: 44800:47000 59800:62000 89700:93000
returned: 44652	59538	89304

Sysctl 	Sysctl_user_max_cgroup_namespaces
expected: 7740:7880
returned: 7706

Sysctl 	Sysctl_user_max_ipc_namespaces
expected: 7740:7880
returned: 7706

Sysctl 	Sysctl_user_max_mnt_namespaces
expected: 7740:7880
returned: 7706

Sysctl 	Sysctl_user_max_net_namespaces
expected: 7740:7880
returned: 7706

Sysctl 	Sysctl_user_max_pid_namespaces
expected: 7740:7880
returned: 7706

Sysctl 	Sysctl_user_max_user_namespaces
expected: 7740:7880
returned: 7706

Sysctl 	Sysctl_user_max_uts_namespaces
expected: 7740:7880
returned: 7706

Sysctl 	Sysctl_vm_user_reserve_kbytes
expected: 61500:63000
returned: 61491

https://openqa.suse.de/tests/11286164/modules/Sysctl/steps/469


## Reproducible

Fails since (at least) Build [:29299:dtb-armv7l](https://openqa.suse.de/tests/11279279)


## Expected result

Last good: [:29295:java-1_8_0-ibm](https://openqa.suse.de/tests/11278780) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Incidents&machine=64bit-2gbram&test=mau-sles-sys-param-check&version=15-SP4)
Comment 2 Takashi Iwai 2023-06-16 07:40:28 UTC
I'm confused; what are you actually testing and what does it have to do with dtb-armv7l...?
Comment 3 Jozef Pupava 2023-06-16 08:04:41 UTC
(In reply to Takashi Iwai from comment #2)
> I'm confused; what are you actually testing and what does it have to do with
> dtb-armv7l...?

The test is checking kernel parameters, if they are different or out of expected range, then test will fail. In happened that some kernel parameters changed and customers had issues. Due to kernel update RAM related parameters are out of expected range.
Comment 4 Takashi Iwai 2023-06-16 08:23:14 UTC
But what does it have to do with dtb-armv7, while you're reporting for x86 hardware?
Comment 5 Jozef Pupava 2023-06-16 08:58:56 UTC
(In reply to Takashi Iwai from comment #4)
> But what does it have to do with dtb-armv7, while you're reporting for x86
> hardware?

Sorry, that was just autofilled, one of many patchinfos from https://build.suse.de/request/show/300877
Comment 6 Takashi Iwai 2023-06-16 13:31:29 UTC
OK, thanks.  So it's a regression between *-63 and *-66 (fd0cc4f7e3a0..98adc02ae60c).  Not so small range, 938 no-merged commits.

Is the problem specific to x86, or the same issue is seen on all architectures?
Comment 7 Jozef Pupava 2023-06-19 15:58:33 UTC
(In reply to Takashi Iwai from comment #6)
> OK, thanks.  So it's a regression between *-63 and *-66
> (fd0cc4f7e3a0..98adc02ae60c).  Not so small range, 938 no-merged commits.
> 
> Is the problem specific to x86, or the same issue is seen on all
> architectures?

aarch64 and s390x have same total RAM with and without update
Comment 8 Takashi Iwai 2023-06-19 16:07:11 UTC
Thanks.  So this appears to be some x86-specific changes.

Jiri, Nikolay, do you guys know of any possible changes?
Comment 9 Jiri Slaby 2023-06-20 05:58:56 UTC
> -Memory: 232188K/2096604K available (14344K kernel code, 3408K rwdata, 9192K rodata, 2840K init, 5936K bss, 353356K reserved, 0K cma-reserved)^M
> +Memory: 260860K/2096604K available (14344K kernel code, 3408K rwdata, 9196K rodata, 2840K init, 18224K bss, 365516K reserved, 0K cma-reserved)^M

BSS is +12M
reserved is + 11.8M

BSS is likely NOT connected to any x86 change. It would be some driver.

reserved comes from x86 changes, likely.
Comment 10 Jiri Slaby 2023-06-20 07:05:15 UTC
(In reply to Jiri Slaby from comment #9)
> BSS is +12M
> 
> BSS is likely NOT connected to any x86 change. It would be some driver.

It's an x86 "driver" after all :D.
commit 4cd1b963d5cf7e20b5bb37eea2790bf2deda4771
Author: Nikolay Borisov <nik.borisov@suse.com>
Date:   Thu May 4 16:19:31 2023 +0300

    x86/microcode/AMD: Fix mixed steppings support (git-fixes).

The patch (upstream commit 7ff6edf4fef3) does:
-static u8 amd_ucode_patch[PATCH_MAX_SIZE];
+static u8 amd_ucode_patch[MAX_NUMNODES][PATCH_MAX_SIZE];

MAX_NUMNODES = 1 << 10 = 1024
PATCH_MAX_SIZE = 12288

sizeof(u8)*MAX_NUMNODES*PATCH_MAX_SIZE = 12M
Comment 11 Sören Schmidt 2023-06-22 12:56:02 UTC
We have discussed the changes with the SAP LinuxLab (and the author of the tests).

The deviations are not critical. In fact the current thresholds are educated guesses since they had no formula to calculate possible outcomes.

For now we can increase the thresholds to:

Sysctl   Sysctl_net_ipv4_tcp_mem
expected: 21800:23000 29500:31000 *44500*:46000

Sysctl   Sysctl_net_ipv4_udp_mem
expected: *44400*:47000 *59400*:62000 *89000*:93000

Sysctl   Sysctl_user_max_cgroup_namespaces
expected: *7640*:7880

Sysctl   Sysctl_user_max_ipc_namespaces
expected: *7640*:7880

Sysctl   Sysctl_user_max_mnt_namespaces
expected: *7640*:7880

Sysctl   Sysctl_user_max_net_namespaces
expected: *7640*:7880
 
Sysctl   Sysctl_user_max_pid_namespaces
expected: *7640*:7880

Sysctl   Sysctl_user_max_user_namespaces
expected: *7640*:7880

Sysctl   Sysctl_user_max_uts_namespaces
expected: *7640*:7880

Sysctl   Sysctl_vm_user_reserve_kbytes
expected: *61000*:63000

After my vacation I'll check if we can create better well-founded thresholds. If you have insights, please let me know.