Bug 1213785 - [Build 20230729] openQA test fails in perf
Summary: [Build 20230729] openQA test fails in perf
Status: CONFIRMED
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Kernel (show other bugs)
Version: Current
Hardware: Other Other
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: Tony Jones
QA Contact: E-mail List
URL: https://openqa.opensuse.org/tests/346...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-07-30 08:12 UTC by Dominique Leuenberger
Modified: 2024-06-29 06:59 UTC (History)
6 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Leuenberger 2023-07-30 08:12:34 UTC
## Observation

tcp_sendmsg is out of .text, skip it.
Probe point 'tcp_sendmsg' not found.
  Error: Failed to add events.

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-extra_tests_misc@64bit fails in
[perf](https://openqa.opensuse.org/tests/3468080/modules/perf/steps/33)

## Test suite description
Maintainer: zluo@suse.de
--
Tests: iotop, gd, perf and journalctlLevels --
see poo#93820 for details


## Reproducible

Fails since (at least) Build [20230729](https://openqa.opensuse.org/tests/3467348)


## Expected result

Last good: [20230728](https://openqa.opensuse.org/tests/3465334) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=extra_tests_misc&version=Tumbleweed)
Comment 1 Zaoliang Luo 2023-07-31 10:36:49 UTC
https://openqa.opensuse.org/tests/3469542/modules/perf/ looks good now.
Comment 2 Dominique Leuenberger 2023-08-02 08:03:55 UTC
(In reply to Zaoliang Luo from comment #1)
> https://openqa.opensuse.org/tests/3469542/modules/perf/ looks good now.

Last snapshot in QA still has perf faiures:

https://openqa.opensuse.org/tests/3474642#step/perf/35
Comment 3 Fabian Vogt 2023-08-09 17:22:59 UTC
I can reproduce the issue if debuginfod is used.
Setting a breakpoint at kprobe_warn_out_range shows that the address of the symbol is directly the address of the symbol in the debuginfod provided ELF file, while the map it is checked against uses the virtual addresses of the running kernel, including the KASLR offset. Depending on how big the KASLR offset is, this check fails.
Comment 4 Fabian Vogt 2023-08-09 18:21:14 UTC
It works when using the debuginfod provided file (somewhere in ~/.debug/...) as --vmlinux= explicitly.

For some reason it tries to load both the debuginfo file as well as /proc/kallsyms later. It probably gets confused by that and the loading of /proc/kallsyms, which contains virtual addresses with KASLR offset, sets some flag that indicates that all symbols are directly RIP mapped.
Comment 5 Fabian Vogt 2023-08-10 07:33:27 UTC
I tried it again today and there's also the issue that there is no indication that debuginfo is being downloaded, not even when using "perf probe -vv --add". As vmlinux.debug is > 500MiB large, the download can take a long time. It appears like the process is stuck for several minutes, unless setting DEBUGINFOD_PROGRESS=1.

I suggest to disable perf debuginfod integration for now until these shortcomings are fixed.