Bug 1213785

Summary:	[Build 20230729] openQA test fails in perf
Product:	[openSUSE] openSUSE Tumbleweed	Reporter:	Dominique Leuenberger <dimstar>
Component:	Kernel	Assignee:	Tony Jones <tonyj>
Status:	CONFIRMED ---	QA Contact:	E-mail List <qa-bugs>
Severity:	Normal
Priority:	P5 - None	CC:	fvogt, grasland, kernel-bugs, martin.liska, matz, zluo
Version:	Current
Target Milestone:	---
Hardware:	Other
OS:	Other
URL:	https://openqa.opensuse.org/tests/3468080/modules/perf/steps/33
Whiteboard:
Found By:	openQA	Services Priority:
Business Priority:		Blocker:	Yes
Marketing QA Status:	---	IT Deployment:	---

Description Dominique Leuenberger 2023-07-30 08:12:34 UTC

## Observation

tcp_sendmsg is out of .text, skip it.
Probe point 'tcp_sendmsg' not found.
  Error: Failed to add events.

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-extra_tests_misc@64bit fails in
[perf](https://openqa.opensuse.org/tests/3468080/modules/perf/steps/33)

## Test suite description
Maintainer: zluo@suse.de
--
Tests: iotop, gd, perf and journalctlLevels --
see poo#93820 for details


## Reproducible

Fails since (at least) Build [20230729](https://openqa.opensuse.org/tests/3467348)


## Expected result

Last good: [20230728](https://openqa.opensuse.org/tests/3465334) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=extra_tests_misc&version=Tumbleweed)

Comment 1 Zaoliang Luo 2023-07-31 10:36:49 UTC

https://openqa.opensuse.org/tests/3469542/modules/perf/ looks good now.

Comment 2 Dominique Leuenberger 2023-08-02 08:03:55 UTC

(In reply to Zaoliang Luo from comment #1)
> https://openqa.opensuse.org/tests/3469542/modules/perf/ looks good now.

Last snapshot in QA still has perf faiures:

https://openqa.opensuse.org/tests/3474642#step/perf/35

Comment 3 Fabian Vogt 2023-08-09 17:22:59 UTC

I can reproduce the issue if debuginfod is used.
Setting a breakpoint at kprobe_warn_out_range shows that the address of the symbol is directly the address of the symbol in the debuginfod provided ELF file, while the map it is checked against uses the virtual addresses of the running kernel, including the KASLR offset. Depending on how big the KASLR offset is, this check fails.

Comment 4 Fabian Vogt 2023-08-09 18:21:14 UTC

It works when using the debuginfod provided file (somewhere in ~/.debug/...) as --vmlinux= explicitly.

For some reason it tries to load both the debuginfo file as well as /proc/kallsyms later. It probably gets confused by that and the loading of /proc/kallsyms, which contains virtual addresses with KASLR offset, sets some flag that indicates that all symbols are directly RIP mapped.

Comment 5 Fabian Vogt 2023-08-10 07:33:27 UTC

I tried it again today and there's also the issue that there is no indication that debuginfo is being downloaded, not even when using "perf probe -vv --add". As vmlinux.debug is > 500MiB large, the download can take a long time. It appears like the process is stuck for several minutes, unless setting DEBUGINFOD_PROGRESS=1.

I suggest to disable perf debuginfod integration for now until these shortcomings are fixed.