Bug 1166406 - Consider moving /boot/vmlinux-*.xz to a different package
Consider moving /boot/vmlinux-*.xz to a different package
Status: NEW
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Kernel
Current
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: openSUSE Kernel Bugs
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-03-11 16:25 UTC by Fabian Vogt
Modified: 2021-03-04 16:04 UTC (History)
11 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Fabian Vogt 2020-03-11 16:25:38 UTC
Currently the kernel is installed twice into /boot (at least on x86_64 and arm64), once as self-decompressing vmlinuz and then the raw compressed kernel as vmlinux.xz

The latter isn't needed for booting and only used for debugging or by using special tools (AFAIK, please correct me if I'm wrong), so not useful on most systems.

Especially with kernel-default-base, this ~10MiB file is almost 30% of the whole package in size, so dropping/moving that would have a noticeable impact on size reduction.
Comment 1 Takashi Iwai 2020-03-11 16:40:34 UTC
Sounds like a good idea.  Not sure where vmlinux*.xz should be moved to, though.

I thought crash utility required vmlinux, but it's obviously for debugging.
Comment 2 Michal Suchanek 2020-03-11 16:49:44 UTC
is it not needed for some *trace?
Comment 3 Takashi Iwai 2020-03-11 17:22:47 UTC
Maybe Tony knows of such?
Comment 4 Jeff Mahoney 2020-03-11 20:02:24 UTC
It's needed for pretty much anything that needs a symbol table (and also the -debuginfo package).
Comment 5 Michal Suchanek 2020-03-11 20:19:49 UTC
The point here is if also -debuginfo package is needed it can be moved there or some other -debugwhatever.

If there are tools that can work with the vmlinux only then those will break.
Comment 6 Jeff Mahoney 2020-03-11 20:33:06 UTC
I can't imagine what utility those tools could have.  vmlinux should have the debuginfo stripped out.
Comment 7 Michal Suchanek 2020-03-11 20:38:06 UTC
I don't think you need debuginfo for profiling.
Comment 8 Jeff Mahoney 2020-03-11 21:35:16 UTC
It appears perf top uses /proc/kcore to do its annotation and /proc/kallsyms to get the symbol table, for example.
Comment 9 Michal Suchanek 2020-03-11 21:43:49 UTC
There is one last problem:

If vmlinux is needed for debugging only how do we ensure it's installed with the debuginfo package if we package it separately?

debuginfo packages are created automagically so it's difficult to inject additional dependencies or files.
Comment 10 Tony Jones 2020-03-12 03:10:11 UTC
(In reply to Jeff Mahoney from comment #4)
> It's needed for pretty much anything that needs a symbol table (and also the
> -debuginfo package).

for perf,  /proc/kallsyms is generally used (subject to kptr_restrict. 
debuginfo vmlinux is used for annotation

i'd need to look closer at what exactly systemtap can do, without vmlinux,  without the debug version the answer is usually very little (some trace point stuff).

I think it's best to move slowly here.
Comment 11 Tony Jones 2020-03-12 03:11:26 UTC
(In reply to Michal Suchanek from comment #5)

> If there are tools that can work with the vmlinux only then those will break.

Correct. Hence my previous comment about moving slowly here. 

It may well be that most of these have been converted over to the various /proc interfaces (when full debuginfo isn't needed) but we should check.
Comment 12 Tony Jones 2020-03-12 17:23:38 UTC
For systemtap,  man 3 stapprobes breaks down what stap functions need DWARF (i.e debuginfo) and what need symbol table (i.e stripped vmlinux). Systemtap doesn't use /proc (like perf) for symbol-table info as it supports cross-instrumentation.

I've never actually looked at this as I always install debuginfo but as you can see below,  systemtap does make use of the stripped vmlinux in question (though it needs to be manually uncompressed)

Obviously we could ship this stripped vmlinux in a separate package and have systemtap depend on it (as we do for kernel-devel which is necessary to compile "hello world")

whether other tools make use of the stripped vmlinux is maybe a question for kernel list,  there may well be some I'm not aware of

------------------------------


# cat do_fork.stp 
probe kernel.function("_do_fork"){
	if (@defined($clone_flags)) 
		printf("do_fork: clone_flags=0x%lx\n", $clone_flags);
	else
		printf("do fork: no dwarf\n");
}

# ls -l /boot/vmlinux-4.12.14-197.34-default /usr/lib/debug/boot/vmlinux-4.12.14-197.34-default.debug
ls: cannot access '/boot/vmlinux-4.12.14-197.34-default': No such file or directory
ls: cannot access '/usr/lib/debug/boot/vmlinux-4.12.14-197.34-default.debug': No such file or directory

# stap do_fork.stp
semantic error: while resolving probe point: identifier 'kernel' at /home/tonyj/stap/do_fork.stp:1:7
        source: probe kernel.function("_do_fork"){
                      ^

semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/4.12.14-197.34-default/build'

Pass 2: analysis failed.  [man error::pass2]

# ls -l /boot/vmlinux-4.12.14-197.34-default /usr/lib/debug/boot/vmlinux-4.12.14-197.34-default.debug
ls: cannot access '/usr/lib/debug/boot/vmlinux-4.12.14-197.34-default.debug': No such file or directory
-rw-r--r-- 1 root root 51122824 Mar 12 09:51 /boot/vmlinux-4.12.14-197.34-default

# stap --poison-cache do_fork.stp
do fork: no dwarf
^C 

# ls -l /boot/vmlinux-4.12.14-197.34-default /usr/lib/debug/boot/vmlinux-4.12.14-197.34-default.debug
-rw-r--r-- 1 root root  51122824 Mar 12 09:51 /boot/vmlinux-4.12.14-197.34-default
-rw-r--r-- 2 root root 370477552 Feb 25 09:27 /usr/lib/debug/boot/vmlinux-4.12.14-197.34-default.debug
# 
# stap --poison-cache do_fork.stp
do_fork: clone_flags=0x1200011
^C
Comment 13 Jiri Slaby 2020-03-16 15:45:33 UTC
When I added support for xz compression of vmlinux in bug 1155921, I investigated and only kdump and crash read it. 

kdump only copies the image to /var/crash/...

crash uncompresses it and reads symbols from there.
Comment 14 Jiri Slaby 2020-03-16 15:51:33 UTC
(In reply to Michal Suchanek from comment #5)
> The point here is if also -debuginfo package is needed it can be moved there

I am strongly against moving it to -debuginfo. I don't want to download gigabytes just because of few MBs. I often live w/o debuginfo when debugging, symbols are far enough.

> or some other -debugwhatever.

But this is OK by me. Maybe simply -vmlinux.

pesign-obs-integration might need a change when the location or rpm name changes too.
Comment 15 Tony Jones 2020-03-16 19:34:19 UTC
(In reply to Jiri Slaby from comment #13)
> When I added support for xz compression of vmlinux in bug 1155921, I
> investigated and only kdump and crash read it. 

Per comment 12, this is clearly not true.

I don't thinking moving it to debuginfo is a good idea.
Comment 16 Jiri Slaby 2020-03-17 19:03:42 UTC
(In reply to Tony Jones from comment #15)
> (In reply to Jiri Slaby from comment #13)
> > When I added support for xz compression of vmlinux in bug 1155921, I
> > investigated and only kdump and crash read it. 
> 
> Per comment 12, this is clearly not true.

Right, I broke it by bug 1155921 then. I will fix it.
Comment 17 Jiri Slaby 2020-03-18 10:07:16 UTC
(In reply to Tony Jones from comment #12)
> # stap do_fork.stp
> semantic error: while resolving probe point: identifier 'kernel' at
> /home/tonyj/stap/do_fork.stp:1:7
>         source: probe kernel.function("_do_fork"){
>                       ^
> 
> semantic error: missing x86_64 kernel/module debuginfo [man
> warning::debuginfo] under '/lib/modules/4.12.14-197.34-default/build'

Hmm, I don't see this:
# ll /boot/vmlinux*
-rw-r--r-- 1 root root 10030200 Mar  9 14:20 /boot/vmlinux-5.5.7-1-default.xz
# strace -fo bubak -e trace=%file stap -v b.stap 
Pass 1: parsed user script and 476 library scripts using 97296virt/88352res/6588shr/81884data kb, in 280usr/90sys/559real ms.
Pass 2: analyzed script: 1 probe, 0 functions, 0 embeds, 0 globals using 119936virt/112752res/8172shr/104524data kb, in 1110usr/100sys/1225real ms.
Pass 3: translated to C into "/tmp/stapcPCpg1/stap_81ae959f3a822f58c54b6ac1ed4df3ae_916_src.c" using 119936virt/113000res/8420shr/104524data kb, in 990usr/60sys/1066real ms.
Pass 4: compiled C into "stap_81ae959f3a822f58c54b6ac1ed4df3ae_916.ko" in 16370usr/8760sys/37761real ms.
Pass 5: starting run.
do fork: no dwarf 
do fork: no dwarf 
^Cdo fork: no dwarf 
Pass 5: run completed in 20usr/200sys/50435real ms.
# grep '"/boot' bubak
24675 openat(AT_FDCWD, "/boot/System.map-5.5.7-1-default", O_RDONLY) = 3
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/.debug/vmlinux-5.5.7-1-default.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/.debug/vmlinux-5.5.7-1-default", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/build/vmlinux-5.5.7-1-default.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/build/vmlinux-5.5.7-1-default", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 lstat("/boot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
24675 lstat("/boot/vmlinux-5.5.7-1-default", 0x7ffc1e6778e0) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default.gz", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default.bz2", O_RDONLY) = -1 ENOENT (No such file or directory)
24675 openat(AT_FDCWD, "/boot/vmlinux-5.5.7-1-default.xz", O_RDONLY) = 3

So it can and does open .xz. Tony, do you have some old version of systemtap?
Comment 18 Jiri Slaby 2020-03-18 10:53:03 UTC
(In reply to Jiri Slaby from comment #17)
> So it can and does open .xz. Tony, do you have some old version of systemtap?

Stap seems to use libelf to handle that and this was added there by
commit 6ecdead8c0fbba51e8b2561e4d54dd7be3f69204
Author: Roland McGrath <roland@redhat.com>
Date:   Fri Feb 11 12:29:45 2011 -0800

    libdwfl: Search for Linux kernel binaries with compression file name suffixes.

So xz should not be a problem. That said, I don't understand what your issue is. Something seems to be broken on your system.
Comment 19 Tony Jones 2020-03-18 17:38:25 UTC
(In reply to Jiri Slaby from comment #18)
> (In reply to Jiri Slaby from comment #17)
> > So it can and does open .xz. Tony, do you have some old version of systemtap?
> 
> Stap seems to use libelf to handle that and this was added there by
> commit 6ecdead8c0fbba51e8b2561e4d54dd7be3f69204
> Author: Roland McGrath <roland@redhat.com>
> Date:   Fri Feb 11 12:29:45 2011 -0800
> 
>     libdwfl: Search for Linux kernel binaries with compression file name
> suffixes.
> 
> So xz should not be a problem. That said, I don't understand what your issue
> is. Something seems to be broken on your system.

I don't have an issue.   The vmlinux I was using wasn't xz compressed and so systemtap wasn't decompressing it.  Maybe it can decompress xz.

My point was that you said only "kdump and crash read it [vmlinux]" which isn't true, systemtap also does.  That is all.
Comment 20 Jiri Slaby 2020-03-19 08:45:23 UTC
(In reply to Tony Jones from comment #19)
> I don't have an issue.   The vmlinux I was using wasn't xz compressed and so
> systemtap wasn't decompressing it.  Maybe it can decompress xz.
> 
> My point was that you said only "kdump and crash read it [vmlinux]" which
> isn't true, systemtap also does.  That is all.

Aaah. It also looks (from the source code) that libdw (elfutils) can read also vmlinuz if there is no vmlinux.
Comment 21 Tony Jones 2020-03-19 15:49:27 UTC
(In reply to Jiri Slaby from comment #20)
> (In reply to Tony Jones from comment #19)
> > I don't have an issue.   The vmlinux I was using wasn't xz compressed and so
> > systemtap wasn't decompressing it.  Maybe it can decompress xz.
> > 
> > My point was that you said only "kdump and crash read it [vmlinux]" which
> > isn't true, systemtap also does.  That is all.
> 
> Aaah. It also looks (from the source code) that libdw (elfutils) can read
> also vmlinuz if there is no vmlinux.

This wasn't working for me when I tried it, so I had to manually decompress the installed vmlinux. Possible I'm using too old an elfutils, or issue was elsewhere.  Regardless, not relevant to this bug.  I'll verify it works off-line.
Comment 22 Tony Jones 2020-03-19 15:53:42 UTC
(In reply to Tony Jones from comment #21)

> This wasn't working for me when I tried it, so I had to manually decompress
> the installed vmlinux. Possible I'm using too old an elfutils, or issue was
> elsewhere.  Regardless, not relevant to this bug.  I'll verify it works
> off-line.

Sorry, I misread what you wrote. Ignore above. 

I knew elfutils decompression of vmlinux.* was possible for Systemtap but I was not aware that vmlinuz could be handled by Systemtap. I will look at it some more.
Comment 23 Tony Jones 2020-03-19 18:17:00 UTC
(In reply to Jiri Slaby from comment #20)

> Aaah. It also looks (from the source code) that libdw (elfutils) can read
> also vmlinuz if there is no vmlinux.

Our vmlinuz is stripped of all ELF symbol data.

# eu-readelf -s /boot/vmlinux-5.5.7-1-default.xz | wc -l
122468

# eu-readelf -s /boot/vmlinuz | wc -l
0
Comment 24 Michal Suchanek 2020-03-19 21:22:38 UTC
This is not the way to tell. You must first unpack it. You could 'prove' any rpm package is stripped of all symbols by running readelf on the rpm.
Comment 25 Tony Jones 2020-03-19 23:32:10 UTC
(In reply to Michal Suchanek from comment #24)
> This is not the way to tell. You must first unpack it. You could 'prove' any
> rpm package is stripped of all symbols by running readelf on the rpm.

My understanding (as was Jiri's from looking at the code) is that libdw handles the unpacking.
Comment 26 Tony Jones 2020-03-19 23:50:39 UTC
(In reply to Tony Jones from comment #25)
> (In reply to Michal Suchanek from comment #24)
> > This is not the way to tell. You must first unpack it. You could 'prove' any
> > rpm package is stripped of all symbols by running readelf on the rpm.
> 
> My understanding (as was Jiri's from looking at the code) is that libdw
> handles the unpacking.

Regardless, if I manually unpack the vmlinux from the vmlinuz using scripts/extract-vmlinux, the result contains no symbols.
Comment 27 Michal Suchanek 2020-03-31 18:51:02 UTC
get_kernel_version is using vmlinux (see bug 1162581)
Comment 28 Ludwig Nussel 2020-04-21 12:43:12 UTC
get_kernel_version works fine with vmlinuz too:

# for i in /boot/vmlinu*; do echo -n $i:; get_kernel_version $i; done
/boot/vmlinux-5.6.0-1-default.xz:/boot/vmlinuz:5.6.0-1-default
/boot/vmlinuz-5.6.0-1-default:5.6.0-1-default

Wrt the open question about how to install the package containing vmlinux. I guess what you want is to install that package if both kernel-default as well as kernel-default-debuginfo are installed. You can use boolean deps¹ for that.

So in kernel-default something like that should work:
Requires:       (%name-vmlinux = %version-%release if %name-debuginfo = %version-%release)

[1] https://rpm.org/user_doc/boolean_dependencies.html
Comment 29 Takashi Iwai 2020-04-21 12:49:50 UTC
A separate *-vmlinux package would give another benefit: we can package the uncompressed vmlinux if its installation is optional.  Some applications need the raw vmlinux file and user had to uncompress manually, and this step can be avoided.  Though, I'm not sure how many such applications exist now...
Comment 30 Tony Jones 2020-04-21 15:48:49 UTC
(In reply to Ludwig Nussel from comment #28)

> I guess what you want is to install that package if both kernel-default as
> well as kernel-default-debuginfo are installed. 

Maybe I'm misunderstanding.

How is that different from just adding vmlinux to the debuginfo package?

Also, as already shown, systemtap can make use of the symbol information in vmlinux without debuginfo.
Comment 31 Michal Suchanek 2020-05-06 08:34:49 UTC
get_kernel_version does not work on 32bit arm without vmlinux
Comment 32 Miroslav Beneš 2020-09-02 11:41:11 UTC
Has there been any new development?

tl;dr summary (hopefully correct) is that at least systemtap and get_kernel_version on arm need vmlinux. As far as systemtap is concerned, vmlinux could be stored in a different package. Ludwig proposed boolean deps for that in comment 28.
Comment 33 Fabian Vogt 2021-02-08 14:12:58 UTC
Any news? What are the next necessary steps?
Comment 34 Dominique Leuenberger 2021-03-04 12:22:17 UTC
(In reply to Michal Suchanek from comment #9)
> There is one last problem:
> 
> If vmlinux is needed for debugging only how do we ensure it's installed with
> the debuginfo package if we package it separately?
> 
> debuginfo packages are created automagically so it's difficult to inject
> additional dependencies or files.

(as noted by Ludwig already):
Requires: (kernel-default-vmlinux if kernel-default-debuginfo)

That would make the (new) -vmlinux mandatory as soon as -debuginfo is being triggered for install next to -default.

(In reply to Tony Jones from comment #30)
> (In reply to Ludwig Nussel from comment #28)
> 
> > I guess what you want is to install that package if both kernel-default as
> > well as kernel-default-debuginfo are installed. 
> 
> Maybe I'm misunderstanding.
> 
> How is that different from just adding vmlinux to the debuginfo package?
> 
> Also, as already shown, systemtap can make use of the symbol information in
> vmlinux without debuginfo.

It is different insofar that you can install -vmlinux without the -debuginfo (but not the -debuginfo without the -vmlinux)
Comment 35 Michal Suchanek 2021-03-04 12:47:34 UTC
if you move vmlinux into a separate package its debuginfo will move into a separate debuginfo package because it is created automagically.

So you will have kernel-default and kernel-default-vmlinux with the kernel debuginfo also split between kernel-default-debuginfo and kernel-default-vmlinux-debuginfo. Or am I missing something?
Comment 36 Dominique Leuenberger 2021-03-04 13:05:45 UTC
(In reply to Michal Suchanek from comment #35)
> if you move vmlinux into a separate package its debuginfo will move into a
> separate debuginfo package because it is created automagically.
> 
> So you will have kernel-default and kernel-default-vmlinux with the kernel
> debuginfo also split between kernel-default-debuginfo and
> kernel-default-vmlinux-debuginfo. Or am I missing something?

true, so we just make the dep a bit more complicated - but the system stays valid:

kernel-default:
Requires: ((kernel-default-vmlinux and kernel-default-vmlinux-debuginfo) if kernel-default-debuginfo)
Comment 37 Michal Suchanek 2021-03-04 13:18:10 UTC
And since what release are conditional dependencies available?
Comment 38 Dominique Leuenberger 2021-03-04 13:57:42 UTC
(In reply to Michal Suchanek from comment #37)
> And since what release are conditional dependencies available?

boolean deps exist since RPM 4.13
Comment 39 Michal Suchanek 2021-03-04 16:04:55 UTC
So since SLE15