Bug 1197415 - gdb fails to find debugging symbols for libraries
Summary: gdb fails to find debugging symbols for libraries
Status: NEW
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Basesystem (show other bugs)
Version: Current
Hardware: Other Other
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: Tom de Vries
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-03-23 07:55 UTC by Dario Faggioli
Modified: 2022-03-24 12:22 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dario Faggioli 2022-03-23 07:55:37 UTC
Trying to debug a binary, like this: `gdb /usr/libexec/virtiofsd 7596`

I see the following:

> Reading symbols from /usr/libexec/virtiofsd...
> Reading symbols from /usr/lib/debug/usr/libexec/virtiofsd-6.2.0-42.1.x86_64.debug...
> Attaching to program: /usr/libexec/virtiofsd, process 7596
> [New LWP 7600]
> Error while mapping shared library sections:
> Could not open `target:/lib64/libseccomp.so.2' as an executable file: No such file or directory
> Error while mapping shared library sections:
> Could not open `target:/lib64/libcap-ng.so.0' as an executable file: No such file or directory
> Error while mapping shared library sections:
> Could not open `target:/lib64/libgmodule-2.0.so.0' as an executable file: No such file or directory
> Error while mapping shared library sections:
> Could not open `target:/lib64/libglib-2.0.so.0' as an executable file: No such file or directory
> Error while mapping shared library sections:
> Could not open `target:/lib64/libc.so.6' as an executable file: No such file or directory
> Error while mapping shared library sections:
> Could not open `target:/lib64/ld-linux-x86-64.so.2' as an executable file: No such file or directory
> Error while mapping shared library sections:
> Could not open `target:/lib64/libpcre.so.1' as an executable file: No such file or directory

And this is despite the fact that (I think!) I have the debugging symbols installed

> Solace:~ # rpm -qa|grep 'glibc\|libseccomp'
> glibc-2.35-2.1.x86_64
> libseccomp2-2.5.3-4.5.x86_64
> glibc-locale-2.35-2.1.x86_64
> glibc-extra-2.35-2.1.x86_64
> glibc-locale-base-2.35-2.1.x86_64
> glibc-debugsource-2.35-2.1.x86_64
> glibc-debuginfo-2.35-2.1.x86_64
> libseccomp-debugsource-2.5.3-4.5.x86_64
> libseccomp2-debuginfo-2.5.3-4.5.x86_64
> glibc-extra-debuginfo-2.35-2.1.x86_64

And:

> Solace:~ # ls -l /lib64/libc.so.6
> -rwxr-xr-x 1 root root 2551568 Mar  2 18:58 /lib64/libc.so.6
> Solace:~ # ls -l /usr/lib64/libc.so.6
> -rwxr-xr-x 1 root root 2551568 Mar  2 18:58 /usr/lib64/libc.so.6
> Solace:~ # ls -l /usr/lib/debug/usr/lib64/libc.so.6*
> -rw-r--r-- 1 root root 6211808 Mar  2 18:58 /usr/lib/debug/usr/lib64/libc.so.6-2.35-2.1.x86_64.debug

As we see, it fails with libraries, while it seems to work with binaries.

I tried installing `elfutils-debuginfod-0.186-2.1.x86_64` and starting the `debuginfod` service, but this does not change much.

Am I missing anything or misuing something?
Comment 1 Richard Biener 2022-03-23 10:31:08 UTC
Could not open `target:/lib64/libseccomp.so.2' as an executable file: No such file or directory

I'm not sure why gdb(?) tries to open this but for me it's in /usr/lib64, not /lib64.  Is virtiofsd maybe run inside some container/namespace that confuses gdb?
Comment 2 Michael Matz 2022-03-23 14:02:50 UTC
At least virtiofsd supports sandboxes/chroots:

% /usr/libexec/virtiofsd --help
...
    -o sandbox=namespace|chroot
                               sandboxing mode:
                               - namespace: mount, pid, and net
                                 namespaces with pivot_root(2)
                                 into shared directory
                               - chroot: chroot(2) into shared
                                 directory (use in containers)
                               default: namespace
...

(in particular: note the default!).

Attaching to running processes requires some heuristics in gdb in order to find
the correct files that the process has loaded, and in this case it seems that
those heuristics fail.  In some cases those heuristics can't find out the correct
rootdir at all, think about a process that loads "/foo/bar.so", then chroots
into "/X", loads "/a.so", then chroots into "/Y", then loads "/b.so".  The process
image of the process will say it loaded "/b.so" and "/a.so", but there's no
way to know that the latter was "/X/a.so", not "/X/Y/a.so" (should that exist).

So, it might be that gdb's heuristics could be made to work in this scenario,
until then you need to help gdb, e.g. by setting gdb's sysroot to "/":

(gdb) show sysroot
The current system root is "target:".
(gdb) set sysroot /

Giving to Tom for investigating if something could be improved on gdb's side
for this case.  For that, Dario: please provide a snippet of how you (or qemu)
starts virtiofsd, I didn't manage to start it myself to see your example break.
Comment 3 Dario Faggioli 2022-03-24 11:48:17 UTC
(In reply to Richard Biener from comment #1)
> Could not open `target:/lib64/libseccomp.so.2' as an executable file: No
> such file or directory
> 
> I'm not sure why gdb(?) tries to open this but for me it's in /usr/lib64,
> not /lib64.  
>
It's in both places, here in my system:

> # ls -l /usr/lib64/libseccomp.so.2
> lrwxrwxrwx 1 root root 19 Feb 19 22:52 /usr/lib64/libseccomp.so.2 -> libseccomp.so.2.5.3
> # ls -l /lib64/libseccomp.so.2
> lrwxrwxrwx 1 root root 19 Feb 19 22:52 /lib64/libseccomp.so.2 -> libseccomp.so.2.5.3

Maybe due to this?

> # ls -l /|grep lib64
> lrwxrwxrwx   1 root root    9 Mar  9 20:00 lib64 -> usr/lib64

> Is virtiofsd maybe run inside some container/namespace that
> confuses gdb?
>
cgroups are involved for sure, due to how libvirt configures things:

> # pstree -pa |grep virtiofs
>   |-virtiofsd,8045 --fd=34 -o source=/home/test/share -d
>   |   |-virtiofsd,8072 --fd=34 -o source=/home/test/share -d
>   |   |   |-{virtiofsd},8073
>   |   |   |-{virtiofsd},8395
>   |   |   `-{virtiofsd},8396
>   |   `-{virtiofsd},8048

# cat /proc/8045/task/*/cgroup
0::/machine.slice/machine-qemu\x2d1\x2dopensusetumbleweed\x2djeos.scope/libvirt/emulator
0::/machine.slice/machine-qemu\x2d1\x2dopensusetumbleweed\x2djeos.scope/libvirt/emulator

# cat /proc/8072/task/*/cgroup
0::/machine.slice/machine-qemu\x2d1\x2dopensusetumbleweed\x2djeos.scope/libvirt/emulator
0::/machine.slice/machine-qemu\x2d1\x2dopensusetumbleweed\x2djeos.scope/libvirt/emulator
0::/machine.slice/machine-qemu\x2d1\x2dopensusetumbleweed\x2djeos.scope/libvirt/emulator
0::/machine.slice/machine-qemu\x2d1\x2dopensusetumbleweed\x2djeos.scope/libvirt/emulator

Namespace wise, can you tell what kind of information I can share, to help you understand the situation? For now:

> # readlink /proc/8045/task/*/ns/* | sort -u
> cgroup:[4026531835]
> ipc:[4026531839]
> mnt:[4026531840]
> mnt:[4026532396]
> net:[4026531992]
> net:[4026532399]
> pid:[4026531836]
> pid:[4026532397]
> time:[4026531834]
> user:[4026531837]
> uts:[4026531838]

> # readlink /proc/8072/task/*/ns/* | sort -u
> cgroup:[4026531835]
> ipc:[4026531839]
> mnt:[4026532396]
> net:[4026532399]
> pid:[4026532397]
> time:[4026531834]
> user:[4026531837]
> uts:[4026531838]
Comment 4 Dario Faggioli 2022-03-24 12:22:28 UTC
(In reply to Michael Matz from comment #2)
> So, it might be that gdb's heuristics could be made to work in this scenario,
> until then you need to help gdb, e.g. by setting gdb's sysroot to "/":
> 
> (gdb) show sysroot
> The current system root is "target:".
> (gdb) set sysroot /
> 
> Giving to Tom for investigating if something could be improved on gdb's side
> for this case.  For that, Dario: please provide a snippet of how you (or
> qemu)
> starts virtiofsd, I didn't manage to start it myself to see your example
> break.
>
In theory, with something like this should work:

> # /usr/libexec/virtiofsd --socket-path=/tmp/vsock -o source=/home/test/share -o cache=always
> # qemu-system-x86_64 -M pc -cpu host --enable-kvm -smp 2 -m 4G \
>   -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem \
>   -chardev socket,id=char0,path=/tmp/vsock -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \
>   -chardev stdio,mux=on,id=mon -mon chardev=mon,mode=readline -device virtio-serial-pci -device virtconsole,chardev=mon \
>   -vga none -display none -drive if=virtio,file=/home/test/openSUSE-Tumbleweed-JeOS.x86_64-kvm-and-xen.qcow2

It's different than my usecase as, e.g., libvirt is not there creating the cgroups etc.

But I still see the problem if I try to attach with gdb.

Instead, in order to replicate exactly what I'm using, I guess the best way is to create a simple VM, e.g., with virt-manager, and then add a virtiofs shared filesystem, either from the GUI or adding this to the XML:

> <filesystem type="mount" accessmode="passthrough">
>   <driver type="virtiofs"/>
>   <source dir="/home/test/share"/>
>   <target dir="host_share"/>
> </filesystem>

Let me know if this helps.