Bug 1221763

Summary: ptrace(PTRACE_ATTACH) fails on processes of the same user
Product: [openSUSE] openSUSE Tumbleweed Reporter: Giuliano Belinassi <giuliano.belinassi>
Component: BasesystemAssignee: Johannes Segitz <jsegitz>
Status: NEW --- QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P5 - None CC: giuliano.belinassi, jslaby, lnussel, matz, mjambor, mrueckert, tdevries
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE Tumbleweed   
See Also: https://bugzilla.suse.com/show_bug.cgi?id=1217051
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Giuliano Belinassi 2024-03-20 14:34:46 UTC
Recent updates of tumbleweed broke `ptrace(PTRACE_ATTACH, ...)` when attaching to a process from the same user. This breaks attaching a debugger (gdb) to a process and userspace livepatching. A single line reproducer in a clean system is:

```
$ sleep 5000 & gdb -p $(pidof sleep)
```

If you see the following message:

```
Attaching to process 12606
ptrace: Operation not permitted.
```
This means ptrace is not working. As a contrast, running gdb with sudo works as intended.

Value of  /proc/sys/kernel/yama/ptrace_scope:
```
$ cat /proc/sys/kernel/yama/ptrace_scope
1
```
Comment 1 Giuliano Belinassi 2024-03-20 14:35:33 UTC
$ uname -a
Linux localhost.localdomain 6.7.9-1-default #1 SMP PREEMPT_DYNAMIC Thu Mar  7 06:07:11 UTC 2024 (6049de6) x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/os-release
NAME="openSUSE Tumbleweed"
# VERSION="20240318"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20240318"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
# CPE 2.3 format, boo#1217921
CPE_NAME="cpe:2.3:o:opensuse:tumbleweed:20240318:*:*:*:*:*:*:*"
#CPE 2.2 format
#CPE_NAME="cpe:/o:opensuse:tumbleweed:20240318"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"
Comment 2 Michael Matz 2024-03-20 15:28:22 UTC
Seems to have come in via https://bugzilla.suse.com/show_bug.cgi?id=1128245
CCing Jiri.  Maybe it's only the support at all that came in via the above and not
the default switch to "on"?

Either way, I don't think having this on by default is a good idea, it prevents
_each and all_ ptrace to non-childs (and hence debugging of running processes in
general), when not being root.  People who want system-wide ptrace separation
(and for unknown reasons don't want to use real sandboxes, like separate PID namespaces!?#) can enable this on an opt-in basis.
Comment 3 Jiri Slaby 2024-03-21 05:52:02 UTC
yama (and others) were enabled only in
commit 720c38318edb68b138a4bc4c86bb8ff0fbcda672
Author: Jeff Mahoney <jeffm@suse.com>
Date:   Thu Dec 8 14:32:18 2022 -0500

    config: update CONFIG_LSM defaults (bsc#1205603).

via bug 1205603. I wonder how it came noone noticed until now?
Comment 4 Jiri Slaby 2024-03-21 06:45:47 UTC
(In reply to Giuliano Belinassi from comment #0)
> Recent updates of tumbleweed broke `ptrace(PTRACE_ATTACH, ...)` when
> attaching to a process from the same user. This breaks attaching a debugger
> (gdb) to a process and userspace livepatching. A single line reproducer in a
> clean system is:
> 
> ```
> $ sleep 5000 & gdb -p $(pidof sleep)
> ```
> 
> If you see the following message:
> 
> ```
> Attaching to process 12606
> ptrace: Operation not permitted.
> ```
> This means ptrace is not working. As a contrast, running gdb with sudo works
> as intended.
> 
> Value of  /proc/sys/kernel/yama/ptrace_scope:
> ```
> $ cat /proc/sys/kernel/yama/ptrace_scope
> 1
> ```

So this actually works as expected and was supposed to work like this forever. Could you clarify what "Recent updates of tumbleweed" broke this?
Comment 5 Tom de Vries 2024-03-21 09:46:16 UTC
I can confirm this.

I booted into tumbleweed, and looked at the value of /proc/sys/kernel/yama/ptrace_scope.  It was 0.

Then I did zypper dup, and it changed to 1.  Same after a reboot.

I worked around this by adding "kernel.yama.ptrace_scope = 1" in /etc/sysctl.conf.
Comment 6 Martin Jambor 2024-03-21 09:59:48 UTC
(In reply to Jiri Slaby from comment #4)
> Could you clarify what "Recent updates of tumbleweed" broke this?

I'm not sure if it helps but the most recent TW that I can find where
/proc/sys/kernel/yama/ptrace_scope still defaults to 0 is "20240209."
For example TW with version "20240218" already defaults to 1.
Comment 7 Jiri Slaby 2024-03-21 11:19:15 UTC
(In reply to Martin Jambor from comment #6)
> (In reply to Jiri Slaby from comment #4)
> > Could you clarify what "Recent updates of tumbleweed" broke this?
> 
> I'm not sure if it helps but the most recent TW that I can find where
> /proc/sys/kernel/yama/ptrace_scope still defaults to 0 is "20240209."
> For example TW with version "20240218" already defaults to 1.

Do you switch between snapshots or only kernels?

1 is the default for over a decade in the kernel.

Maybe we used to set it in some package to 0 in /usr/lib/sysctl.d?

Could you:
  grep -r ptrace_scope /usr/lib/sysctl* /etc/sysctl*
in 20240209?
Comment 8 Jiri Slaby 2024-03-21 11:24:14 UTC
Got it.
/usr/lib/sysctl.d/52-yama.conf
coming from aaa_base

the change is:
https://build.opensuse.org/package/rdiff/Base:System/aaa_base?linkrev=base&rev=772

coming from:
https://github.com/openSUSE/aaa_base/commit/b59d2fdb3c03b1a85fd49d6f2f4b2b71e7313761

Without any explanation or reference.
Comment 9 Jiri Slaby 2024-03-21 11:26:08 UTC
The PR is more verbose:
https://github.com/openSUSE/aaa_base/pull/138

So this boils down to bug 1217051.
Comment 10 Johannes Segitz 2024-03-21 12:30:27 UTC
that's how it should be after the change. Users with different needs (which shouldn't be too many, not a lot of users do debugging of running processes) should change the sysctl setting
Comment 11 Tom de Vries 2024-03-21 13:09:42 UTC
If ptrace_scope=1 remains the default, this ( https://wiki.archlinux.org/title/Capabilities ) suggests a way of dealing with it without becoming root:
...
$ sudo -E capsh --caps="cap_setpcap,cap_setuid,cap_setgid+ep cap_sys_ptrace+eip" --keep=1 --user="$USER" --addamb="cap_sys_ptrace" --shell=/usr/bin/gdb -- -p <pid>
...
[ Doesn't work for me though on openSUSE Leap 15.4, I get "usage: capsh [args ...]", to be debugged. ]

I've filed a gdb PR ( https://sourceware.org/bugzilla/show_bug.cgi?id=31520 ) pointing out that IWBN to have a script somewhere that supports this.
Comment 12 Tom de Vries 2024-03-21 13:13:52 UTC
(In reply to Tom de Vries from comment #11)
> [ Doesn't work for me though on openSUSE Leap 15.4, I get "usage: capsh
> [args ...]", to be debugged. ]

Scripting problems...  This works:
...
$ cat sudo-allow-ptrace.sh
#!/bin/sh

shell="$1"
shift

set -x

sudo \
    -E capsh \
    --caps="cap_setpcap,cap_setuid,cap_setgid+ep cap_sys_ptrace+eip" \
    --keep=1 \
    --user="$USER" \
    --addamb="cap_sys_ptrace" \
    --shell=$shell -- \
    "$@"
$ sleep 60 & ./sudo-allow-ptrace.sh gdb -p $(pidof sleep)
  ...
$
...
Comment 13 Giuliano Belinassi 2024-03-21 13:47:45 UTC
In summary: will this change be reverted or we will have to handle it as a new default?
Comment 14 Michael Matz 2024-03-21 14:16:09 UTC
(In reply to Johannes Segitz from comment #10)
> that's how it should be after the change. Users with different needs (which
> shouldn't be too many, not a lot of users do debugging of running processes)
> should change the sysctl setting

So, just to be sure I completely understand that: because "Archer Allstars" complains about chrome sandboxing showing stuff in "red" and somewhere says "no",
the security team implemented this change without further discussion (or rather:
after it actually got rejected in jira) and without clear documentation of how to
get back a working system (or that such a far-reaching change was done at all, a
.changes entry in aaa_base reading "Restrict ptrace with Yama LSM by default" goes
unnoticed).

Well, super.  And there I thought the whole namespace container stuff was done
for separation, just to see that the non-containerized distro now goes down the drain as well.

So, how can I disable yama?  The whole module, all of it, not just this ptrace_scope.  I don't want to fiddle with it again if the security team decides to
further "enhance security" by randomly enabling other good-sounding options like "disable syscalls".
Comment 15 Johannes Segitz 2024-03-21 15:28:22 UTC
That's not really how it came to be, but I get the feeling that you're not really interested in a constructive dialogue, so lets not simulate one.

Just set the sysctl that matches your use case, so for you kernel.yama.ptrace_scope=0
Comment 16 Michael Matz 2024-03-21 17:03:26 UTC
(In reply to Johannes Segitz from comment #15)
> That's not really how it came to be, but I get the feeling that you're not
> really interested in a constructive dialogue, so lets not simulate one.

Indeed.

> Just set the sysctl that matches your use case, so for you
> kernel.yama.ptrace_scope=0

After reading
  https://github.com/torvalds/linux/blob/master/security/yama/yama_lsm.c
it's really only ptrace that yama deals with, so that sysctl is indeed the only
thing necessary to disable all of it.

Further, if I read the kernel docu correctly I should be able to disable all LSMs by adding 'lsm=capability' to the boot command line, which integrates better with
the disabling of spectre mitigations I'm doing anyway.  So I'll (try to) add that
to my list of things to do on new machine installs.
Comment 17 Marcus Rückert 2024-05-02 12:49:39 UTC
why not just add a sysctl file that gets installed with kernel-obs-build and be done? and i dont see why disabling spectre things in the build time kernel should be done.
Comment 18 Giuliano Belinassi 2024-05-02 13:56:38 UTC
(In reply to Marcus Rückert from comment #17)
> why not just add a sysctl file that gets installed with kernel-obs-build and
> be done? and i dont see why disabling spectre things in the build time
> kernel should be done.

We are experimenting with a custom package for that:

https://build.opensuse.org/package/show/home:gbelinassi/yama-disable-ptrace-scope

using /usr/sbin/sysctl -w kernel.yama.ptrace_scope=0 on %post seems to work for libpulp on tumbleweed. We will check if it also works for gdb.
Comment 19 Marcus Rückert 2024-05-02 14:32:10 UTC
yeah and a much cleaner approach would be shipping a proper file in kernel-obs-build
Comment 21 Ludwig Nussel 2024-05-06 08:14:09 UTC
RFC https://github.com/openSUSE/aaa_base/pull/149
Comment 22 Maintenance Automation 2024-05-27 12:30:01 UTC
SUSE-RU-2024:1785-1: An update that has two fixes can now be installed.

Category: recommended (moderate)
Bug References: 1221763, 1223306
Maintenance Incident: [SUSE:Maintenance:33680](https://smelt.suse.de/incident/33680/)
Sources used:
openSUSE Leap 15.4 (src):
 libpulp-0.3.3-150400.3.26.2
openSUSE Leap 15.5 (src):
 libpulp-0.3.3-150400.3.26.2
openSUSE Leap 15.6 (src):
 libpulp-0.3.3-150400.3.26.2
SUSE Linux Enterprise Live Patching 15-SP4 (src):
 libpulp-0.3.3-150400.3.26.2
SUSE Linux Enterprise Live Patching 15-SP5 (src):
 libpulp-0.3.3-150400.3.26.2
SUSE Linux Enterprise Live Patching 15-SP6 (src):
 libpulp-0.3.3-150400.3.26.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.