Bug 1221050

15-SP2 ppc64le is also affected (https://openqa.suse.de/tests/13725203) and I could reproduce the issue in a ppc64le VM on my laptop - I needed to build a go application in the registry.suse.com/bci/golang:latest container and then run this application in the registry.suse.de/suse/sle-15-sp6/update/cr/totest/images/bci/bci-base:15.6 container

> supermutant:~/hans # docker run --rm 123:latest
> runtime/cgo: pthread_create failed: Operation not permitted
> SIGABRT: abort
> PC=0x7fff9b834c1c m=0 sigcode=4294967290
> 
> goroutine 0 gp=0x4a2140 m=0 mp=0x4a2780 [idle]:
> runtime: g 0 gp=0x4a2140: unknown pc 0x7fff9b834c1c
> stack: frame={sp:0x7ffff6dacd00, fp:0x0} stack=[0x7ffff65b0000,0x7ffff6dad200)
> ...
> runtime: g 0 gp=0x4a2140: unknown pc 0x7fff9b834c1c
> stack: frame={sp:0x7ffff6dacd00, fp:0x0} stack=[0x7ffff65b0000,0x7ffff6dad200)
> ...
> 
> goroutine 1 gp=0xc0000021c0 m=0 mp=0x4a2780 [running]:
> runtime.systemstack_switch()
>         /usr/lib64/go/1.22/src/runtime/asm_ppc64x.s:212 +0x10 fp=0xc000034738 sp=0xc000034718 pc=0x853d0
> runtime.main()
>         /usr/lib64/go/1.22/src/runtime/proc.go:171 +0x7c fp=0xc0000347c0 sp=0xc000034738 pc=0x5135c
> runtime.goexit({})
>         /usr/lib64/go/1.22/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000347c0 sp=0xc0000347c0 pc=0x87f24
> 
> r0   0xfa       r1   0x7ffff6dacd00
> r2   0x7fff9b9e6d00     r3   0x0
> r4   0x1        r5   0x6
> r6   0x8        r7   0x7ffff6dace08
> r8   0x1        r9   0x0
> r10  0x0        r11  0x0
> r12  0x0        r13  0x7fff9bafb120
> r14  0xc000002a80       r15  0x3
> r16  0x0        r17  0xc000002000
> r18  0x6        r19  0x7
> r20  0x12       r21  0x7
> r22  0x7ffff65b03a0     r23  0x7ffff6dacfe8
> r24  0x900efb64b        r25  0x893838388
> r26  0x7c35f4564        r27  0x6
> r28  0x358f0a10 r29  0x7ffff6dacf70
> r30  0x7fff9baf3960     r31  0x1
> pc   0x7fff9b834c1c     ctr  0x0
> link 0x7fff9b834b5c     xer  0x0
> ccr  0x44420448 trap 0xc00


## Reproducer

main.go

> package main
> 
> import "net/http"
> 
> func main() {
>         _, err := http.Get("https://suse.com/")
>         if err != nil {
>                 panic(err)
>         }
> }

Dockerfile

> FROM registry.suse.com/bci/golang:latest as builder
> WORKDIR /src
> COPY main.go .
> RUN go build main.go
> 
> FROM registry.suse.de/suse/sle-15-sp6/update/cr/totest/images/bci/bci-base:15.6
> ENTRYPOINT []
> WORKDIR /fetcher/
> COPY --from=builder /src/main .
> CMD ["/fetcher/main"]

And then do

> # docker build -t 123 .
> # docker run --rm 123:latest
> ...
> runtime/cgo: pthread_create failed: Operation not permitted
> SIGABRT: abort

Just for completeness: The 15-SP2 ppc64le VM is fully patched, and the LTSS and Container module are enabled.

Also zypper inside a pristine 15-SP6 container shows a similar error

> supermutant:~/hans # docker run -ti --rm registry.suse.de/suse/sle-15-sp6/update/cr/totest/images/bci/bci-base:15.6
> b37fa36e893a:/ # zypper in strace
> terminate called after throwing an instance of 'std::system_error'
>   what():  Operation not permitted
> Aborted (core dumped)

It's still weird that it only happens on ppc64le.

It works with an unconfined seccomp profile:

> supermutant:~ # docker run -ti --rm --security-opt seccomp=unconfined registry.suse.de/suse/sle-15-sp6/update/cr/totest/images/bci/bci-base:15.6 zypper -n in strace
> Refreshing service 'container-suseconnect-zypp'.
> ...

background is:

SLES 15 SP2 libseccomp is outdated and does not know "clone3" system call.

I will queue an update for libseccomp.

Same on RES7.

More background: With the runc fixes for bug 1182451, syscalls unknown to libseccomp on any platform should be handled properly and result in -ENOSYS instead of -EPERM. Apparently that does not work on ppc64 and x86_64?

https://github.com/opencontainers/runc/pull/2750

Maybe the clone3 syscall number falls into one of the gaps of the profiles used by older runc.

(In reply to Fabian Vogt from comment #5)
> More background: With the runc fixes for bug 1182451, syscalls unknown to
> libseccomp on any platform should be handled properly and result in -ENOSYS
> instead of -EPERM. Apparently that does not work on ppc64 and x86_64?
> 
> https://github.com/opencontainers/runc/pull/2750
> 
> Maybe the clone3 syscall number falls into one of the gaps of the profiles
> used by older runc.

The Leap 15.2 x86_64 failure in openQA was because it pulled in ancient docker-runc for some reason, I guess the setup is broken.

If runc on affected systems is new enough to contain the linked PR (it should), please provide the output of

podman --debug --runtime-flag debug --runtime-flag log=runclog run --rm <15.6 image> zypper al foobar; cat runclog

Is it possible that it's broken if crun is used? https://github.com/containers/crun/issues/867

In that case it might be broken by design as it does not have a proper -ENOSYS fallback.

I can't figure out the Docker and runc versions used in the openQA tests because there doesn't appear to be a list of all installed packages available in the "Logs & Assets" tab.

To be clear, docker-runc should not be used anymore. We have an ETO open to remove it entirely from SLES, but it has been obsoleted by the correct package (runc) for at least 4 years now. In addition, Docker has required a minimum version of runc that docker-runc cannot provide for quite a few years as well, so unless the Docker package is truly ancient it seems very strange that docker-runc would still be used.

(In reply to Fabian Vogt from comment #5)
> More background: With the runc fixes for bug 1182451, syscalls unknown to
> libseccomp on any platform should be handled properly and result in -ENOSYS
> instead of -EPERM. Apparently that does not work on ppc64 and x86_64?
> 
> https://github.com/opencontainers/runc/pull/2750

As for this issue, an old runc will generate seccomp profiles that will return -EPERM for any syscall not specified in the seccomp profile for the container. This was fixed in runc 1.0.0. However, clone3 has specified as an allowed syscall in Docker's profile since 20.10.10[1], so even an old runc is not the issue if the Docker version is 20.10.10 (our 20.10.11 package was released in November 2021).

If libseccomp is old enough (clone3 was added to upstream libseccomp in 2.5.0, in 2020) that it doesn't know what "clone3" is, then an old runc along with an old libseccomp could result in this behaviour. Newer runcs should return -ENOSYS even with an old libseccomp that doesn't know about newer syscalls.

As an aside, there was also a kernel bug on s390x which would result in similar behaviour due to how syscall number multiplexing works on s390x. But this was an s390x-specific issue so it's not super relevant here (bug 1192594 has all the gory details).

> Maybe the clone3 syscall number falls into one of the gaps of the profiles
> used by older runc.

The profile is decided by Docker, and clone3 is in the set of new "unified" syscalls where the syscall numbers are assigned in order -- so it's quite unlikely this is the problem (though, never say never...).

[1]: https://github.com/moby/moby/pull/42836

Huh, it seems that the system is actually using the newer packages:

Client:
 Version:    24.0.7-ce

Server:
 Server Version: 24.0.7-ce
 containerd version: 8e4b0bde866788eec76735cc77c4720144248fb7
 runc version: v1.1.12-0-g51d5e94601ce

This is very strange...

I tried to take a look at the qcow image but the x86_64 one is missing very basic packages and so is unusable (zypper and docker in particular), and the ppc64le one appears to work but I don't know the root password (and because of bug 998663, I cannot mount the btrfs partition to change the root password because ppc64le has a PAGE_SIZE of 64k while my machine is 4k).

Does anyone know where I can get the root password for these images? Or is it randomly generated?

The requested package versions for docker and runc are:

> docker:    20.0.7_ce-98.106.1
> runc:      1.0.0rc10+gitr3981_dc9208a3303f-1.52.1

This is from the 12-SP5 ppc64le image

> I tried to take a look at the qcow image but the x86_64 one is missing very
> basic packages and so is unusable (zypper and docker in particular), and the
> ppc64le one appears to work but I don't know the root password (and because
> of bug 998663, I cannot mount the btrfs partition to change the root
> password because ppc64le has a PAGE_SIZE of 64k while my machine is 4k).

Be aware that RES7 is Liberty Linux, so zypper should not be used but you need to use yum instead. zypper is only there for registration purposes AFAIK and yes this is confusing.

Addendum: docker-runc is not used in the 12-SP5 ppc64le image.

(In reply to Aleksa Sarai from comment #9)
> Huh, it seems that the system is actually using the newer packages:
> 
> Client:
>  Version:    24.0.7-ce
> 
> Server:
>  Server Version: 24.0.7-ce
>  containerd version: 8e4b0bde866788eec76735cc77c4720144248fb7
>  runc version: v1.1.12-0-g51d5e94601ce

Now that I have access to the image, it turns out libseccomp is pre-2.5.0:

% runc --version
runc version 1.1.12
commit: v1.1.12-0-g51d5e94601ce
spec: 1.0.2-dev
go: go1.21.6
libseccomp: 2.4.1

Which means that clone3 is not included in the known syscall list. The seccomp stub generated by runc should handle this, but re-reading bug 1192051 reminded me of a comment by Fabian that I missed at the time:

(In reply to Fabian Vogt from Bug 1192051)
> I did some debugging. Updating just libseccomp2 did not help on either
> architecture.
> 
> I think it's two separate bugs for s390x and ppc64le.
> 
[snip]
> 
> On ppc64le, the patch looks like this:
> 
> [   0] ld [4]                  
> [   1] ja 2                    
> [   2] ja 1                    
> [   3] ret #327718             
> [....] --- original filter --- 
> 
> Which looks like there is some arch-specific issue, and it never return
> -ENOSYS.
> In addition, it does not appear to include faccessat2 in the seccomp policy,
> it's not found in runc's state.json and it did not call
> seccomp_syscall_resolve_name_arch with faccessat2 either.

I suspect this is the cause of the issue on ppc64le. I will take a look (Fabian also did some debugging of the issue at the time...).

---

As for x86_64, the runc version appears to be old (it is built in a way that it doesn't tell you the version but RPM lists the version as 1.0.0 which was released in 2021). libseccomp is also very old (2.3.1). I suspect an update of those packages would resolve the issue.

(In reply to Aleksa Sarai from comment #13)
> As for x86_64, the runc version appears to be old (it is built in a way that
> it doesn't tell you the version but RPM lists the version as 1.0.0 which was
> released in 2021). libseccomp is also very old (2.3.1). I suspect an update
> of those packages would resolve the issue.

After looking at the debugging information in the binary, the runc version is actually 1.0.0-rc90 (which was released in 2020). In short, for the x86_64 tests, you need a newer runc version because that bug was definitely fixed for x86_64 in v1.0.0.

There are also several security vulnerabilities that were fixed after 1.0.0-rc90 so the package absolutely needs to be updated to runc 1.1.12 regardless of this issue.

https://github.com/opencontainers/runc/pull/4219 should fix the ppc64le issue. Is it possible to set up an openQA run with that patch applied to the runc package?

(In reply to Aleksa Sarai from comment #15)
> https://github.com/opencontainers/runc/pull/4219 should fix the ppc64le
> issue. Is it possible to set up an openQA run with that patch applied to the
> runc package?

We need a rpm repository that can be injected into the test run. Maintenance SR are anyways tested.

If you need additional testing before that, please provide us a repo and we see if it's easier to do it within openQA or manually.

I put a build with the proposed patch into:

https://dist.suse.de/ibs/home:/msmeissn:/branches:/SUSE:/SLE-15:/Update/SUSE_SLE-15_Update/

(had to adjust the patch a bit as it did not apply clearly.)

(In reply to Marcus Meissner from comment #17)
> I put a build with the proposed patch into:
> 
> https://dist.suse.de/ibs/home:/msmeissn:/branches:/SUSE:/SLE-15:/Update/
> SUSE_SLE-15_Update/
> 
> (had to adjust the patch a bit as it did not apply clearly.)

Verified working on 15-SP2 ppc64le using the reproducer from https://bugzilla.suse.com/show_bug.cgi?id=1221050#c1.

Aleksa can you submit the fix for

SUSE:SLE-15:Update
SUSE:SLE-12:Update

?

SUSE-RU-2024:0981-1: An update that has two fixes can now be installed.

Category: recommended (important)
Bug References: 1192051, 1221050
Maintenance Incident: [SUSE:Maintenance:33022](https://smelt.suse.de/incident/33022/)
Sources used:
Containers Module 12 (src):
 runc-1.1.12-16.49.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2024:0984-1: An update that has two fixes can now be installed.

Category: recommended (important)
Bug References: 1192051, 1221050
Maintenance Incident: [SUSE:Maintenance:33023](https://smelt.suse.de/incident/33023/)
Sources used:
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP4 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise High Performance Computing LTSS 15 SP4 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Server 15 SP4 LTSS 15-SP4 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src):
 runc-1.1.12-150000.64.1
SUSE Enterprise Storage 7.1 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro 5.1 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro 5.2 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro for Rancher 5.2 (src):
 runc-1.1.12-150000.64.1
openSUSE Leap Micro 5.3 (src):
 runc-1.1.12-150000.64.1
openSUSE Leap Micro 5.4 (src):
 runc-1.1.12-150000.64.1
openSUSE Leap 15.5 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro for Rancher 5.3 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro 5.3 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro for Rancher 5.4 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro 5.4 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise Micro 5.5 (src):
 runc-1.1.12-150000.64.1
Containers Module 15-SP5 (src):
 runc-1.1.12-150000.64.1
SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src):
 runc-1.1.12-150000.64.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

This should be fixed now.

SUSE-RU-2024:0984-2: An update that has two fixes can now be installed.

Category: recommended (important)
Bug References: 1192051, 1221050
Maintenance Incident: [SUSE:Maintenance:33023](https://smelt.suse.de/incident/33023/)
Sources used:
openSUSE Leap 15.6 (src):
 runc-1.1.12-150000.64.1
Containers Module 15-SP6 (src):
 runc-1.1.12-150000.64.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.