Bug 1218114

Summary:	modprobe wireguard broken on ppc64le
Product:	[openSUSE] openSUSE Tumbleweed	Reporter:	Petr Vorel <petr.vorel>
Component:	Kernel:Networking	Assignee:	Kernel Bugs <kernel-bugs>
Status:	RESOLVED FIXED	QA Contact:	E-mail List <qa-bugs>
Severity:	Normal
Priority:	P3 - Medium	CC:	hector.oron, msuchanek, petr.pavlu, petr.vorel
Version:	Current
Target Milestone:	---
Hardware:	PowerPC-64
OS:	Other
See Also:	https://bugzilla.suse.com/show_bug.cgi?id=1217990 https://bugzilla.suse.com/show_bug.cgi?id=1218205
Whiteboard:
Found By:	---	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---

Description Petr Vorel 2023-12-15 16:55:25 UTC

It's not possible to modprobe wireguard on Tumbleweed ppc64le:
Failure started with kernel 6.6 (first failure on 6.6.1, last ok was in 6.5.9).

# modprobe wireguard; echo $?
modprobe: ERROR: could not insert 'wireguard': No such device
1

Module exists and modinfo works:

$ rpm -ql kernel-default |grep wireguard
/usr/lib/modules/6.6.6-1-default/kernel/drivers/net/wireguard
/usr/lib/modules/6.6.6-1-default/kernel/drivers/net/wireguard/wireguard.ko.zst

$ modinfo wireguard; echo $?
filename:       /usr/lib/modules/6.6.6-1-default/kernel/drivers/net/wireguard/wireguard.ko.zst
alias:          net-pf-16-proto-16-family-wireguard
...
0

# modprobe -v wireguard
insmod /usr/lib/modules/6.6.6-1-default/kernel/lib/crypto/libpoly1305.ko.zst 
insmod /usr/lib/modules/6.6.6-1-default/kernel/lib/crypto/libchacha.ko.zst 
insmod /usr/lib/modules/6.6.6-1-default/kernel/lib/crypto/libcurve25519-generic.ko.zst 
insmod /usr/lib/modules/6.6.6-1-default/kernel/net/ipv4/udp_tunnel.ko.zst 
insmod /usr/lib/modules/6.6.6-1-default/kernel/net/ipv6/ip6_udp_tunnel.ko.zst 
insmod /usr/lib/modules/6.6.6-1-default/kernel/arch/powerpc/crypto/chacha-p10-crypto.ko.zst 
modprobe: ERROR: could not insert 'wireguard': No such device


We have in our ppc64le CONFIG_CRYPTO_CHACHA20_P10=m

# strace modprobe -v wireguard
openat(AT_FDCWD, "/sys/module/ip6_udp_tunnel/initstate", O_RDONLY|O_CLOEXEC) = 3
read(3, "live\n", 31)                   = 5
read(3, "", 26)                         = 0
close(3)                                = 0
openat(AT_FDCWD, "/sys/module/chacha_p10_crypto/initstate", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/sys/module/chacha_p10_crypto", 0x7fffdd858a38, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/modules/6.6.6-1-default/kernel/arch/powerpc/crypto/chacha-p10-crypto.ko.zst", O_RDONLY|O_CLOEXEC) = 3
read(3, "(\265/\375d[", 6)              = 6
_llseek(3, 0, [0], SEEK_SET)            = 0
read(3, "(\265/\375d", 5)               = 5
mmap(NULL, 196608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fff980f0000
read(3, "[S\315\343\0", 5)              = 5
brk(0x10036350000)                      = 0x10036350000
read(3, "\312h\25RY \216\271;kv(\357b\343\324\263\310\250\306\227\3169\224\362\303\326\\\22+(\301"..., 7289) = 7289
mremap(0x7fff980f0000, 196608, 327680, MREMAP_MAYMOVE) = 0x7fff980a0000
read(3, " y&\220", 4)                   = 4
read(3, "", 4)                          = 0
brk(0x10036340000)                      = 0x10036340000
brk(0x10036330000)                      = 0x10036330000
init_module(0x7fff980a0010, 21595, "")  = -1 ENODEV (No such device)
write(2, "modprobe: ERROR: could not inser"..., 62modprobe: ERROR: could not insert 'wireguard': No such device
) = 62


I looked also at chacha-p10-crypto.ko.zst (the dependency), it's also installed but cannot be loaded:

# modprobe -v chacha-p10-crypto; echo $?
insmod /usr/lib/modules/6.6.6-1-default/kernel/arch/powerpc/crypto/chacha-p10-crypto.ko.zst 
modprobe: ERROR: could not insert 'chacha_p10_crypto': No such device
1

# strace modprobe chacha-p10-crypto
...
openat(AT_FDCWD, "/sys/module/chacha_p10_crypto/initstate", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/sys/module/chacha_p10_crypto", 0x7fffc922d618, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/modules/6.6.6-1-default/kernel/arch/powerpc/crypto/chacha-p10-crypto.ko.zst", O_RDONLY|O_CLOEXEC) = 3
read(3, "(\265/\375d[", 6)              = 6
_llseek(3, 0, [0], SEEK_SET)            = 0
read(3, "(\265/\375d", 5)               = 5
mmap(NULL, 196608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fff88680000
read(3, "[S\315\343\0", 5)              = 5
read(3, "\312h\25RY \216\271;kv(\357b\343\324\263\310\250\306\227\3169\224\362\303\326\\\22+(\301"..., 7289) = 7289
mremap(0x7fff88680000, 196608, 327680, MREMAP_MAYMOVE) = 0x7fff88630000
read(3, " y&\220", 4)                   = 4
read(3, "", 4)                          = 0
init_module(0x7fff88630010, 21595, "")  = -1 ENODEV (No such device)
write(2, "modprobe: ERROR: could not inser"..., 70modprobe: ERROR: could not insert 'chacha_p10_crypto': No such device
) = 70


I wonder if this can be related to bsc#1217990 (wireguard works on aarch64 and x86_64, that's because chacha-p10-crypto is ppc64le specific module):

ls -la /usr/lib/sysctl.d/ /etc/sysctl.conf
ls: cannot access '/etc/sysctl.conf': No such file or directory
/usr/lib/sysctl.d/:
total 384
drwxr-xr-x 1 root root  156 Dec 13 16:03 .
dr-xr-xr-x 1 root root  988 Dec 13 18:16 ..
-rw-r--r-- 1 root root 1816 Dec 12 14:01 50-coredump.conf
-rw-r--r-- 1 root root 2280 Oct 23 03:13 50-default.conf
-rw-r--r-- 1 root root  252 Oct 23 03:13 51-network.conf
-rw-r--r-- 1 root root  951 Oct 23 03:13 52-yama.conf
lrwxrwxrwx 1 root root   24 Dec 12 14:03 99-sysctl.conf -> ../../../etc/sysctl.conf
-rw-r--r-- 1 root root  387 Dec 12 02:27 README

For a record, found with ltp_net_features test https://openqa.opensuse.org/tests/3810783, where I also did the debugging.

Comment 1 Michal Suchanek 2023-12-19 13:21:57 UTC

Obviously if you do not have a Power10 machine chacha-p10-crypto won't load. A generic fallback chacha module should be available.

Comment 2 Petr Vorel 2024-01-18 16:16:41 UTC

(In reply to Michal Suchanek from comment #1)
> Obviously if you do not have a Power10 machine chacha-p10-crypto won't load.
> A generic fallback chacha module should be available.

Michal, yes, test is running on POWER8E. 

"ip link add ltp_v0 type wireguard" fails with "Error: Unknown device type."

There is only libchacha loaded by ip (and vmx_crypto, but that was not triggered by ip). Should it work with it or not on this older HW?

Comment 3 Michal Suchanek 2024-01-18 16:32:35 UTC

If the chacha cipher is needed it should work with this one:

/lib/modules/6.4.11-lp154.2.g2a5b3f6-default/kernel/crypto/chacha_generic.ko.zst

and it should get loaded automatically when chacha-p10-crypto fails to load.

The situation is similar on most architectures except for the fact that the hardware-specific module is likely always applicable. That means if the fallback mechanism is broken nobody might have noticed until now:

config/arm64/default:CONFIG_CRYPTO_CHACHA20=m
config/arm64/default:CONFIG_CRYPTO_CHACHA20POLY1305=m
config/arm64/default:CONFIG_CRYPTO_CHACHA20_NEON=m
config/arm64/default:CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
config/arm64/default:CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
config/arm64/default:CONFIG_CRYPTO_LIB_CHACHA=m
config/arm64/default:CONFIG_CRYPTO_LIB_CHACHA20POLY1305=m
config/armv7hl/default:CONFIG_CRYPTO_CHACHA20=m
config/armv7hl/default:CONFIG_CRYPTO_CHACHA20POLY1305=m
config/armv7hl/default:CONFIG_CRYPTO_CHACHA20_NEON=m
config/armv7hl/default:CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
config/armv7hl/default:CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
config/armv7hl/default:CONFIG_CRYPTO_LIB_CHACHA=m
config/armv7hl/default:CONFIG_CRYPTO_LIB_CHACHA20POLY1305=m
config/ppc64le/default:CONFIG_CRYPTO_CHACHA20=m
config/ppc64le/default:CONFIG_CRYPTO_CHACHA20POLY1305=m
config/ppc64le/default:CONFIG_CRYPTO_CHACHA20_P10=m
config/ppc64le/default:CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
config/ppc64le/default:CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
config/ppc64le/default:CONFIG_CRYPTO_LIB_CHACHA=m
config/ppc64le/default:CONFIG_CRYPTO_LIB_CHACHA20POLY1305=m
config/s390x/default:CONFIG_CRYPTO_CHACHA20=m
config/s390x/default:CONFIG_CRYPTO_CHACHA20POLY1305=m
config/s390x/default:CONFIG_CRYPTO_CHACHA_S390=m
config/s390x/default:CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
config/s390x/default:CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
config/s390x/default:CONFIG_CRYPTO_LIB_CHACHA=m
config/s390x/default:CONFIG_CRYPTO_LIB_CHACHA20POLY1305=m
config/x86_64/default:CONFIG_CRYPTO_CHACHA20=m
config/x86_64/default:CONFIG_CRYPTO_CHACHA20POLY1305=m
config/x86_64/default:CONFIG_CRYPTO_CHACHA20_X86_64=m
config/x86_64/default:CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
config/x86_64/default:CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
config/x86_64/default:CONFIG_CRYPTO_LIB_CHACHA=m
config/x86_64/default:CONFIG_CRYPTO_LIB_CHACHA20POLY1305=m

Comment 4 Petr Vorel 2024-01-18 19:26:06 UTC

@Michal FYI loading chacha_generic before the test did not help.

So is it still a bug with upstream kernel autoload mechanism for some chacha module (or other)?

Comment 5 Petr Vorel 2024-03-07 13:07:50 UTC

Michal, if I understood correctly wireguard should be working on power8 (or anything older than power10), but autoload mechanism is broken.

Do you plan to work on this?

Comment 6 Michal Suchanek 2024-03-15 12:26:37 UTC

Does not work with Linux 6.8.0 either, reported upstream.

Comment 7 Michal Suchanek 2024-03-21 13:47:42 UTC

It's broken since the p10 chacha driver was added, the fallback mechanism is broken.

Workaround patch should get into Tumbleweed soon.

There is ongoing upstream discussion about the correct solution.

Comment 8 Petr Vorel 2024-04-04 14:46:56 UTC

Yes, works in build 20240403, thanks!