Bugzilla – Bug 1190561
OOPS / Invalid RIP in when attaching XDP program to veth interface
Last modified: 2025-11-07 10:21:36 UTC
Attaching an xdp program to a veth interface may lead to a kernel OOPS in openSUSE Leap 15.3 (and most probably in SLES 15.3, too). To reproduce the crash, run the following command in multiple parallel loops: docker run --rm --privileged -v /tmp/bpf:/tmp/bpf builder15-3 ip link set eth0 xdp obj /tmp/bpf/kern.o sec xdp On our development machine (72 cores), I managed to get the crash within 1 to 20 minutes when starting 16 parallel loops. /tmp/bpf/kern.o can be a trivial xdp program (e.g. "return XDP_PASS") # uname -a Linux uranos 5.3.18-59.19-default #1 SMP Tue Aug 3 14:11:23 UTC 2021 (055c4fd) x86_64 x86_64 x86_64 GNU/Linux I reproduced this behavior multiple times, the lower part of the backtrace always differs, but it always ends up in "veth_poll+0x337". I'll attach different dmesg.txt files. Two of them contain an unsupported module (mxm_wmi), but I disabled it as soon as I noticed, so it has nothing to do with the crash.
Created attachment 852562 [details] dmesg.txt (1) This is the dmesg which occurred on my own application. All others were generated with the "docker ip link set" command.
Created attachment 852563 [details] dmesg.txt (2)
Created attachment 852564 [details] dmesg.txt (3)
Created attachment 852565 [details] dmesg.txt (4)
Created attachment 852566 [details] dmesg.txt (5)
Created attachment 852569 [details] dmesg.txt (macvlan) The problem is not limited to veth, it also occurs on macvlan interfaces.
Looks like the issue with net core stuff. Adding relevant people to Cc.
Created attachment 852615 [details] partial output from crash "foreach bt" Attached a cropped output from the "foreach bt" command from "crash", showing the backtrace of the crashing veth_poll+0x337 (I suppose this is veth.c:580, bpf_prog_run_xdp) and the backtrace of the installation of the xdp program. It seems that the call to bpf_prog_run_xdp finally leads to an half-patched bpf_dispatcher_xdp_func: crash> dis bpf_dispatcher_xdp_func 0xffffffff99594790 <bpf_dispatcher_xdp_func>: int3 0xffffffff99594791 <bpf_dispatcher_xdp_func+1>: imul $0xffffffc6,0x76e92729(%rax),%ebx 0xffffffff99594798 <bpf_dispatcher_xdp_func+8>: rex.RX add %r12b,0xf(%rsi) 0xffffffff9959479c <bpf_dispatcher_xdp_func+12>: (bad) 0xffffffff9959479d <bpf_dispatcher_xdp_func+13>: add %r8b,(%rax) Not 100% sure about the following, but: crash> p bp_patching bp_patching = $2 = { vec = 0xffffb0d965ff7488, nr_entries = 1 } crash> p bp_patching.vec[0] $4 = { detour = 0x0, addr = 0xffffffff99594790, len = 5, opcode = "\351k\240)'" } in combination with the line regs->ip = (unsigned long) tp->detour; from the poke_int3_handler could be the problem (note that the text_poke_bp call from __bpf_arch_text_poke supplies "NULL" as detour handler). I re-tested with a recent Tumbleweed kernel (5.13.13-1-default) on a desktop machine with 8 cores (Intel Xeon W-2123) and couldn't reproduce the problem there. The text_poke_bp_patch / poke_int3_handler combination is completely different here and seems to be safe.
Reverting the commit 28924f7b0963da7e4ce098baa067c3d36dbf49ff could solve the problem: bpf: Simplify __bpf_arch_text_poke poke type handling (bsc#1154353). suse-commit: e9229505be158934fd6d02eb5003ad7c3da74663
It's a blind shot but perhaps the problem may be missing backport of c3d6324f841b ("x86/alternatives: Teach text_poke_bp() to emulate instructions") while we have some later backports from times when there was no detour member in struct text_poke_loc, e.g. the commit from comment 9 or backport of commit 285a54efe386 ("x86/alternatives: Sync bp_patching update for avoiding NULL pointer exception"). Boris, what do you think?
Possible. This here: https://lwn.net/Articles/804937/ talks about taking this patch into a bpf set in order to fix a race: "The patch is necessary to do s/text_poke/text_poke_bp/ in patch 3 to fix the race." And this here looks like a race ... HTH.
If you could supply an RPM with the c3d6324f841b ("x86/alternatives: Teach text_poke_bp() to emulate instructions") patch applied, I'd confirm if this fixes the problem. It should cherry-pick just fine. Without that patch, not supplying a valid detour (which currently only happens in __bpf_arch_text_poke) will definitely lead to a crash if another CPU runs into the int3 handler. With that patch, not supplying a detour to text_poke_bp will be the default (that is, if the patched instruction is trivial, e.g. int3, call, jmp).
Ok, try this here: https://beta.suse.com/private/bpetkov/1190561/ You might need new suse-module-tools package which is there too. HTH.
5.3.18-1 looks older than the official kernel, which was 5.3.18-59.19. Was that intentional? Anyway, the new kernel version is now loading/unloading xdp programs for more than one hour. With the last kernel version, the crash occurred after approx. 20mins, so this looks pretty good. Nevertheless, I'll leave it running through the weekend. Thanks for your effort.
(In reply to Mark Rohrbacher from comment #14) > 5.3.18-1 looks older than the official kernel, which was 5.3.18-59.19. Was > that intentional? Bah, that's just wrong numbering. I'm sure there's a reason why those build service kernels don't get numbered with the official releases numbers. > Anyway, the new kernel version is now loading/unloading xdp programs for > more than one hour. With the last kernel version, the crash occurred after > approx. 20mins, so this looks pretty good. > Nevertheless, I'll leave it running through the weekend. Sounds good - lemme know if it has survived and I can queue it next week. Thx.
The stress test is now running for more than 48 hrs without any crashes, so the problem seems to be solved with your kernel. As this bug also affects the latest SLES15 SP3 kernel, can I expect a patch for that, too?
(In reply to Mark Rohrbacher from comment #16) > The stress test is now running for more than 48 hrs without any crashes, so > the problem seems to be solved with your kernel. Cool, thanks for testing! > As this bug also affects the latest SLES15 SP3 kernel, can I expect a patch > for that, too? Yeah, Leap and SLES use the same kernel. Fix will land in the next maintenance update. Pushing out and closing. Thx.
.
SUSE-SU-2021:3338-1: An update that solves 6 vulnerabilities and has 54 fixes is now available. Category: security (important) Bug References: 1065729,1148868,1152489,1154353,1159886,1167773,1170774,1171688,1173746,1174003,1176447,1176940,1177028,1178134,1184439,1184804,1185302,1185550,1185677,1185726,1185762,1187211,1188067,1188418,1188651,1188986,1189257,1189297,1189841,1189884,1190023,1190062,1190115,1190138,1190159,1190358,1190406,1190432,1190467,1190523,1190534,1190543,1190544,1190561,1190576,1190595,1190596,1190598,1190620,1190626,1190679,1190705,1190717,1190746,1190758,1190784,1190785,1191172,1191193,1191292 CVE References: CVE-2020-3702,CVE-2021-3669,CVE-2021-3744,CVE-2021-3752,CVE-2021-3764,CVE-2021-40490 JIRA References: Sources used: SUSE Linux Enterprise Module for Public Cloud 15-SP3 (src): kernel-azure-5.3.18-38.25.2, kernel-source-azure-5.3.18-38.25.2, kernel-syms-azure-5.3.18-38.25.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
openSUSE-SU-2021:3338-1: An update that solves 6 vulnerabilities and has 54 fixes is now available. Category: security (important) Bug References: 1065729,1148868,1152489,1154353,1159886,1167773,1170774,1171688,1173746,1174003,1176447,1176940,1177028,1178134,1184439,1184804,1185302,1185550,1185677,1185726,1185762,1187211,1188067,1188418,1188651,1188986,1189257,1189297,1189841,1189884,1190023,1190062,1190115,1190138,1190159,1190358,1190406,1190432,1190467,1190523,1190534,1190543,1190544,1190561,1190576,1190595,1190596,1190598,1190620,1190626,1190679,1190705,1190717,1190746,1190758,1190784,1190785,1191172,1191193,1191292 CVE References: CVE-2020-3702,CVE-2021-3669,CVE-2021-3744,CVE-2021-3752,CVE-2021-3764,CVE-2021-40490 JIRA References: Sources used: openSUSE Leap 15.3 (src): kernel-azure-5.3.18-38.25.2, kernel-source-azure-5.3.18-38.25.2, kernel-syms-azure-5.3.18-38.25.1
openSUSE-SU-2021:3387-1: An update that solves 7 vulnerabilities and has 53 fixes is now available. Category: security (important) Bug References: 1065729,1148868,1152489,1154353,1159886,1167773,1170774,1171688,1173746,1174003,1176447,1176940,1177028,1178134,1184439,1184804,1185302,1185550,1185677,1185726,1185762,1187211,1188067,1188418,1188651,1188986,1189257,1189297,1189841,1189884,1190023,1190062,1190115,1190138,1190159,1190358,1190406,1190432,1190467,1190523,1190534,1190543,1190544,1190561,1190576,1190595,1190596,1190598,1190620,1190626,1190679,1190705,1190717,1190746,1190758,1190784,1190785,1191172,1191193,1191292 CVE References: CVE-2020-3702,CVE-2021-3669,CVE-2021-3744,CVE-2021-3752,CVE-2021-3759,CVE-2021-3764,CVE-2021-40490 JIRA References: Sources used: openSUSE Leap 15.3 (src): dtb-aarch64-5.3.18-59.27.1, kernel-64kb-5.3.18-59.27.1, kernel-debug-5.3.18-59.27.1, kernel-default-5.3.18-59.27.1, kernel-default-base-5.3.18-59.27.1.18.15.1, kernel-docs-5.3.18-59.27.1, kernel-kvmsmall-5.3.18-59.27.1, kernel-obs-build-5.3.18-59.27.1, kernel-obs-qa-5.3.18-59.27.1, kernel-preempt-5.3.18-59.27.1, kernel-source-5.3.18-59.27.1, kernel-syms-5.3.18-59.27.1, kernel-zfcpdump-5.3.18-59.27.1
SUSE-SU-2021:3387-1: An update that solves 7 vulnerabilities and has 53 fixes is now available. Category: security (important) Bug References: 1065729,1148868,1152489,1154353,1159886,1167773,1170774,1171688,1173746,1174003,1176447,1176940,1177028,1178134,1184439,1184804,1185302,1185550,1185677,1185726,1185762,1187211,1188067,1188418,1188651,1188986,1189257,1189297,1189841,1189884,1190023,1190062,1190115,1190138,1190159,1190358,1190406,1190432,1190467,1190523,1190534,1190543,1190544,1190561,1190576,1190595,1190596,1190598,1190620,1190626,1190679,1190705,1190717,1190746,1190758,1190784,1190785,1191172,1191193,1191292 CVE References: CVE-2020-3702,CVE-2021-3669,CVE-2021-3744,CVE-2021-3752,CVE-2021-3759,CVE-2021-3764,CVE-2021-40490 JIRA References: Sources used: SUSE MicroOS 5.1 (src): kernel-default-5.3.18-59.27.1, kernel-default-base-5.3.18-59.27.1.18.15.1 SUSE Linux Enterprise Workstation Extension 15-SP3 (src): kernel-default-5.3.18-59.27.1, kernel-preempt-5.3.18-59.27.1 SUSE Linux Enterprise Module for Live Patching 15-SP3 (src): kernel-default-5.3.18-59.27.1, kernel-livepatch-SLE15-SP3_Update_7-1-7.3.1 SUSE Linux Enterprise Module for Legacy Software 15-SP3 (src): kernel-default-5.3.18-59.27.1 SUSE Linux Enterprise Module for Development Tools 15-SP3 (src): kernel-docs-5.3.18-59.27.1, kernel-obs-build-5.3.18-59.27.1, kernel-preempt-5.3.18-59.27.1, kernel-source-5.3.18-59.27.1, kernel-syms-5.3.18-59.27.1 SUSE Linux Enterprise Module for Basesystem 15-SP3 (src): kernel-64kb-5.3.18-59.27.1, kernel-default-5.3.18-59.27.1, kernel-default-base-5.3.18-59.27.1.18.15.1, kernel-preempt-5.3.18-59.27.1, kernel-source-5.3.18-59.27.1, kernel-zfcpdump-5.3.18-59.27.1 SUSE Linux Enterprise High Availability 15-SP3 (src): kernel-default-5.3.18-59.27.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2021:3415-1: An update that solves 18 vulnerabilities and has 119 fixes is now available. Category: security (important) Bug References: 1065729,1124431,1127650,1135481,1148868,1152489,1154353,1159886,1167032,1167773,1168202,1170774,1171420,1171688,1173746,1174003,1175543,1176447,1176940,1177028,1177399,1178134,1180141,1180347,1181006,1181972,1184114,1184439,1184611,1184804,1185302,1185550,1185675,1185677,1185726,1185762,1185898,1187211,1187455,1187591,1187619,1188067,1188172,1188270,1188412,1188418,1188439,1188616,1188651,1188694,1188700,1188878,1188924,1188983,1188985,1188986,1189153,1189225,1189257,1189262,1189297,1189301,1189399,1189400,1189503,1189504,1189505,1189506,1189507,1189562,1189563,1189564,1189565,1189566,1189567,1189568,1189569,1189573,1189574,1189575,1189576,1189577,1189579,1189581,1189582,1189583,1189585,1189586,1189587,1189696,1189706,1189760,1189762,1189832,1189841,1189870,1189872,1189883,1189884,1190022,1190023,1190025,1190062,1190115,1190117,1190131,1190138,1190159,1190181,1190358,1190406,1190412,1190413,1190428,1190467,1190523,1190534,1190543,1190544,1190561,1190576,1190595,1190596,1190598,1190620,1190626,1190679,1190705,1190717,1190746,1190758,1190784,1190785,1191172,1191193,1191292,859220 CVE References: CVE-2020-12770,CVE-2020-3702,CVE-2021-34556,CVE-2021-35477,CVE-2021-3653,CVE-2021-3656,CVE-2021-3669,CVE-2021-3732,CVE-2021-3739,CVE-2021-3743,CVE-2021-3744,CVE-2021-3752,CVE-2021-3753,CVE-2021-3759,CVE-2021-3764,CVE-2021-38160,CVE-2021-38198,CVE-2021-40490 JIRA References: Sources used: SUSE MicroOS 5.1 (src): kernel-rt-5.3.18-57.1 SUSE Linux Enterprise Module for Realtime 15-SP3 (src): kernel-rt-5.3.18-57.1, kernel-rt_debug-5.3.18-57.1, kernel-source-rt-5.3.18-57.1, kernel-syms-rt-5.3.18-57.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.