Bugzilla – Bug 1214969
r8169: transmit queue 0 timed out
Last modified: 2023-09-19 10:50:31 UTC
I hit a possible regression in kernel 6.4.x, it seems to me that this is a very similar (or same) issue like bsc#1106833 and similar. After some time (depending on the network usage, usually within 24 hours) the kernel panic happens and the network interface becomes unusable (in fact it probably generates some nonsense on the line as it also takes down the whole network segment). This behavior can be observed in 6.4.3-1-default, 6.4.6-1-default, 6.4.9-1-default and now in 6.4.11-1-default. The issue is not happening in 6.3.9-1-default which I am using right now as a workaround. --- zář 04 21:39:12 valiant kernel: ------------[ cut here ]------------ zář 04 21:39:12 valiant kernel: NETDEV WATCHDOG: enp2s0 (r8169): transmit queue 0 timed out 8160 ms zář 04 21:39:12 valiant kernel: WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x21e/0x230 zář 04 21:39:12 valiant kernel: Modules linked in: tun rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device af_packet vboxnetadp(O) vboxnetflt(O) vboxdrv(O) cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_hda_codec_realtek snd_sof_utils snd_hda_codec_generic mt7921e snd_hda_codec_hdmi snd_soc_core mt7921_common mt76_connac_lib snd_compress intel_rapl_msr btusb mt76 snd_hda_intel snd_pcm_dmaengine intel_rapl_common btrtl snd_intel_dspcfg btbcm snd_intel_sdw_acpi snd_pci_ps btintel edac_mce_amd snd_hda_codec btmtk asus_nb_wmi eeepc_wmi mac80211 asus_wmi snd_rpl_pci_acp6x r8169 snd_hda_core snd_acp_pci battery snd_pci_acp6x kvm_amd snd_hwdep ledtrig_audio bluetooth snd_pcm libarc4 snd_pci_acp5x sparse_keymap realtek kvm joydev ecdh_generic platform_profile snd_rn_pci_acp3x snd_timer snd_acp_config mdio_devres irqbypass wmi_bmof cfg80211 pcspkr snd acpi_cpufreq snd_soc_acpi tiny_power_button efi_pstore soundcore zář 04 21:39:12 valiant kernel: k10temp libphy i2c_piix4 snd_pci_acp3x rfkill serial_multi_instantiate button fuse configfs dmi_sysfs ip_tables x_tables hid_generic usbhid dm_crypt essiv authenc trusted asn1_encoder tee amdgpu crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 i2c_algo_bit drm_ttm_helper ttm drm_suballoc_helper iommu_v2 drm_buddy xhci_pci xhci_pci_renesas gpu_sched xhci_hcd aesni_intel drm_display_helper nvme crypto_simd ucsi_acpi usbcore cryptd typec_ucsi cec nvme_core roles ccp rc_core sp5100_tco typec video wmi btrfs blake2b_generic libcrc32c crc32c_intel xor raid6_pq sg br_netfilter bridge stp llc dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efivarfs zář 04 21:39:12 valiant kernel: CPU: 11 PID: 0 Comm: swapper/11 Tainted: G O 6.4.11-1-default #1 openSUSE Tumbleweed b71e9da2e7bdf20ad15c36d48af8b3d0ec2bfcc0 zář 04 21:39:12 valiant kernel: Hardware name: ASUSTeK COMPUTER INC. MINIPC PN52/PN52, BIOS 11800 09/07/2022 zář 04 21:39:12 valiant kernel: RIP: 0010:dev_watchdog+0x21e/0x230 zář 04 21:39:12 valiant kernel: Code: ff ff ff 48 89 df c6 05 ba 31 5d 01 01 e8 fa 38 fa ff 45 89 f8 44 89 f1 48 89 de 48 89 c2 48 c7 c7 70 16 cb a6 e8 12 d9 5e ff <0f> 0b e9 2d ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 zář 04 21:39:12 valiant kernel: RSP: 0018:ffffb3a180438ea0 EFLAGS: 00010282 zář 04 21:39:12 valiant kernel: RAX: 0000000000000000 RBX: ffff942ec3f90000 RCX: 0000000000000000 zář 04 21:39:12 valiant kernel: RDX: ffff943af5db4528 RSI: ffff943af5da74c0 RDI: 0000000000000300 zář 04 21:39:12 valiant kernel: RBP: ffff942ec3f904c8 R08: 0000000000000000 R09: ffffb3a180438d48 zář 04 21:39:12 valiant kernel: R10: 0000000000000003 R11: ffff943b2e0fcfe8 R12: ffff942e08ca6000 zář 04 21:39:12 valiant kernel: R13: ffff942ec3f9041c R14: 0000000000000000 R15: 0000000000001fe0 zář 04 21:39:12 valiant kernel: FS: 0000000000000000(0000) GS:ffff943af5d80000(0000) knlGS:0000000000000000 zář 04 21:39:12 valiant kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 zář 04 21:39:12 valiant kernel: CR2: 00007f1ff7fec980 CR3: 00000000bbc36000 CR4: 0000000000750ee0 zář 04 21:39:12 valiant kernel: PKRU: 55555554 zář 04 21:39:12 valiant kernel: Call Trace: zář 04 21:39:12 valiant kernel: <IRQ> zář 04 21:39:12 valiant kernel: ? dev_watchdog+0x21e/0x230 zář 04 21:39:12 valiant kernel: ? __warn+0x81/0x130 zář 04 21:39:12 valiant kernel: ? dev_watchdog+0x21e/0x230 zář 04 21:39:12 valiant kernel: ? report_bug+0x171/0x1a0 zář 04 21:39:12 valiant kernel: ? handle_bug+0x3c/0x80 zář 04 21:39:12 valiant kernel: ? exc_invalid_op+0x17/0x70 zář 04 21:39:12 valiant kernel: ? asm_exc_invalid_op+0x1a/0x20 zář 04 21:39:12 valiant kernel: ? dev_watchdog+0x21e/0x230 zář 04 21:39:12 valiant kernel: ? __pfx_dev_watchdog+0x10/0x10 zář 04 21:39:12 valiant kernel: ? __pfx_dev_watchdog+0x10/0x10 zář 04 21:39:12 valiant kernel: call_timer_fn+0x27/0x130 zář 04 21:39:12 valiant kernel: __run_timers.part.0+0x1d8/0x280 zář 04 21:39:12 valiant kernel: ? __x86_return_thunk+0x9/0x10 zář 04 21:39:12 valiant kernel: ? __x86_return_thunk+0x9/0x10 zář 04 21:39:12 valiant kernel: ? ktime_get+0x3c/0xa0 zář 04 21:39:12 valiant kernel: ? __x86_return_thunk+0x9/0x10 zář 04 21:39:12 valiant kernel: run_timer_softirq+0x2a/0x50 zář 04 21:39:12 valiant kernel: __do_softirq+0xca/0x2a5 zář 04 21:39:12 valiant kernel: __irq_exit_rcu+0xae/0xe0 zář 04 21:39:12 valiant kernel: sysvec_apic_timer_interrupt+0x72/0x90 zář 04 21:39:12 valiant kernel: </IRQ> zář 04 21:39:12 valiant kernel: <TASK> zář 04 21:39:12 valiant kernel: asm_sysvec_apic_timer_interrupt+0x1a/0x20 zář 04 21:39:12 valiant kernel: RIP: 0010:cpuidle_enter_state+0xcc/0x440 zář 04 21:39:12 valiant kernel: Code: 2a ea 47 ff e8 d5 f1 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 b3 f6 46 ff 45 84 ff 0f 85 56 02 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d zář 04 21:39:12 valiant kernel: RSP: 0018:ffffb3a1801d7e90 EFLAGS: 00000246 zář 04 21:39:12 valiant kernel: RAX: ffff943af5dba080 RBX: ffff942e08d7bc00 RCX: 000000000000045a zář 04 21:39:12 valiant kernel: RDX: 000000000000000b RSI: 000011088cb991b0 RDI: 0000000000000000 zář 04 21:39:12 valiant kernel: RBP: 0000000000000003 R08: ffffffffffcd2371 R09: 000000002690b399 zář 04 21:39:12 valiant kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa7632be0 zář 04 21:39:12 valiant kernel: R13: 000011088cb991b0 R14: 0000000000000003 R15: 0000000000000000 zář 04 21:39:12 valiant kernel: cpuidle_enter+0x2d/0x40 zář 04 21:39:12 valiant kernel: do_idle+0x20d/0x270 zář 04 21:39:12 valiant kernel: cpu_startup_entry+0x1d/0x20 zář 04 21:39:12 valiant kernel: start_secondary+0x12e/0x150 zář 04 21:39:12 valiant kernel: secondary_startup_64_no_verify+0x10b/0x10b zář 04 21:39:12 valiant kernel: </TASK> zář 04 21:39:12 valiant kernel: ---[ end trace 0000000000000000 ]--- --- 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) Subsystem: ASUSTeK Computer Inc. Device 87d7 Flags: bus master, fast devsel, latency 0, IRQ 36, IOMMU group 12 I/O ports at f000 [size=256] Memory at fce00000 (64-bit, non-prefetchable) [size=64K] Memory at fce10000 (64-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] Express Endpoint, MSI 01 Capabilities: [b0] MSI-X: Enable+ Count=32 Masked- Capabilities: [d0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [168] Device Serial Number 01-00-00-00-68-4c-e0-00 Capabilities: [178] Transaction Processing Hints Capabilities: [204] Latency Tolerance Reporting Capabilities: [20c] L1 PM Substates Capabilities: [21c] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Kernel driver in use: r8169 Kernel modules: r8169
Does it persist with 6.5.x kernel?
I am running 6.5.1-1.gc181e32-default now, it is already 28 hours and so far so good. I was not able to reproduce the issue yet. I do not want to jinx it, but it seems that the issue is really only for 6.4.x. I will try to keep it running for some more days to see and update accordingly.
It is over 5 days now, still no crash. Now I believe the issue does *not* persist on 6.5.1. The solution is to wait for the kernel to land in TW then.
My enjoyment was a little premature it seems, or there is a regression 6.5.2. After 6 days of uptime with 6.5.2-1-default --- zář 19 12:03:56 valiant kernel: ------------[ cut here ]------------ zář 19 12:03:56 valiant kernel: NETDEV WATCHDOG: enp2s0 (r8169): transmit queue 0 timed out 7264 ms zář 19 12:03:56 valiant kernel: WARNING: CPU: 10 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x21e/0x230 zář 19 12:03:56 valiant kernel: Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink binfmt_misc ftdi_sio usbserial ext4 mbcache jbd2 raid1 md_mod uas usb_storage tun rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache netfs uinput rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device joydev af_packet vboxnetadp(O) vboxnetflt(O) vboxdrv(O) cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof mt7921e snd_hda_codec_realtek mt7921_common snd_sof_utils snd_hda_codec_generic snd_hda_codec_hdmi mt76_connac_lib snd_hda_intel snd_soc_core intel_rapl_msr btusb mt76 snd_intel_dspcfg snd_intel_sdw_acpi intel_rapl_common btrtl btbcm snd_compress btintel edac_mce_amd snd_hda_codec asus_nb_wmi eeepc_wmi snd_pcm_dmaengine btmtk mac80211 snd_hda_core asus_wmi snd_pci_ps kvm_amd bluetooth snd_rpl_pci_acp6x snd_acp_pci libarc4 battery snd_hwdep snd_pci_acp6x ecdh_generic kvm ledtrig_audio sparse_keymap zář 19 12:03:56 valiant kernel: snd_pci_acp5x cfg80211 irqbypass snd_pcm r8169 platform_profile wmi_bmof snd_timer realtek snd_rn_pci_acp3x snd mdio_devres pcspkr efi_pstore snd_acp_config i2c_piix4 k10temp snd_soc_acpi libphy snd_pci_acp3x rfkill soundcore serial_multi_instantiate tiny_power_button button fuse configfs dmi_sysfs ip_tables x_tables hid_generic usbhid dm_crypt essiv authenc trusted asn1_encoder tee amdgpu crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 i2c_algo_bit drm_ttm_helper ttm xhci_pci xhci_pci_renesas drm_suballoc_helper amdxcp iommu_v2 drm_buddy xhci_hcd gpu_sched aesni_intel crypto_simd drm_display_helper cryptd nvme ucsi_acpi usbcore cec typec_ucsi nvme_core ccp roles rc_core sp5100_tco typec video wmi btrfs blake2b_generic libcrc32c crc32c_intel xor raid6_pq sg br_netfilter bridge stp llc dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efivarfs zář 19 12:03:56 valiant kernel: CPU: 10 PID: 0 Comm: swapper/10 Tainted: G O 6.5.2-1-default #1 openSUSE Tumbleweed 0b829be613d5ef740105318e5d9ba7d410158922 zář 19 12:03:56 valiant kernel: Hardware name: ASUSTeK COMPUTER INC. MINIPC PN52/PN52, BIOS 11800 09/07/2022 zář 19 12:03:56 valiant kernel: RIP: 0010:dev_watchdog+0x21e/0x230 zář 19 12:03:56 valiant kernel: Code: ff ff ff 48 89 df c6 05 af fe 5b 01 01 e8 fa 2e fa ff 45 89 f8 44 89 f1 48 89 de 48 89 c2 48 c7 c7 b8 1e cc 96 e8 42 9c 5d ff <0f> 0b e9 2d ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 zář 19 12:03:56 valiant kernel: RSP: 0018:ffffb58680300ea0 EFLAGS: 00010286 zář 19 12:03:56 valiant kernel: RAX: 0000000000000000 RBX: ffff9838078e4000 RCX: 0000000000000027 zář 19 12:03:56 valiant kernel: RDX: ffff9844f5f274c8 RSI: 0000000000000001 RDI: ffff9844f5f274c0 zář 19 12:03:56 valiant kernel: RBP: ffff9838078e44c8 R08: 0000000000000000 R09: ffffb58680300d48 zář 19 12:03:56 valiant kernel: R10: 0000000000000003 R11: ffff9844f58fffe8 R12: ffff983802843e00 zář 19 12:03:56 valiant kernel: R13: ffff9838078e441c R14: 0000000000000000 R15: 0000000000001c60 zář 19 12:03:56 valiant kernel: FS: 0000000000000000(0000) GS:ffff9844f5f00000(0000) knlGS:0000000000000000 zář 19 12:03:56 valiant kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 zář 19 12:03:56 valiant kernel: CR2: 000055dca5d95608 CR3: 000000026dce6000 CR4: 0000000000750ee0 zář 19 12:03:56 valiant kernel: PKRU: 55555554 zář 19 12:03:56 valiant kernel: Call Trace: zář 19 12:03:56 valiant kernel: <IRQ> zář 19 12:03:56 valiant kernel: ? dev_watchdog+0x21e/0x230 zář 19 12:03:56 valiant kernel: ? __warn+0x81/0x130 zář 19 12:03:56 valiant kernel: ? dev_watchdog+0x21e/0x230 zář 19 12:03:56 valiant kernel: ? report_bug+0x171/0x1a0 zář 19 12:03:56 valiant kernel: ? handle_bug+0x3c/0x80 zář 19 12:03:56 valiant kernel: ? exc_invalid_op+0x17/0x70 zář 19 12:03:56 valiant kernel: ? asm_exc_invalid_op+0x1a/0x20 zář 19 12:03:56 valiant kernel: ? dev_watchdog+0x21e/0x230 zář 19 12:03:56 valiant kernel: ? __pfx_dev_watchdog+0x10/0x10 zář 19 12:03:56 valiant kernel: ? __pfx_dev_watchdog+0x10/0x10 zář 19 12:03:56 valiant kernel: call_timer_fn+0x27/0x130 zář 19 12:03:56 valiant kernel: __run_timers.part.0+0x1d8/0x280 zář 19 12:03:56 valiant kernel: ? enqueue_hrtimer+0x2f/0x80 zář 19 12:03:56 valiant kernel: ? srso_alias_return_thunk+0x5/0x7f zář 19 12:03:56 valiant kernel: ? __hrtimer_run_queues+0x141/0x2b0 zář 19 12:03:56 valiant kernel: ? srso_alias_return_thunk+0x5/0x7f zář 19 12:03:56 valiant kernel: ? ktime_get+0x3c/0xa0 zář 19 12:03:56 valiant kernel: ? srso_alias_return_thunk+0x5/0x7f zář 19 12:03:56 valiant kernel: run_timer_softirq+0x2a/0x50 zář 19 12:03:56 valiant kernel: __do_softirq+0xca/0x2a5 zář 19 12:03:56 valiant kernel: __irq_exit_rcu+0x96/0xb0 zář 19 12:03:56 valiant kernel: sysvec_apic_timer_interrupt+0x72/0x90 zář 19 12:03:56 valiant kernel: </IRQ> zář 19 12:03:56 valiant kernel: <TASK> zář 19 12:03:56 valiant kernel: asm_sysvec_apic_timer_interrupt+0x1a/0x20 zář 19 12:03:56 valiant kernel: RIP: 0010:cpuidle_enter_state+0xcc/0x440 zář 19 12:03:56 valiant kernel: Code: 6a d4 46 ff e8 e5 f1 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 73 e2 45 ff 45 84 ff 0f 85 56 02 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d zář 19 12:03:56 valiant kernel: RSP: 0018:ffffb586801cfe90 EFLAGS: 00000246 zář 19 12:03:56 valiant kernel: RAX: ffff9844f5f3a080 RBX: ffff983808fc7000 RCX: 0000000000000064 zář 19 12:03:56 valiant kernel: RDX: 000000000000000a RSI: 0002049371adc9d9 RDI: 0000000000000000 zář 19 12:03:56 valiant kernel: RBP: 0000000000000002 R08: ffffffffffcd235c R09: 0000000000000018 zář 19 12:03:56 valiant kernel: R10: ffff9844f5f38a84 R11: 000000000000068a R12: ffffffff97633860 zář 19 12:03:56 valiant kernel: R13: 0002049371adc9d9 R14: 0000000000000002 R15: 0000000000000000 zář 19 12:03:56 valiant kernel: cpuidle_enter+0x2d/0x40 zář 19 12:03:56 valiant kernel: do_idle+0x20d/0x270 zář 19 12:03:56 valiant kernel: cpu_startup_entry+0x1d/0x20 zář 19 12:03:56 valiant kernel: start_secondary+0x11e/0x140 zář 19 12:03:56 valiant kernel: secondary_startup_64_no_verify+0x17e/0x18b zář 19 12:03:56 valiant kernel: </TASK> zář 19 12:03:56 valiant kernel: ---[ end trace 0000000000000000 ]---
Then better report it to the upstream devs. At best, netdev ML. You can open an entry at bugzilla.kernel.org in addition, although it catches less attention in general.