Bug 1212483

Summary: NetworkManager crashes frequently
Product: [openSUSE] openSUSE Distribution Reporter: Andreas Schwab <schwab>
Component: NetworkAssignee: Jonathan Kang <songchuan.kang>
Status: NEW --- QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None    
Version: Leap 15.5   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: /var/log/NetworkManager

Description Andreas Schwab 2023-06-18 09:56:46 UTC 
Core was generated by `/usr/sbin/NetworkManager --no-daemon'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000559e95354a35 in ipv6_dad_timeout (user_data=0x559e95a730f0)
    at ../src/core/dhcp/nm-dhcp-client.c:606
606         nm_clear_g_source_inst(&priv->v6.dad_timeout_source);
(gdb) bt
#0  0x0000559e95354a35 in ipv6_dad_timeout (
    user_data=user_data@entry=0x559e95a730f0)
    at ../src/core/dhcp/nm-dhcp-client.c:606
#1  0x00007fb27117337d in g_timeout_dispatch (source=0x559e95b028a0, 
    callback=0x559e95354a10 <ipv6_dad_timeout>, user_data=0x559e95a730f0)
    at ../glib/gmain.c:4933
#2  0x00007fb27117282b in g_main_dispatch (context=0x559e959bd8f0)
    at ../glib/gmain.c:3381
#3  g_main_context_dispatch (context=context@entry=0x559e959bd8f0)
    at ../glib/gmain.c:4099
#4  0x00007fb271172bd0 in g_main_context_iterate (context=0x559e959bd8f0, 
    block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at ../glib/gmain.c:4175
#5  0x00007fb271172ea2 in g_main_loop_run (loop=0x559e959bd9e0)
    at ../glib/gmain.c:4373
#6  0x0000559e9534a9a1 in main (argc=<optimized out>, argv=<optimized out>)
    at ../src/core/main.c:509
(gdb) p priv
$1 = (NMDhcpClientPrivate *) 0x0
Comment 1 Andreas Schwab 2023-07-02 06:47:25 UTC 
(gdb) bt
#0  0x0000563f3c8b4a35 in ipv6_dad_timeout (
    user_data=user_data@entry=0x563f3db6f240)
    at ../src/core/dhcp/nm-dhcp-client.c:606
#1  0x00007fd88889e37d in g_timeout_dispatch (source=0x563f3db414e0, 
    callback=0x563f3c8b4a10 <ipv6_dad_timeout>, user_data=0x563f3db6f240)
    at ../glib/gmain.c:4933
#2  0x00007fd88889d82b in g_main_dispatch (context=0x563f3dac98f0)
    at ../glib/gmain.c:3381
#3  g_main_context_dispatch (context=context@entry=0x563f3dac98f0)
    at ../glib/gmain.c:4099
#4  0x00007fd88889dbd0 in g_main_context_iterate (context=0x563f3dac98f0, 
    block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at ../glib/gmain.c:4175
#5  0x00007fd88889dea2 in g_main_loop_run (loop=0x563f3dac99e0)
    at ../glib/gmain.c:4373
#6  0x0000563f3c8aa9a1 in main (argc=<optimized out>, argv=<optimized out>)
    at ../src/core/main.c:509
(gdb) p/x *self
$1 = {parent = {g_type_instance = {g_class = 0xaaaaaaaaaaaaaaaa}, 
    ref_count = 0xaaaaaaaa, qdata = 0xaaaaaaaaaaaaaaaa}, 
  _priv = 0xaaaaaaaaaaaaaaaa}
Comment 2 Jonathan Kang 2023-07-04 12:09:37 UTC 
Thanks for the report, Andreas.

A detailed NetworkManager logs would be helpful. To collect this, add the following to /etc/NetworkManager/NetworkManager.conf(if exists), restart NetworkManager, reproduce this issue and attach the output of "sudo journalctl -b -u NetworkManager" here.

> [logging]
> level=trace
Comment 3 Andreas Schwab 2023-07-05 05:35:39 UTC 
WTF?

(gdb) bt
#0  l3_cfg_notify_cb (l3cfg=0x564dd7a91400, notify_data=0x7ffe61e515d0, 
    self=0x564dd7a88220) at ../src/core/dhcp/nm-dhcp-client.c:774
#1  0x00007fd87c549973 in g_closure_invoke ()
   from /usr/lib64/libgobject-2.0.so.0
#2  0x00007fd87c55da7e in ?? () from /usr/lib64/libgobject-2.0.so.0
#3  0x00007fd87c567684 in g_signal_emit_valist ()
   from /usr/lib64/libgobject-2.0.so.0
#4  0x00007fd87c56811f in g_signal_emit () from /usr/lib64/libgobject-2.0.so.0
#5  0x0000564dd6917158 in _nm_l3cfg_emit_signal_notify (
    self=self@entry=0x564dd7a91400, 
    notify_data=notify_data@entry=0x7ffe61e515d0) at ../src/core/nm-l3cfg.c:609
#6  0x0000564dd6918e00 in _nm_l3cfg_notify_platform_change_on_idle (
    obj_type_flags=2, self=0x564dd7a91400) at ../src/core/nm-l3cfg.c:1332
#7  _platform_signal_on_idle_cb (user_data=user_data@entry=0x564dd79efb00)
    at ../src/core/nm-netns.c:208
#8  0x00007fd87c4514d7 in g_idle_dispatch (source=0x564dd7b00e50, 
    callback=0x564dd6918cd0 <_platform_signal_on_idle_cb>, 
    user_data=0x564dd79efb00) at ../glib/gmain.c:5897
#9  0x00007fd87c45582b in g_main_dispatch (context=0x564dd79c6b00)
    at ../glib/gmain.c:3381
#10 g_main_context_dispatch (context=context@entry=0x564dd79c6b00)
    at ../glib/gmain.c:4099
#11 0x00007fd87c455bd0 in g_main_context_iterate (context=0x564dd79c6b00, 
    block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at ../glib/gmain.c:4175
#12 0x00007fd87c455ea2 in g_main_loop_run (loop=0x564dd79c6bf0)
    at ../glib/gmain.c:4373
#13 0x0000564dd674c9a1 in main (argc=<optimized out>, argv=<optimized out>)
    at ../src/core/main.c:509
(gdb) l
769                                      }));
770                     }
771                 } else {
772                     _LOGD("decline the lease");
773                     if (!_dhcp_client_decline(self, priv->l3cd, "DAD failed", &error))
774                         _LOGD("decline failed: %s", error->message);
775                 }
776             }
777         }
778
(gdb) p error
$3 = (GError *) 0x0
Comment 4 Jonathan Kang 2023-07-12 01:08:49 UTC 
A test package is available in this repo. Can you try NetworkManager in that repo and see whether it fixes the issue?

* https://mirrorcache-us.opensuse.org/repositories/home:/JonathanKang:/branches:/SUSE:/SLE-15-SP5:/GA/standard/
Comment 5 Andreas Schwab 2023-07-16 08:47:19 UTC 
Why did you disable debuginfo?
Comment 6 Jonathan Kang 2023-07-17 07:22:11 UTC 
(In reply to Andreas Schwab from comment #5)
> Why did you disable debuginfo?

Just enabled it. obs defaults to disabling debuginfo.
Comment 7 Andreas Schwab 2023-07-29 07:46:29 UTC 
Created attachment 868496 [details]
/var/log/NetworkManager

(gdb) bt
#0  0x0000555b2e2e0a35 in ipv6_dad_timeout (
    user_data=user_data@entry=0x555b2f23ef20)
    at ../src/core/dhcp/nm-dhcp-client.c:606
#1  0x00007f51ca1b737d in g_timeout_dispatch (source=0x555b2f37d7f0, 
    callback=0x555b2e2e0a10 <ipv6_dad_timeout>, user_data=0x555b2f23ef20)
    at ../glib/gmain.c:4933
#2  0x00007f51ca1b682b in g_main_dispatch (context=0x555b2f21fb00)
    at ../glib/gmain.c:3381
#3  g_main_context_dispatch (context=context@entry=0x555b2f21fb00)
    at ../glib/gmain.c:4099
#4  0x00007f51ca1b6bd0 in g_main_context_iterate (context=0x555b2f21fb00, 
    block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at ../glib/gmain.c:4175
#5  0x00007f51ca1b6ea2 in g_main_loop_run (loop=0x555b2f21fbf0)
    at ../glib/gmain.c:4373
#6  0x0000555b2e2d69a1 in main (argc=<optimized out>, argv=<optimized out>)
    at ../src/core/main.c:509
(gdb) l
601     ipv6_dad_timeout(gpointer user_data)
602     {
603         NMDhcpClient        *self = user_data;
604         NMDhcpClientPrivate *priv = NM_DHCP_CLIENT_GET_PRIVATE(self);
605
606         nm_clear_g_source_inst(&priv->v6.dad_timeout_source);
607
608         _emit_notify(
609             self,
610             &((NMDhcpClientNotifyData){
(gdb) p priv
$1 = (NMDhcpClientPrivate *) 0x0
Comment 8 Jonathan Kang 2023-07-31 01:48:21 UTC 
(In reply to Andreas Schwab from comment #7)
> Created attachment 868496 [details]
> /var/log/NetworkManager
> 
> (gdb) bt
> #0  0x0000555b2e2e0a35 in ipv6_dad_timeout (
>     user_data=user_data@entry=0x555b2f23ef20)
>     at ../src/core/dhcp/nm-dhcp-client.c:606
> #1  0x00007f51ca1b737d in g_timeout_dispatch (source=0x555b2f37d7f0, 
>     callback=0x555b2e2e0a10 <ipv6_dad_timeout>, user_data=0x555b2f23ef20)
>     at ../glib/gmain.c:4933
> #2  0x00007f51ca1b682b in g_main_dispatch (context=0x555b2f21fb00)
>     at ../glib/gmain.c:3381
> #3  g_main_context_dispatch (context=context@entry=0x555b2f21fb00)
>     at ../glib/gmain.c:4099
> #4  0x00007f51ca1b6bd0 in g_main_context_iterate (context=0x555b2f21fb00, 
>     block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
>     at ../glib/gmain.c:4175
> #5  0x00007f51ca1b6ea2 in g_main_loop_run (loop=0x555b2f21fbf0)
>     at ../glib/gmain.c:4373
> #6  0x0000555b2e2d69a1 in main (argc=<optimized out>, argv=<optimized out>)
>     at ../src/core/main.c:509
> (gdb) l
> 601     ipv6_dad_timeout(gpointer user_data)
> 602     {
> 603         NMDhcpClient        *self = user_data;
> 604         NMDhcpClientPrivate *priv = NM_DHCP_CLIENT_GET_PRIVATE(self);
> 605
> 606         nm_clear_g_source_inst(&priv->v6.dad_timeout_source);
> 607
> 608         _emit_notify(
> 609             self,
> 610             &((NMDhcpClientNotifyData){
> (gdb) p priv
> $1 = (NMDhcpClientPrivate *) 0x0

What about `p self` when NetworkManager crashes? Is it also null?
Comment 9 Andreas Schwab 2023-10-22 07:35:30 UTC 
Of course not, otherwise NM_DHCP_CLIENT_GET_PRIVATE(self) would have crashed.
Comment 10 Andreas Schwab 2023-11-18 15:02:37 UTC 
Any news?  Still crashing and crashing.
Comment 11 Jonathan Kang 2023-11-22 06:44:40 UTC 
I was occupied by other things in the past weeks. I'll take a further look at this issue this week.

Can you provide another NetworkManager journal and note exactly when the crash happened? I didn't find much useful information in the previous one.

Thanks.
Comment 12 Andreas Schwab 2023-11-22 18:25:31 UTC 
The crash ends the log, of course.
Comment 13 Andreas Schwab 2023-11-26 08:50:57 UTC 
#0  g_source_destroy (source=0x373a37383a31393a) at ../glib/gmain.c:1386
#1  0x000056253edcda61 in nm_g_source_destroy_and_unref ()
    at ../src/libnm-glib-aux/nm-shared-utils.h:1699
#2  ipv6_dad_timeout (user_data=user_data@entry=0x562540a40df0)
    at ../src/core/dhcp/nm-dhcp-client.c:607
#3  0x00007f4140f9637d in g_timeout_dispatch (source=0x562540bead50, 
    callback=0x56253edcda20 <ipv6_dad_timeout>, user_data=0x562540a40df0)
    at ../glib/gmain.c:4933
#4  0x00007f4140f9582b in g_main_dispatch (context=0x562540a23890)
    at ../glib/gmain.c:3381
#5  g_main_context_dispatch (context=context@entry=0x562540a23890)
    at ../glib/gmain.c:4099
#6  0x00007f4140f95bd0 in g_main_context_iterate (context=0x562540a23890, 
    block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at ../glib/gmain.c:4175
#7  0x00007f4140f95ea2 in g_main_loop_run (loop=0x562540a21c70)
    at ../glib/gmain.c:4373
#8  0x000056253edc39a1 in main (argc=<optimized out>, argv=<optimized out>)
    at ../src/core/main.c:509
Comment 14 Jonathan Kang 2023-11-27 09:03:36 UTC 
(In reply to Andreas Schwab from comment #13)
> #0  g_source_destroy (source=0x373a37383a31393a) at ../glib/gmain.c:1386
> #1  0x000056253edcda61 in nm_g_source_destroy_and_unref ()
>     at ../src/libnm-glib-aux/nm-shared-utils.h:1699
> #2  ipv6_dad_timeout (user_data=user_data@entry=0x562540a40df0)
>     at ../src/core/dhcp/nm-dhcp-client.c:607
> #3  0x00007f4140f9637d in g_timeout_dispatch (source=0x562540bead50, 
>     callback=0x56253edcda20 <ipv6_dad_timeout>, user_data=0x562540a40df0)
>     at ../glib/gmain.c:4933
> #4  0x00007f4140f9582b in g_main_dispatch (context=0x562540a23890)
>     at ../glib/gmain.c:3381
> #5  g_main_context_dispatch (context=context@entry=0x562540a23890)
>     at ../glib/gmain.c:4099
> #6  0x00007f4140f95bd0 in g_main_context_iterate (context=0x562540a23890, 
>     block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
>     at ../glib/gmain.c:4175
> #7  0x00007f4140f95ea2 in g_main_loop_run (loop=0x562540a21c70)
>     at ../glib/gmain.c:4373
> #8  0x000056253edc39a1 in main (argc=<optimized out>, argv=<optimized out>)
>     at ../src/core/main.c:509

Can you share another NetworkManager journal when it crashes again?

Thanks
Comment 15 Andreas Schwab 2024-03-03 10:19:24 UTC 
#0  0x0000562208a33a45 in ipv6_dad_timeout (
    user_data=user_data@entry=0x562208e1ef20)
    at ../src/core/dhcp/nm-dhcp-client.c:606
#1  0x00007fdb5fc7237d in g_timeout_dispatch (source=0x562208f5fec0, 
    callback=0x562208a33a20 <ipv6_dad_timeout>, user_data=0x562208e1ef20)
    at ../glib/gmain.c:4933
#2  0x00007fdb5fc7182b in g_main_dispatch (context=0x562208e01890)
    at ../glib/gmain.c:3381
#3  g_main_context_dispatch (context=context@entry=0x562208e01890)
    at ../glib/gmain.c:4099
#4  0x00007fdb5fc71bd0 in g_main_context_iterate (context=0x562208e01890, 
    block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at ../glib/gmain.c:4175
#5  0x00007fdb5fc71ea2 in g_main_loop_run (loop=0x562208dffc70)
    at ../glib/gmain.c:4373
#6  0x0000562208a299a1 in main (argc=<optimized out>, argv=<optimized out>)
    at ../src/core/main.c:509
Comment 16 Andreas Schwab 2024-03-03 10:23:31 UTC 
$ coredumpctl list | tail -n 20
Sun 2024-03-03 10:57:31 CET   1138    0   0 SIGSEGV present  /usr/sbin/NetworkManager   791.7K
Sun 2024-03-03 10:59:44 CET   6859    0   0 SIGSEGV present  /usr/sbin/NetworkManager   815.7K
Sun 2024-03-03 11:00:55 CET   7649    0   0 SIGSEGV present  /usr/sbin/NetworkManager   822.0K
Sun 2024-03-03 11:01:30 CET   8470    0   0 SIGSEGV present  /usr/sbin/NetworkManager   837.9K
Sun 2024-03-03 11:03:24 CET   9288    0   0 SIGSEGV present  /usr/sbin/NetworkManager   851.2K
Sun 2024-03-03 11:04:30 CET  10548    0   0 SIGSEGV present  /usr/sbin/NetworkManager   804.0K
Sun 2024-03-03 11:05:09 CET  11443    0   0 SIGSEGV present  /usr/sbin/NetworkManager   807.8K
Sun 2024-03-03 11:05:43 CET  12483    0   0 SIGSEGV present  /usr/sbin/NetworkManager   803.5K
Sun 2024-03-03 11:06:18 CET  12968    0   0 SIGSEGV present  /usr/sbin/NetworkManager   783.5K
Sun 2024-03-03 11:07:36 CET  13463    0   0 SIGSEGV present  /usr/sbin/NetworkManager   786.8K
Sun 2024-03-03 11:08:11 CET  14098    0   0 SIGSEGV present  /usr/sbin/NetworkManager   789.0K
Sun 2024-03-03 11:08:46 CET  14595    0   0 SIGSEGV present  /usr/sbin/NetworkManager   789.1K
Sun 2024-03-03 11:10:21 CET  15076    0   0 SIGSEGV present  /usr/sbin/NetworkManager   813.4K
Sun 2024-03-03 11:10:56 CET  15775    0   0 SIGSEGV present  /usr/sbin/NetworkManager   805.3K
Sun 2024-03-03 11:12:43 CET  16541    0   0 SIGSEGV present  /usr/sbin/NetworkManager   784.6K
Sun 2024-03-03 11:13:17 CET  17534    0   0 SIGSEGV present  /usr/sbin/NetworkManager   797.0K
Sun 2024-03-03 11:14:23 CET  18012    0   0 SIGSEGV present  /usr/sbin/NetworkManager   820.9K
Sun 2024-03-03 11:15:50 CET  19235    0   0 SIGSEGV present  /usr/sbin/NetworkManager   825.4K
Sun 2024-03-03 11:16:58 CET  20154    0   0 SIGSEGV present  /usr/sbin/NetworkManager   826.9K
Sun 2024-03-03 11:19:33 CET  20720    0   0 SIGSEGV present  /usr/sbin/NetworkManager   793.8K
Comment 17 Jonathan Kang 2024-03-22 02:02:26 UTC 
Hi Andreas

Sorry for lacking of response here. I'm occupied with other work items atm. You can report this at upstream gitlab so that they can also take a look at this issue.

* https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues