Bug 1215837

Summary: systemd[1]: "Started Process Core Dump" in journals on localhost, recurring issue with NetworkManager
Product: [openSUSE] openSUSE Tumbleweed Reporter: Varun Kumar Ojha <varun.ojha>
Component: NetworkAssignee: Jonathan Kang <songchuan.kang>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: jcao, qe-virt, varun.ojha, xguo
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE Tumbleweed   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: Support Config KVM
/usr/sbin/NetworkManager EXE file
core file

Description Varun Kumar Ojha 2023-09-29 15:32:32 UTC 
Created attachment 869843 [details]
Support Config KVM

## Description

[Virtualization: KVM] systemd[1]: "Started Process Core Dump" in journals on localhost 

In the logs, there's a recurring issue with NetworkManager, which is triggering a "Started Process Core Dump" error in the system journals on the localhost. This issue is causing the NetworkManager service to fail with a core dump. The issue seems to be affecting the stability of the network management service on the system.

NetworkManager Version: 1.44.0-1.1
Architecture: x86_64
Operating System: openSUSE-20230927-0

## Information

blackbauhinia:~ # zypper info NetworkManager
Loading repository data...
Reading installed packages...


Information for package NetworkManager:
---------------------------------------
Repository     : openSUSE-20230927-0
Name           : NetworkManager
Version        : 1.44.0-1.1
Arch           : x86_64
Vendor         : openSUSE
Installed Size : 5.9 MiB
Installed      : Yes
Status         : up-to-date
Source package : NetworkManager-1.44.0-1.1.src
Upstream URL   : https://networkmanager.dev/
Summary        : Standard Linux network configuration tool suite
Description    : 
    NetworkManager attempts to keep an active network connection available
    at all times. The point of NetworkManager is to make networking
    configuration and setup as painless and automatic as possible. If
    using DHCP, NetworkManager is intended to replace default routes,
    obtain IP addresses from a DHCP server, and change name servers
    whenever it sees fit.

blackbauhinia:~ # coredumpctl --all
TIME                         PID UID GID SIG     COREFILE EXE                        SIZE
Fri 2023-09-29 09:40:48 EDT 1787   0   0 SIGABRT present  /usr/sbin/NetworkManager 522.1K
blackbauhinia:~ # 


## Reproduce steps 

The exact steps to reproduce the issue are not provided in the bug report. However, the issue has been observed in the logs of the system.
Possible steps
1. Install openSUSE Tumbleweed with Latest build
2. After Autoyast installation, file: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/data/virt_autotest/host_unattended_installation_files/autoyast_opensuse_kvm_two_disk.xml
3. During virt_autotest/login_console, after KVM modules are loaded and NetworkManager is seen with Core dump in journalctl


Can be seen here:
https://openqa.opensuse.org/tests/3609023#step/login_console/17
https://openqa.opensuse.org/tests/3608243#step/login_console/34
https://openqa.opensuse.org/tests/3603405#step/login_console/19


Expected result: The NetworkManager service should run without errors and manage network connections as expected.


Real result: The NetworkManager service is encountering a critical error and failing with a core dump.



## Reproducibility

seen multiple times, frequently: Not 100%

Was seen in build 20230927 

## Last good build 
was
20230922, 20230925

Now Seen at recent builds,
20230927
20230926


## Environment(optional)

Linux blackbauhinia 6.5.4-1-default #1 SMP PREEMPT_DYNAMIC Wed Sep 20 05:07:04 UTC 2023 (fdd7e9e) x86_64 x86_64 x86_64 GNU/Linux

Linux Kernel Version: 6.5.4-1-default
Hostname: blackbauhinia


## Testsuite description(optional, apply for complex scenarios)

Test suit: virt-guest-installation-kvm@64bit-ipmi & virt-guest-installation-xen@64bit-ipmi

## Error

https://openqa.opensuse.org/tests/3597532#step/login_console/19

Sep 25 00:03:57 blackbauhinia systemd[1]: Started Process Core Dump (PID 2366/UID 0).
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=>
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'.
Sep 25 00:03:57 blackbauhinia systemd[1]: Failed to start Network Manager Wait Online.
Sep 25 00:03:57 blackbauhinia systemd[1]: Reached target Network is Online.
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Login and scanning of iSCSI devices...
Sep 25 00:03:57 blackbauhinia iscsiadm[2369]: iscsiadm: No records found
Sep 25 00:03:57 blackbauhinia iscsiadm[2371]: iscsiadm: No records found
Sep 25 00:03:57 blackbauhinia iscsiadm[2373]: iscsiadm: Could not get list of targets from firmware. (err 21)
Sep 25 00:03:57 blackbauhinia systemd-coredump[2367]: [🡕] Process 1781 (NetworkManager) of user 0 dumped core.

                                                      Stack trace of thread 1781:
                                                      #0  0x00007f42f2691e0c __pthread_kill_implementation (libc.so.6 +>
                                                      #1  0x00007f42f263f0e6 raise (libc.so.6 + 0x3f0e6)
                                                      #2  0x00007f42f26268d7 abort (libc.so.6 + 0x268d7)
                                                      #3  0x00007f42f2e58b1d g_mutex_clear (libglib-2.0.so.0 + 0xb7b1d)
                                                      #4  0x00007f42f30094a2 n/a (libgio-2.0.so.0 + 0xbe4a2)
                                                      #5  0x00007f42f3005b4c n/a (libgio-2.0.so.0 + 0xbab4c)
                                                      #6  0x00007f42f2f05970 g_object_unref (libgobject-2.0.so.0 + 0x1c>
                                                      #7  0x00007f42f2dfaf89 n/a (libglib-2.0.so.0 + 0x59f89)
                                                      #8  0x00007f42f2dfcb46 n/a (libglib-2.0.so.0 + 0x5bb46)
                                                      #9  0x00007f42f2dfcf6f n/a (libglib-2.0.so.0 + 0x5bf6f)
                                                      #10 0x00007f42f2dfeb18 n/a (libglib-2.0.so.0 + 0x5db18)
                                                      #11 0x00007f42f2dff3ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3e>
                                                      #12 0x00005643adfd6493 n/a (NetworkManager + 0x3f493)
                                                      #13 0x00007f42f26281b0 __libc_start_call_main (libc.so.6 + 0x281b>
                                                      #14 0x00007f42f2628279 __libc_start_main@@GLIBC_2.34 (libc.so.6 +>
                                                      #15 0x00005643adfd6915 n/a (NetworkManager + 0x3f915)

                                                      Stack trace of thread 1791:
                                                      #0  0x00007f42f2709d2f __poll (libc.so.6 + 0x109d2f)
                                                      #1  0x00007f42f2dfeabf n/a (libglib-2.0.so.0 + 0x5dabf)
                                                      #2  0x00007f42f2dff1cc g_main_context_iteration (libglib-2.0.so.0>
                                                      #3  0x00007f42f2dff211 n/a (libglib-2.0.so.0 + 0x5e211)
                                                      #4  0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e)
                                                      #5  0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64)
                                                      #6  0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c)

                                                      Stack trace of thread 1792:
                                                      #0  0x00007f42f271616d syscall (libc.so.6 + 0x11616d)
                                                      #1  0x00007f42f2e58cb0 g_cond_wait (libglib-2.0.so.0 + 0xb7cb0)
                                                      #2  0x00007f42f2dc902b n/a (libglib-2.0.so.0 + 0x2802b)
                                                      #3  0x00007f42f2e2ba92 n/a (libglib-2.0.so.0 + 0x8aa92)
                                                      #4  0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e)
                                                      #5  0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64)
                                                      #6  0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c)

                                                      Stack trace of thread 1793:
                                                      #0  0x00007f42f2709d2f __poll (libc.so.6 + 0x109d2f)
                                                      #1  0x00007f42f2dfeabf n/a (libglib-2.0.so.0 + 0x5dabf)
                                                      #2  0x00007f42f2dff3ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3e>
                                                      #3  0x00007f42f30708e6 n/a (libgio-2.0.so.0 + 0x1258e6)
                                                      #4  0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e)
                                                      #5  0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64)
                                                      #6  0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c)
                                                      Stack trace of thread 2365:
                                                      #0  0x00007f42f271616d syscall (libc.so.6 + 0x11616d)
                                                      #1  0x00007f42f2e58e5c g_cond_wait_until (libglib-2.0.so.0 + 0xb7>
                                                      #2  0x00007f42f2dc9003 n/a (libglib-2.0.so.0 + 0x28003)
                                                      #3  0x00007f42f2e2be0a n/a (libglib-2.0.so.0 + 0x8ae0a)
                                                      #4  0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e)
                                                      #5  0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64)
                                                      #6  0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c)
                                                      ELF object binary architecture: AMD x86-64
Sep 25 00:03:57 blackbauhinia systemd[1]: Finished Login and scanning of iSCSI devices.
Sep 25 00:03:57 blackbauhinia systemd[1]: systemd-coredump@0-2366-0.service: Deactivated successfully.
Sep 25 00:03:57 blackbauhinia systemd[1]: Reached target Remote File Systems.
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Autoyast2 Init Scripts...
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Virtualization qemu daemon...
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Virtualization storage daemon...
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager.service: Main process exited, code=dumped, status=6/ABRT
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager.service: Failed with result 'core-dump'.




## Logs/ crash

 # /bin/coredumpctl info 1787
           PID: 1787 (NetworkManager)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Fri 2023-09-29 09:40:48 EDT (53min ago)
  Command Line: /usr/sbin/NetworkManager --no-daemon
    Executable: /usr/sbin/NetworkManager
 Control Group: /system.slice/NetworkManager.service
          Unit: NetworkManager.service
         Slice: system.slice
       Boot ID: 752a88f1ac2b4381b0fc7591961e743a
    Machine ID: 629cb90a95bb451e9026683e7cb65723
      Hostname: blackbauhinia
       Storage: /var/lib/systemd/coredump/core.NetworkManager.0.752a88f1ac2b4381b0fc7591961e743a.1787.1695994848000000.zst (present)
  Size on Disk: 522.1K
       Message: Process 1787 (NetworkManager) of user 0 dumped core.
                
                Stack trace of thread 1787:
                #0  0x00007f4ed8491e0c __pthread_kill_implementation (libc.so.6 + 0x91e0c)
                #1  0x00007f4ed843f0e6 raise (libc.so.6 + 0x3f0e6)
                #2  0x00007f4ed84268d7 abort (libc.so.6 + 0x268d7)
                #3  0x00007f4ed8bcdb1d g_mutex_clear (libglib-2.0.so.0 + 0xb7b1d)
                #4  0x00007f4ed8d7e4a2 n/a (libgio-2.0.so.0 + 0xbe4a2)
                #5  0x00007f4ed8d7ab4c n/a (libgio-2.0.so.0 + 0xbab4c)
                #6  0x00007f4ed8c7a970 g_object_unref (libgobject-2.0.so.0 + 0x1c970)
                #7  0x00007f4ed8b6ff89 n/a (libglib-2.0.so.0 + 0x59f89)
                #8  0x00007f4ed8b71b46 n/a (libglib-2.0.so.0 + 0x5bb46)
                #9  0x00007f4ed8b71f6f n/a (libglib-2.0.so.0 + 0x5bf6f)
                #10 0x00007f4ed8b73b18 n/a (libglib-2.0.so.0 + 0x5db18)
                #11 0x00007f4ed8b743ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3ef)
                #12 0x00005652d1963493 n/a (NetworkManager + 0x3f493)
                #13 0x00007f4ed84281b0 __libc_start_call_main (libc.so.6 + 0x281b0)
                #14 0x00007f4ed8428279 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x28279)
                #15 0x00005652d1963915 n/a (NetworkManager + 0x3f915)
                
                Stack trace of thread 1796:
                #0  0x00007f4ed8509d2f __poll (libc.so.6 + 0x109d2f)
                #1  0x00007f4ed8b73abf n/a (libglib-2.0.so.0 + 0x5dabf)
                #2  0x00007f4ed8b741cc g_main_context_iteration (libglib-2.0.so.0 + 0x5e1cc)
                #3  0x00007f4ed8b74211 n/a (libglib-2.0.so.0 + 0x5e211)
                #4  0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
                #5  0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
                #6  0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
                
                Stack trace of thread 1797:
                #0  0x00007f4ed851616d syscall (libc.so.6 + 0x11616d)
                #1  0x00007f4ed8bcdcb0 g_cond_wait (libglib-2.0.so.0 + 0xb7cb0)
                #2  0x00007f4ed8b3e02b n/a (libglib-2.0.so.0 + 0x2802b)
                #3  0x00007f4ed8ba0a92 n/a (libglib-2.0.so.0 + 0x8aa92)
                #4  0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
                #5  0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
                #6  0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
                
                Stack trace of thread 2369:
                #0  0x00007f4ed851616d syscall (libc.so.6 + 0x11616d)
                #1  0x00007f4ed8bcde5c g_cond_wait_until (libglib-2.0.so.0 + 0xb7e5c)
                #2  0x00007f4ed8b3e003 n/a (libglib-2.0.so.0 + 0x28003)
                #3  0x00007f4ed8ba0e0a n/a (libglib-2.0.so.0 + 0x8ae0a)
                #4  0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
                #5  0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
                #6  0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
                
                Stack trace of thread 1798:
                #0  0x00007f4ed8509d2f __poll (libc.so.6 + 0x109d2f)
                #1  0x00007f4ed8b73abf n/a (libglib-2.0.so.0 + 0x5dabf)
                #2  0x00007f4ed8b743ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3ef)
                #3  0x00007f4ed8de58e6 n/a (libgio-2.0.so.0 + 0x1258e6)
                #4  0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
                #5  0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
                #6  0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
                ELF object binary architecture: AMD x86-64


The stack trace of the core dump is also provided in the logs.



## Workaround

No

Please note that resolving this issue may require further investigation, debugging, and potentially collaboration with the NetworkManager development team.
Comment 1 Varun Kumar Ojha 2023-09-29 15:35:16 UTC 
Created attachment 869844 [details]
/usr/sbin/NetworkManager EXE file
Comment 2 Varun Kumar Ojha 2023-09-29 15:38:24 UTC 
Created attachment 869845 [details]
core file
Comment 3 James Fehlig 2023-09-29 16:45:46 UTC 
Why is the component KVM? IIUC the issue is with NetworkManager, right? What does that have to do with KVM or virtualization in general?
Comment 4 Varun Kumar Ojha 2023-09-29 16:51:19 UTC 
You are absolutely correct, and I apologize for the confusion. The issue indeed appears to be related to NetworkManager, and there is no direct connection to KVM or virtualization. 

I kept KVM because it is seen twice on KVM.

The use of "KVM" in the component field seems to be a misclassification. It should be corrected to accurately reflect that the problem is with NetworkManager and not KVM or virtualization.

Thank you for pointing out this discrepancy, and I appreciate your attention to detail.
Comment 5 Varun Kumar Ojha 2023-09-29 16:57:42 UTC 
Please let me know if you want the following actions: Shall I?

1. I can update the component field from "KVM" to "Network" to accurately reflect the software component involved in this issue.

2. Remove the term "virtualization" from the issue title to provide a more precise description of the problem.

These changes will help us maintain clear and accurate records of reported issues. Your feedback is valuable in improving our issue-tracking process.

Thanks!
Comment 6 James Fehlig 2023-09-29 17:02:21 UTC 
(In reply to Varun Kumar Ojha from comment #5)
> Please let me know if you want the following actions: Shall I?
> 
> 1. I can update the component field from "KVM" to "Network" to accurately
> reflect the software component involved in this issue.
> 
> 2. Remove the term "virtualization" from the issue title to provide a more
> precise description of the problem.

These changes sound good. I might as well make them while writing this...
Comment 7 Varun Kumar Ojha 2023-10-05 06:10:22 UTC 
Following up, is the fix applied?
The issue is not seen in recent builds but please confirm and update what has been changed and its resolution.

Latest tests
KVM: https://openqa.opensuse.org/tests/3618911#next_previous 
XEN: https://openqa.opensuse.org/tests/3618912#next_previous
Comment 8 Jonathan Kang 2023-10-09 02:28:12 UTC 
(In reply to Varun Kumar Ojha from comment #7)
> Following up, is the fix applied?
> The issue is not seen in recent builds but please confirm and update what
> has been changed and its resolution.
> 
> Latest tests
> KVM: https://openqa.opensuse.org/tests/3618911#next_previous 
> XEN: https://openqa.opensuse.org/tests/3618912#next_previous

NetworkManager was updated to version 1.44.2 a few days ago[1]. The changelog indicates a possible segment fault was fixed. Maybe that is related with this issue.

*[1] https://build.opensuse.org/request/show/1115910
Comment 9 Varun Kumar Ojha 2023-10-17 06:23:54 UTC 
Marking the ticket as closed, the issue was not seen. 

We will monitor progress tickets, If it is reproducible,  we can refer back. 

Thanks James Fehlig and Jonathan Kang