Bug 1213209

Summary: nm-openvpn-service crashes
Product: [openSUSE] openSUSE Tumbleweed Reporter: Matej Cepl <mcepl>
Component: SecurityAssignee: openSUSE GNOME <os.gnome.maintainers>
Status: RESOLVED WORKSFORME QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: fvogt, mcepl, pdostal
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Matej Cepl 2023-07-11 13:52:55 UTC
When trying to start OpenVPN via nmcli I get this in the journal

stitny:~# journalctl -f
Jul 11 15:50:25 stitny.cepl.eu systemd[1]: Started Process Core Dump (PID 3056/UID 0).
Jul 11 15:50:25 stitny.cepl.eu systemd-coredump[3057]: [🡕] Process 3050 (nm-openvpn-serv) of user 0 dumped core.
                                                       
                                                       Stack trace of thread 3050:
                                                       #0  0x00007f715817f6ad __strlen_avx2 (libc.so.6 + 0x1686ad)
                                                       #1  0x00005654537109ce n/a (nm-openvpn-service + 0x59ce)
                                                       #2  0x0000565453710c7f n/a (nm-openvpn-service + 0x5c7f)
                                                       #3  0x000056545371100b n/a (nm-openvpn-service + 0x600b)
                                                       #4  0x0000565453711792 n/a (nm-openvpn-service + 0x6792)
                                                       #5  0x00007f71582718d8 g_main_context_dispatch (libglib-2.0.so.0 + 0x5d8d8)
                                                       #6  0x00007f7158271ce8 n/a (libglib-2.0.so.0 + 0x5dce8)
                                                       #7  0x00007f7158271f9f g_main_loop_run (libglib-2.0.so.0 + 0x5df9f)
                                                       #8  0x000056545370fbad n/a (nm-openvpn-service + 0x4bad)
                                                       #9  0x00007f715803ebb0 __libc_start_call_main (libc.so.6 + 0x27bb0)
                                                       #10 0x00007f715803ec79 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c79)
                                                       #11 0x000056545370fe85 n/a (nm-openvpn-service + 0x4e85)
                                                       
                                                       Stack trace of thread 3052:
                                                       #0  0x00007f715811e44f __poll (libc.so.6 + 0x10744f)
                                                       #1  0x00007f7158271c5e n/a (libglib-2.0.so.0 + 0x5dc5e)
                                                       #2  0x00007f7158271d7c g_main_context_iteration (libglib-2.0.so.0 + 0x5dd7c)
                                                       #3  0x00007f7158271dc1 n/a (libglib-2.0.so.0 + 0x5ddc1)
                                                       #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                                                       #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                                                       #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                                                       
                                                       Stack trace of thread 3051:
                                                       #0  0x00007f71581243dd syscall (libc.so.6 + 0x10d3dd)
                                                       #1  0x00007f71582cb35f g_cond_wait (libglib-2.0.so.0 + 0xb735f)
                                                       #2  0x00007f715823bf4b n/a (libglib-2.0.so.0 + 0x27f4b)
                                                       #3  0x00007f715829e552 n/a (libglib-2.0.so.0 + 0x8a552)
                                                       #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                                                       #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                                                       #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                                                       
                                                       Stack trace of thread 3053:
                                                       #0  0x00007f715811e44f __poll (libc.so.6 + 0x10744f)
                                                       #1  0x00007f7158271c5e n/a (libglib-2.0.so.0 + 0x5dc5e)
                                                       #2  0x00007f7158271f9f g_main_loop_run (libglib-2.0.so.0 + 0x5df9f)
                                                       #3  0x00007f71584e08c6 n/a (libgio-2.0.so.0 + 0x1228c6)
                                                       #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                                                       #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                                                       #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                                                       ELF object binary architecture: AMD x86-64
Jul 11 15:50:25 stitny.cepl.eu systemd[1]: systemd-coredump@1-3056-0.service: Deactivated successfully.

Using NetworkManager-openvpn-1.10.2-1.3.x86_64 and openvpn-2.6.5-1.1.x86_64
Comment 1 Fabian Vogt 2023-07-11 13:57:53 UTC
Please run coredumpctl gdb 3050, let it download some debug info and run "thread apply all bt" to get an annotated backtrace.
Comment 2 Matej Cepl 2023-07-11 14:22:18 UTC
stitny:~# coredumpctl gdb 3050
           PID: 3050 (nm-openvpn-serv)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2023-07-11 15:50:25 CEST (30min ago)
  Command Line: /usr/libexec/nm-openvpn-service --bus-name org.freedesktop.NetworkManager.openvpn.Connection_7
    Executable: /usr/libexec/nm-openvpn-service
 Control Group: /system.slice/NetworkManager.service
          Unit: NetworkManager.service
         Slice: system.slice
       Boot ID: 356005d6d7ba4a898e53e764792bdd6f
    Machine ID: 842ead12e6194bbca2276c36a8279657
      Hostname: stitny.cepl.eu
       Storage: /var/lib/systemd/coredump/core.nm-openvpn-serv.0.356005d6d7ba4a898e53e764792bdd6f.3050.1689083425000000.zst (present)
  Size on Disk: 229.4K
       Message: Process 3050 (nm-openvpn-serv) of user 0 dumped core.
                
                Stack trace of thread 3050:
                #0  0x00007f715817f6ad __strlen_avx2 (libc.so.6 + 0x1686ad)
                #1  0x00005654537109ce n/a (nm-openvpn-service + 0x59ce)
                #2  0x0000565453710c7f n/a (nm-openvpn-service + 0x5c7f)
                #3  0x000056545371100b n/a (nm-openvpn-service + 0x600b)
                #4  0x0000565453711792 n/a (nm-openvpn-service + 0x6792)
                #5  0x00007f71582718d8 g_main_context_dispatch (libglib-2.0.so.0 + 0x5d8d8)
                #6  0x00007f7158271ce8 n/a (libglib-2.0.so.0 + 0x5dce8)
                #7  0x00007f7158271f9f g_main_loop_run (libglib-2.0.so.0 + 0x5df9f)
                #8  0x000056545370fbad n/a (nm-openvpn-service + 0x4bad)
                #9  0x00007f715803ebb0 __libc_start_call_main (libc.so.6 + 0x27bb0)
                #10 0x00007f715803ec79 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c79)
                #11 0x000056545370fe85 n/a (nm-openvpn-service + 0x4e85)
                
                Stack trace of thread 3052:
                #0  0x00007f715811e44f __poll (libc.so.6 + 0x10744f)
                #1  0x00007f7158271c5e n/a (libglib-2.0.so.0 + 0x5dc5e)
                #2  0x00007f7158271d7c g_main_context_iteration (libglib-2.0.so.0 + 0x5dd7c)
                #3  0x00007f7158271dc1 n/a (libglib-2.0.so.0 + 0x5ddc1)
                #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                
                Stack trace of thread 3051:
                #0  0x00007f71581243dd syscall (libc.so.6 + 0x10d3dd)
                #1  0x00007f71582cb35f g_cond_wait (libglib-2.0.so.0 + 0xb735f)
                #2  0x00007f715823bf4b n/a (libglib-2.0.so.0 + 0x27f4b)
                #3  0x00007f715829e552 n/a (libglib-2.0.so.0 + 0x8a552)
                #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                
                Stack trace of thread 3053:
                #0  0x00007f715811e44f __poll (libc.so.6 + 0x10744f)
                #1  0x00007f7158271c5e n/a (libglib-2.0.so.0 + 0x5dc5e)
                #2  0x00007f7158271f9f g_main_loop_run (libglib-2.0.so.0 + 0x5df9f)
                #3  0x00007f71584e08c6 n/a (libgio-2.0.so.0 + 0x1228c6)
                #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                ELF object binary architecture: AMD x86-64

GNU gdb (GDB; openSUSE Tumbleweed) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/libexec/nm-openvpn-service...
(No debugging symbols found in /usr/libexec/nm-openvpn-service)
[New LWP 3050]
[New LWP 3052]
[New LWP 3051]
[New LWP 3053]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/nm-openvpn-service --bus-name org.freedesktop.NetworkManager.openv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f715817f6ad in __strlen_avx2 () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f7157c6d080 (LWP 3050))]
Missing separate debuginfos, use: zypper install NetworkManager-openvpn-debuginfo-1.10.2-1.3.x86_64
(gdb) t a a bt

Thread 4 (Thread 0x7f71567fd6c0 (LWP 3053)):
#0  0x00007f715811e44f in poll () from /lib64/libc.so.6
#1  0x00007f7158271c5e in ?? () from /lib64/libglib-2.0.so.0
#2  0x00007f7158271f9f in g_main_loop_run () from /lib64/libglib-2.0.so.0
#3  0x00007f71584e08c6 in ?? () from /lib64/libgio-2.0.so.0
#4  0x00007f715829df0e in ?? () from /lib64/libglib-2.0.so.0
#5  0x00007f71580a4c24 in start_thread () from /lib64/libc.so.6
#6  0x00007f715812c510 in clone3 () from /lib64/libc.so.6

Thread 3 (Thread 0x7f71577ff6c0 (LWP 3051)):
#0  0x00007f71581243dd in syscall () from /lib64/libc.so.6
#1  0x00007f71582cb35f in g_cond_wait () from /lib64/libglib-2.0.so.0
#2  0x00007f715823bf4b in ?? () from /lib64/libglib-2.0.so.0
#3  0x00007f715829e552 in ?? () from /lib64/libglib-2.0.so.0
#4  0x00007f715829df0e in ?? () from /lib64/libglib-2.0.so.0
#5  0x00007f71580a4c24 in start_thread () from /lib64/libc.so.6
#6  0x00007f715812c510 in clone3 () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7156ffe6c0 (LWP 3052)):
#0  0x00007f715811e44f in poll () from /lib64/libc.so.6
#1  0x00007f7158271c5e in ?? () from /lib64/libglib-2.0.so.0
#2  0x00007f7158271d7c in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#3  0x00007f7158271dc1 in ?? () from /lib64/libglib-2.0.so.0
#4  0x00007f715829df0e in ?? () from /lib64/libglib-2.0.so.0
#5  0x00007f71580a4c24 in start_thread () from /lib64/libc.so.6
#6  0x00007f715812c510 in clone3 () from /lib64/libc.so.6

Thread 1 (Thread 0x7f7157c6d080 (LWP 3050)):
#0  0x00007f715817f6ad in __strlen_avx2 () from /lib64/libc.so.6
#1  0x00005654537109ce in ?? ()
#2  0x0000565453710c7f in ?? ()
#3  0x000056545371100b in ?? ()
#4  0x0000565453711792 in ?? ()
#5  0x00007f71582718d8 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#6  0x00007f7158271ce8 in ?? () from /lib64/libglib-2.0.so.0
#7  0x00007f7158271f9f in g_main_loop_run () from /lib64/libglib-2.0.so.0
#8  0x000056545370fbad in ?? ()
#9  0x00007f715803ebb0 in __libc_start_call_main () from /lib64/libc.so.6
#10 0x00007f715803ec79 in __libc_start_main_impl () from /lib64/libc.so.6
#11 0x000056545370fe85 in ?? ()
(gdb) 

Unfortunately this is MicroOS, so I have hard time to install debuginfo packages.
Comment 3 Fabian Vogt 2023-07-11 14:27:32 UTC
(In reply to Matej Cepl from comment #2)
> Unfortunately this is MicroOS, so I have hard time to install debuginfo
> packages.

Sounds like debuginfod isn't configured on MicroOS. Can you try this?

export DEBUGINFOD_URLS=https://debuginfod.opensuse.org/
coredumpctl gdb 3050
Comment 4 Matej Cepl 2023-07-11 21:35:26 UTC
stitny:~# coredumpctl gdb 3050
           PID: 3050 (nm-openvpn-serv)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2023-07-11 15:50:25 CEST (7h ago)
  Command Line: /usr/libexec/nm-openvpn-service --bus-name org.freedesktop.NetworkManager.openvpn.Connection_7
    Executable: /usr/libexec/nm-openvpn-service
 Control Group: /system.slice/NetworkManager.service
          Unit: NetworkManager.service
         Slice: system.slice
       Boot ID: 356005d6d7ba4a898e53e764792bdd6f
    Machine ID: 842ead12e6194bbca2276c36a8279657
      Hostname: stitny.cepl.eu
       Storage: /var/lib/systemd/coredump/core.nm-openvpn-serv.0.356005d6d7ba4a898e53e764792bdd6f.3050.1689083425000000.zst (present)
  Size on Disk: 229.4K
       Message: Process 3050 (nm-openvpn-serv) of user 0 dumped core.
                
                Stack trace of thread 3050:
                #0  0x00007f715817f6ad __strlen_avx2 (libc.so.6 + 0x1686ad)
                #1  0x00005654537109ce n/a (nm-openvpn-service + 0x59ce)
                #2  0x0000565453710c7f n/a (nm-openvpn-service + 0x5c7f)
                #3  0x000056545371100b n/a (nm-openvpn-service + 0x600b)
                #4  0x0000565453711792 n/a (nm-openvpn-service + 0x6792)
                #5  0x00007f71582718d8 g_main_context_dispatch (libglib-2.0.so.0 + 0x5d8d8)
                #6  0x00007f7158271ce8 n/a (libglib-2.0.so.0 + 0x5dce8)
                #7  0x00007f7158271f9f g_main_loop_run (libglib-2.0.so.0 + 0x5df9f)
                #8  0x000056545370fbad n/a (nm-openvpn-service + 0x4bad)
                #9  0x00007f715803ebb0 __libc_start_call_main (libc.so.6 + 0x27bb0)
                #10 0x00007f715803ec79 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c79)
                #11 0x000056545370fe85 n/a (nm-openvpn-service + 0x4e85)
                
                Stack trace of thread 3052:
                #0  0x00007f715811e44f __poll (libc.so.6 + 0x10744f)
                #1  0x00007f7158271c5e n/a (libglib-2.0.so.0 + 0x5dc5e)
                #2  0x00007f7158271d7c g_main_context_iteration (libglib-2.0.so.0 + 0x5dd7c)
                #3  0x00007f7158271dc1 n/a (libglib-2.0.so.0 + 0x5ddc1)
                #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                
                Stack trace of thread 3051:
                #0  0x00007f71581243dd syscall (libc.so.6 + 0x10d3dd)
                #1  0x00007f71582cb35f g_cond_wait (libglib-2.0.so.0 + 0xb735f)
                #2  0x00007f715823bf4b n/a (libglib-2.0.so.0 + 0x27f4b)
                #3  0x00007f715829e552 n/a (libglib-2.0.so.0 + 0x8a552)
                #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                
                Stack trace of thread 3053:
                #0  0x00007f715811e44f __poll (libc.so.6 + 0x10744f)
                #1  0x00007f7158271c5e n/a (libglib-2.0.so.0 + 0x5dc5e)
                #2  0x00007f7158271f9f g_main_loop_run (libglib-2.0.so.0 + 0x5df9f)
                #3  0x00007f71584e08c6 n/a (libgio-2.0.so.0 + 0x1228c6)
                #4  0x00007f715829df0e n/a (libglib-2.0.so.0 + 0x89f0e)
                #5  0x00007f71580a4c24 start_thread (libc.so.6 + 0x8dc24)
                #6  0x00007f715812c510 __clone3 (libc.so.6 + 0x115510)
                ELF object binary architecture: AMD x86-64

GNU gdb (GDB; openSUSE Tumbleweed) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/libexec/nm-openvpn-service...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.opensuse.org/>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/libexec/nm-openvpn-service
Reading symbols from /root/.cache/debuginfod_client/6a8d3fb70460b27acf1a74f29eb546d74cc34f10/debuginfo...
Downloading separate debug info for /root/.cache/debuginfod_client/6a8d3fb70460b27acf1a74f29eb546d74cc34f10/debuginfo
[New LWP 3050]                                                                              
[New LWP 3052]
[New LWP 3051]
[New LWP 3053]
Downloading separate debug info for /lib64/libnm.so.0
Downloading separate debug info for /lib64/libgio-2.0.so.0                                  
Downloading separate debug info for /lib64/libgobject-2.0.so.0                              
Downloading separate debug info for /lib64/libglib-2.0.so.0                                 
Downloading separate debug info for /lib64/libc.so.6                                        
Downloading separate debug info for /lib64/libgmodule-2.0.so.0                              
Downloading separate debug info for /lib64/libudev.so.1                                     
Downloading separate debug info for /lib64/libsmime3.so                                     
Downloading separate debug info for /lib64/libnss3.so                                       
Downloading separate debug info for /lib64/libnspr4.so                                      
--Type <RET> for more, q to quit, c to continue without paging--c
Downloading separate debug info for /lib64/ld-linux-x86-64.so.2                             
Downloading separate debug info for /lib64/libz.so.1                                        
Downloading separate debug info for /lib64/libmount.so.1                                    
Downloading separate debug info for /lib64/libselinux.so.1                                  
Downloading separate debug info for /lib64/libffi.so.8                                      
Downloading separate debug info for /lib64/libpcre2-8.so.0                                  
Downloading separate debug info for /lib64/libcap.so.2                                      
Downloading separate debug info for /lib64/libnssutil3.so                                   
Downloading separate debug info for /lib64/libplc4.so                                       
Downloading separate debug info for /lib64/libplds4.so                                      
Downloading separate debug info for /lib64/libblkid.so.1                                    
Downloading separate debug info for /lib64/libsoftokn3.so                                   
Downloading separate debug info for /lib64/libsqlite3.so.0                                  
Downloading separate debug info for /lib64/libm.so.6                                        
Downloading separate debug info for /lib64/libfreeblpriv3.so                                
Downloading separate debug info for /lib64/libnss_compat.so.2                               
Downloading separate debug info for system-supplied DSO at 0x7ffe337f9000                   
[Thread debugging using libthread_db enabled]                                               
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/nm-openvpn-service --bus-name org.freedesktop.NetworkManager.openv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
Downloading source file /usr/src/debug/glibc-2.37/string/../sysdeps/x86_64/multiarch/strlen-avx2.S
76		VPCMPEQ	(%rdi), %ymm0, %ymm1                                                
[Current thread is 1 (Thread 0x7f7157c6d080 (LWP 3050))]
(gdb) t a a bt

Thread 4 (Thread 0x7f71567fd6c0 (LWP 3053)):
#0  0x00007f715811e44f in __GI___poll (fds=0x7f714c000b90, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f7158271c5e in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7f714c000b90, timeout=<optimized out>, context=0x56545408cef0) at ../glib/gmain.c:4584
#2  g_main_context_iterate (context=0x56545408cef0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4271
#3  0x00007f7158271f9f in g_main_loop_run (loop=0x56545408d020) at ../glib/gmain.c:4479
#4  0x00007f71584e08c6 in gdbus_shared_thread_func (user_data=0x56545407e7d0) at ../gio/gdbusprivate.c:284
#5  0x00007f715829df0e in g_thread_proxy (data=0x565454088c70) at ../glib/gthread.c:831
#6  0x00007f71580a4c24 in start_thread (arg=<optimized out>) at pthread_create.c:444
#7  0x00007f715812c510 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f71577ff6c0 (LWP 3051)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f71582cb35f in g_cond_wait (cond=cond@entry=0x5654540807c8, mutex=mutex@entry=0x5654540807c0) at ../glib/gthread-posix.c:1475
#2  0x00007f715823bf4b in g_async_queue_pop_intern_unlocked (queue=0x5654540807c0, wait=1, end_time=-1) at ../glib/gasyncqueue.c:425
#3  0x00007f715829e552 in g_thread_pool_spawn_thread (data=<optimized out>) at ../glib/gthreadpool.c:311
#4  0x00007f715829df0e in g_thread_proxy (data=0x56545407c270) at ../glib/gthread.c:831
#5  0x00007f71580a4c24 in start_thread (arg=<optimized out>) at pthread_create.c:444
#6  0x00007f715812c510 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f7156ffe6c0 (LWP 3052)):
#0  0x00007f715811e44f in __GI___poll (fds=0x565454080d10, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f7158271c5e in g_main_context_poll (priority=<optimized out>, n_fds=1, fds=0x565454080d10, timeout=<optimized out>, context=0x565454080aa0) at ../glib/gmain.c:4584
#2  g_main_context_iterate (context=context@entry=0x565454080aa0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4271
#3  0x00007f7158271d7c in g_main_context_iteration (context=0x565454080aa0, may_block=may_block@entry=1) at ../glib/gmain.c:4343
#4  0x00007f7158271dc1 in glib_worker_main (data=<optimized out>) at ../glib/gmain.c:6455
#5  0x00007f715829df0e in g_thread_proxy (data=0x565454080cb0) at ../glib/gthread.c:831
#6  0x00007f71580a4c24 in start_thread (arg=<optimized out>) at pthread_create.c:444
#7  0x00007f715812c510 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f7157c6d080 (LWP 3050)):
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1  0x00005654537109ce in write_user_pass (channel=0x56545408a480, authtype=authtype@entry=0x5654540c0920 "Auth", user=user@entry=0x5654540c0f90 "mcepl", pass=<optimized out>) at src/nm-openvpn-service.c:862
#2  0x0000565453710c7f in handle_auth (io_data=0x5654540c0f30, requested_auth=requested_auth@entry=0x5654540c0920 "Auth", out_message=out_message@entry=0x7ffe33798030, out_hints=out_hints@entry=0x7ffe33798040) at src/nm-openvpn-service.c:908
#3  0x000056545371100b in handle_management_socket (plugin=plugin@entry=0x56545407eb90, source=<optimized out>, condition=<optimized out>, out_failure=out_failure@entry=0x7ffe337980a4)--Type <RET> for more, q to quit, c to continue without paging--c
 at src/nm-openvpn-service.c:1021
#4  0x0000565453711792 in nm_openvpn_socket_data_cb (source=<optimized out>, condition=<optimized out>, user_data=0x56545407eb90) at src/nm-openvpn-service.c:1089
#5  0x00007f71582718d8 in g_main_dispatch (context=0x565454087550) at ../glib/gmain.c:3460
#6  g_main_context_dispatch (context=context@entry=0x565454087550) at ../glib/gmain.c:4200
#7  0x00007f7158271ce8 in g_main_context_iterate (context=0x565454087550, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4276
#8  0x00007f7158271f9f in g_main_loop_run (loop=loop@entry=0x56545408d8e0) at ../glib/gmain.c:4479
#9  0x000056545370fbad in main (argc=<optimized out>, argv=<optimized out>) at src/nm-openvpn-service.c:2435
(gdb)
Comment 5 Fabian Vogt 2023-07-12 06:34:09 UTC
(In reply to Matej Cepl from comment #4)
> Thread 1 (Thread 0x7f7157c6d080 (LWP 3050)):
> #0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
> #1  0x00005654537109ce in write_user_pass (channel=0x56545408a480, authtype=authtype@entry=0x5654540c0920 "Auth", user=user@entry=0x5654540c0f90 "mcepl", pass=<optimized out>) at src/nm-openvpn-service.c:862
> #2  0x0000565453710c7f in handle_auth (io_data=0x5654540c0f30, requested_auth=requested_auth@entry=0x5654540c0920 "Auth", out_message=out_message@entry=0x7ffe33798030, out_hints=out_hints@entry=0x7ffe33798040) at src/nm-openvpn-service.c:908
> #3  0x000056545371100b in handle_management_socket (plugin=plugin@entry=0x56545407eb90, source=<optimized out>, condition=<optimized out>, out_failure=out_failure@entry=0x7ffe337980a4) at src/nm-openvpn-service.c:1021

Apparently the password is an empty string here. This causes g_malloc0 and thus ovpn_quote_string to return NULL and thus it crashes in strlen here:

https://gitlab.gnome.org/GNOME/NetworkManager-openvpn/-/blob/1.10.2/src/nm-openvpn-service.c?ref_type=tags#L862

I assume the password is not actually empty?
Comment 6 Matej Cepl 2023-07-18 13:56:27 UTC
This is my configuration file (of course, the password has been masked out):

stitny:~# cat /etc/NetworkManager/system-connections/mcepl.SUSE-UDP-PRG-NM_setting.nmconnection 
[connection]
id=mcepl.SUSE-UDP-PRG-NM
uuid=d4baedfa-0eb0-4c50-bf83-875174c52c33
type=vpn
autoconnect-retries=0
permissions=user:matej:;
timestamp=1682709375

[vpn]
auth=SHA512
ca=/etc/openvpn/SUSE/SUSE_Trust_Root.crt
cert=/etc/openvpn/SUSE/SUSE-mcepl.crt
cert-pass-flags=0
cipher=AES-256-CBC
comp-lzo=adaptive
connection-type=password-tls
dev=tun
dev-type=tun
key=/etc/openvpn/SUSE/SUSE-mcepl.key
password-flags=0
ping=10
ping-restart=30
port=1194
remote=gate.suse.cz
ta=/etc/openvpn/SUSE/ta.key
ta-dir=1
username=mcepl
persistent=true
service-type=org.freedesktop.NetworkManager.openvpn

[vpn-secrets]
password=VERYSECRETPASSWORD

[ipv4]
method=auto
never-default=true

[ipv6]
addr-gen-mode=stable-privacy
method=auto
never-default=true

[proxy]
stitny:~# 

and this is the result:


stitny:~# coredumpctl gdb 19132
           PID: 19132 (nm-openvpn-serv)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2023-07-18 15:52:12 CEST (1min 48s ago)
  Command Line: /usr/libexec/nm-openvpn-service --bus-name org.freedesktop.NetworkManager.openvpn.Connection_5
    Executable: /usr/libexec/nm-openvpn-service
 Control Group: /system.slice/NetworkManager.service
          Unit: NetworkManager.service
         Slice: system.slice
       Boot ID: 70ffc3080ba24f0f9eff395e2fabd102
    Machine ID: 842ead12e6194bbca2276c36a8279657
      Hostname: stitny.cepl.eu
       Storage: /var/lib/systemd/coredump/core.nm-openvpn-serv.0.70ffc3080ba24f0f9eff395e2fabd102.19132.1689688332000000.zst (present)
  Size on Disk: 230.0K
       Message: Process 19132 (nm-openvpn-serv) of user 0 dumped core.
                
                Stack trace of thread 19132:
                #0  0x00007fdea34626ad __strlen_avx2 (libc.so.6 + 0x1686ad)
                #1  0x000055d4b85439ce n/a (nm-openvpn-service + 0x59ce)
                #2  0x000055d4b8543c7f n/a (nm-openvpn-service + 0x5c7f)
                #3  0x000055d4b854400b n/a (nm-openvpn-service + 0x600b)
                #4  0x000055d4b8544792 n/a (nm-openvpn-service + 0x6792)
                #5  0x00007fdea3554988 g_main_context_dispatch (libglib-2.0.so.0 + 0x5d988)
                #6  0x00007fdea3554d98 n/a (libglib-2.0.so.0 + 0x5dd98)
                #7  0x00007fdea355504f g_main_loop_run (libglib-2.0.so.0 + 0x5e04f)
                #8  0x000055d4b8542bad n/a (nm-openvpn-service + 0x4bad)
                #9  0x00007fdea3321bf0 __libc_start_call_main (libc.so.6 + 0x27bf0)
                #10 0x00007fdea3321cb9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27cb9)
                #11 0x000055d4b8542e85 n/a (nm-openvpn-service + 0x4e85)
                
                Stack trace of thread 19133:
                #0  0x00007fdea340741d syscall (libc.so.6 + 0x10d41d)
                #1  0x00007fdea35ae480 g_cond_wait (libglib-2.0.so.0 + 0xb7480)
                #2  0x00007fdea351ef4b n/a (libglib-2.0.so.0 + 0x27f4b)
                #3  0x00007fdea3581602 n/a (libglib-2.0.so.0 + 0x8a602)
                #4  0x00007fdea3580fbe n/a (libglib-2.0.so.0 + 0x89fbe)
                #5  0x00007fdea3387c64 start_thread (libc.so.6 + 0x8dc64)
                #6  0x00007fdea340f550 __clone3 (libc.so.6 + 0x115550)
                
                Stack trace of thread 19135:
                #0  0x00007fdea340148f __poll (libc.so.6 + 0x10748f)
                #1  0x00007fdea3554d0e n/a (libglib-2.0.so.0 + 0x5dd0e)
                #2  0x00007fdea355504f g_main_loop_run (libglib-2.0.so.0 + 0x5e04f)
                #3  0x00007fdea37c38c6 n/a (libgio-2.0.so.0 + 0x1228c6)
                #4  0x00007fdea3580fbe n/a (libglib-2.0.so.0 + 0x89fbe)
                #5  0x00007fdea3387c64 start_thread (libc.so.6 + 0x8dc64)
                #6  0x00007fdea340f550 __clone3 (libc.so.6 + 0x115550)
                
                Stack trace of thread 19134:
                #0  0x00007fdea340148f __poll (libc.so.6 + 0x10748f)
                #1  0x00007fdea3554d0e n/a (libglib-2.0.so.0 + 0x5dd0e)
                #2  0x00007fdea3554e2c g_main_context_iteration (libglib-2.0.so.0 + 0x5de2c)
                #3  0x00007fdea3554e71 n/a (libglib-2.0.so.0 + 0x5de71)
                #4  0x00007fdea3580fbe n/a (libglib-2.0.so.0 + 0x89fbe)
                #5  0x00007fdea3387c64 start_thread (libc.so.6 + 0x8dc64)
                #6  0x00007fdea340f550 __clone3 (libc.so.6 + 0x115550)
                ELF object binary architecture: AMD x86-64

GNU gdb (GDB; openSUSE Tumbleweed) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/libexec/nm-openvpn-service...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.opensuse.org/>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/libexec/nm-openvpn-service
Reading symbols from /root/.cache/debuginfod_client/6a8d3fb70460b27acf1a74f29eb546d74cc34f10/debuginfo...                                                                                 
[New LWP 19132]                                                                                                                                                                           
[New LWP 19133]
[New LWP 19135]
[New LWP 19134]
Downloading separate debug info for /lib64/libgio-2.0.so.0                                                                                                                                
Downloading separate debug info for /lib64/libgobject-2.0.so.0                                                                                                                            
Downloading separate debug info for /lib64/libglib-2.0.so.0                                                                                                                               
Downloading separate debug info for /lib64/libc.so.6                                                                                                                                      
Downloading separate debug info for /lib64/libgmodule-2.0.so.0                                                                                                                            
Downloading separate debug info for /lib64/ld-linux-x86-64.so.2                                                                                                                           
Downloading separate debug info for /lib64/libmount.so.1                                                                                                                                  
                                                                                                                                                                                          --Type <RET> for more, q to quit, c to continue without paging--c
Downloading separate debug info for /lib64/libblkid.so.1                                                                                                                                  
Downloading separate debug info for /lib64/libm.so.6                                                                                                                                      
Downloading separate debug info for /lib64/libnss_compat.so.2                                                                                                                             
Downloading separate debug info for system-supplied DSO at 0x7ffd4bb14000                                                                                                                 
[Thread debugging using libthread_db enabled]                                                                                                                                             
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/nm-openvpn-service --bus-name org.freedesktop.NetworkManager.openv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
Downloading source file /usr/src/debug/glibc-2.37/string/../sysdeps/x86_64/multiarch/strlen-avx2.S
76		VPCMPEQ	(%rdi), %ymm0, %ymm1                                                                                                                                              
[Current thread is 1 (Thread 0x7fdea2f50080 (LWP 19132))]
(gdb) t a a bt

Thread 4 (Thread 0x7fdea23fe6c0 (LWP 19134)):
#0  0x00007fdea340148f in __GI___poll (fds=0x55d4b966ed10, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fdea3554d0e in g_main_context_poll (priority=<optimized out>, n_fds=1, fds=0x55d4b966ed10, timeout=<optimized out>, context=0x55d4b966eaa0) at ../glib/gmain.c:4584
#2  g_main_context_iterate (context=context@entry=0x55d4b966eaa0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4271
#3  0x00007fdea3554e2c in g_main_context_iteration (context=0x55d4b966eaa0, may_block=may_block@entry=1) at ../glib/gmain.c:4343
#4  0x00007fdea3554e71 in glib_worker_main (data=<optimized out>) at ../glib/gmain.c:6455
#5  0x00007fdea3580fbe in g_thread_proxy (data=0x55d4b966ecb0) at ../glib/gthread.c:831
#6  0x00007fdea3387c64 in start_thread (arg=<optimized out>) at pthread_create.c:444
#7  0x00007fdea340f550 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7fdea1bfd6c0 (LWP 19135)):
#0  0x00007fdea340148f in __GI___poll (fds=0x7fde98000b90, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fdea3554d0e in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7fde98000b90, timeout=<optimized out>, context=0x55d4b967aef0) at ../glib/gmain.c:4584
#2  g_main_context_iterate (context=0x55d4b967aef0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4271
#3  0x00007fdea355504f in g_main_loop_run (loop=0x55d4b967b020) at ../glib/gmain.c:4479
#4  0x00007fdea37c38c6 in gdbus_shared_thread_func (user_data=0x55d4b966c7d0) at ../gio/gdbusprivate.c:284
#5  0x00007fdea3580fbe in g_thread_proxy (data=0x55d4b9676c70) at ../glib/gthread.c:831
#6  0x00007fdea3387c64 in start_thread (arg=<optimized out>) at pthread_create.c:444
#7  0x00007fdea340f550 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fdea2bff6c0 (LWP 19133)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fdea35ae480 in g_cond_wait (cond=cond@entry=0x55d4b966e7c8, mutex=mutex@entry=0x55d4b966e7c0) at ../glib/gthread-posix.c:1475
#2  0x00007fdea351ef4b in g_async_queue_pop_intern_unlocked (queue=0x55d4b966e7c0, wait=1, end_time=-1) at ../glib/gasyncqueue.c:425
#3  0x00007fdea3581602 in g_thread_pool_spawn_thread (data=<optimized out>) at ../glib/gthreadpool.c:311
#4  0x00007fdea3580fbe in g_thread_proxy (data=0x55d4b966a270) at ../glib/gthread.c:831
#5  0x00007fdea3387c64 in start_thread (arg=<optimized out>) at pthread_create.c:444
#6  0x00007fdea340f550 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7fdea2f50080 (LWP 19132)):
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1  0x000055d4b85439ce in write_user_pass (channel=0x7fde9800a790, authtype=authtype@entry=0x55d4b96adab0 "Auth", user=user@entry=0x55d4b96aee00 "mcepl", pass=<optimized out>) at src/nm-openvpn-service.c:862
#2  0x000055d4b8543c7f in handle_auth (io_data=0x55d4b96aeda0, requested_auth=requested_auth@entry=0x55d4b96adab0 "Auth", out_message=out_message@entry=0x7ffd4ba8fe40, out_hints=out_hints@entry=0x7ffd4ba8fe50) at src/nm-openvpn-service.c:908
#3  0x000055d4b854400b in handle_management_socket (plugin=plugin@entry=0x55d4b966cb90, source=<optimized out>, condition=<optimized out>, out_failure=out_failure@entry=0x7ffd4ba8feb4) at src/nm-openvpn-service.c:1021
#4  0x000055d4b8544792 in nm_openvpn_socket_data_cb (source=<optimized out>, condition=<optimized out>, user_data=0x55d4b966cb90) at src/nm-openvpn-service.c:1089
#5  0x00007fdea3554988 in g_main_dispatch (context=0x55d4b9675550) at ../glib/gmain.c:3460
#6  g_main_context_dispatch (context=context@entry=0x55d4b9675550) at ../glib/gmain.c:4200
#7  0x00007fdea3554d98 in g_main_context_iterate (context=0x55d4b9675550, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4276
#8  0x00007fdea355504f in g_main_loop_run (loop=loop@entry=0x55d4b967b8e0) at ../glib/gmain.c:4479
#9  0x000055d4b8542bad in main (argc=<optimized out>, argv=<optimized out>) at src/nm-openvpn-service.c:2435
(gdb) quit
stitny:~#
Comment 7 Matej Cepl 2023-07-18 13:57:12 UTC
And of course, even I had a mistake in configuration, it does not give right to anything to crash on me.
Comment 8 Fabian Vogt 2023-07-18 14:57:25 UTC
I imported the config here, changed the values over to my VPN config and it works. If I set the password as empty string ("password="), I can trigger the nullptr deref crash as expected. Question is why the password is empty at that point. Maybe it's not read correctly from the config. Can you check with:

nmcli -s --fields vpn.secrets con s mcepl.SUSE-UDP-PRG-NM

Otherwise it might be some weird condition with agent kept secrets leading to a conflict.
Comment 9 Matej Cepl 2023-07-29 21:54:33 UTC
Damn, no change in NetworkManager-openvpn (still 1.10.2-1.3) and it suddenly works!

Let's close it now as unreproducible, I will reopen if it happens again.