Bug 1217309 - [RPi4] NetworkManager fails to start afer reboot through rebootmgr
Summary: [RPi4] NetworkManager fails to start afer reboot through rebootmgr
Status: RESOLVED NORESPONSE
Alias: None
Product: openSUSE Leap Micro
Classification: openSUSE
Component: Base (show other bugs)
Version: 5.5
Hardware: Other Other
: P5 - None : Normal
Target Milestone: ---
Assignee: Jonathan Kang
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-11-19 15:14 UTC by Matthias Brugger
Modified: 2024-07-05 04:00 UTC (History)
5 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
reboot log (139.63 KB, text/plain)
2023-11-22 08:17 UTC, Matthias Brugger
Details
Boot log, no wifi (774.34 KB, text/plain)
2023-12-01 13:08 UTC, Matthias Brugger
Details
nmcli connection error (151.03 KB, text/plain)
2023-12-01 13:09 UTC, Matthias Brugger
Details
wpa_supplicant log file (586.31 KB, text/x-log)
2023-12-04 14:58 UTC, Matthias Brugger
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Brugger 2023-11-19 15:14:03 UTC
When trying to reboot the system, system services is shut down but no reboot happens. That leaves the system in a state without networking.

> printi:~ # rebootmgrctl get-strategy
> Reboot strategy: best-effort
> printi:~ # rebootmgrctl get-window
> Maintenance window is set to *-*-* 03:30:00, lasting 01h30m.
> printi:~ # rebootmgrctl is-active
> RebootMgr is active

journalctl output can be found here:
https://paste.opensuse.org/pastes/c0c9f158cadd

Interestingly running
> rebootmgrctl reboot now

does actually reboot the system.
Comment 1 Matthias Brugger 2023-11-19 22:22:42 UTC
I found this on the firstboot log:

> setroubleshoot[1921]: SELinux is preventing /usr/sbin/rebootmgrd from read access on the file cpuinfo. For complete SELinux messages run: sealert -l 054a40c1-c97a-4717-a8b9-0e814935fcc7
> systemd[1]: Starting Socket for Cockpit Web Service http instance...
> systemd[1]: Starting Socket for Cockpit Web Service https instance factory...
> systemd[1]: Listening on Socket for Cockpit Web Service http instance.
> systemd[1]: Listening on Socket for Cockpit Web Service https instance factory.
> systemd[1]: Starting Cockpit Web Service...
> kernel: BTRFS info (device mmcblk0p2): balance: start -musage=3 -susage=3
> kernel: BTRFS info (device mmcblk0p2): relocating block group 2562654208 flags metadata|dup
> setroubleshoot[1921]: SELinux is preventing /usr/sbin/rebootmgrd from read access on the file cpuinfo.
>                                                             
>                                                             *****  Plugin catchall (100. confidence) suggests   **************************
>                                                             
>                                                             If you believe that rebootmgrd should be allowed read access on the cpuinfo file by default.
>                                                             Then you should report this as a bug.
>                                                             You can generate a local policy module to allow this access.
>                                                             Do
>                                                             allow this access for now by executing:
>                                                             # ausearch -c 'rebootmgrd' --raw | audit2allow -M my-rebootmgrd
>                                                             # semodule -X 300 -i my-rebootmgrd.pp
>                                                             
> cockpit-certificate-ensure[2620]: /usr/lib/cockpit-certificate-helper: line 25: sscg: command not found
Comment 2 Guillaume GARDET 2023-11-20 11:02:06 UTC
Are you able to reboot without SELinux ?
Comment 3 Matthias Brugger 2023-11-21 13:27:36 UTC
That's my state now, let's see if tomorrow morning it was able to properly reboot:

> printi:~ # rebootmgrctl status
> Status: Reboot requested, waiting for maintenance window
> printi:~ # rebootmgrctl get-window
> Maintenance window is set to *-*-* 03:30:00, lasting 01h30m.
> printi:~ # getenforce 
> Permissive
Comment 4 Matthias Brugger 2023-11-22 08:17:35 UTC
Created attachment 870894 [details]
reboot log

I can now see that the system rebooted, maybe it even rebooted with SELinux enabled, I would need to check. The problem is, that it does not get wifi connection and therefor the clock + date is wrong.

I attach the journal for the boot (-b -0) as well as what gets written to the journal after running
> printi:~ # systemctl restart NetworkManager

You can see after reboot the system is not able to start NetworkManager
> Aug 16 12:01:12 printi systemd[1]: Failed to start Network Manager Wait Online.

After starting it from hand I have my expected access to the network.

Guillaume any idea why this could happen?
Comment 5 Guillaume GARDET 2023-11-22 08:59:43 UTC
(In reply to Matthias Brugger from comment #4)
> Created attachment 870894 [details]
> reboot log
> 
> I can now see that the system rebooted, maybe it even rebooted with SELinux
> enabled, I would need to check. The problem is, that it does not get wifi
> connection and therefor the clock + date is wrong.

It rebooted with SELinux enabled and journal still shows error:
**************************************************
Aug 16 12:00:52 printi setroubleshoot[853]: SELinux is preventing /usr/sbin/rebootmgrd from open access on the file /proc/cpuinfo. For complete SELinux messages run: sealert -l e22cee2c-725a-4bd3-b092-65a27cb7c205
Aug 16 12:00:52 printi setroubleshoot[853]: SELinux is preventing /usr/sbin/rebootmgrd from open access on the file /proc/cpuinfo.
                                            
                                            *****  Plugin catchall (100. confidence) suggests   **************************
                                            
                                            If you believe that rebootmgrd should be allowed open access on the cpuinfo file by default.
                                            Then you should report this as a bug.
                                            You can generate a local policy module to allow this access.
                                            Do
                                            allow this access for now by executing:
                                            # ausearch -c 'rebootmgrd' --raw | audit2allow -M my-rebootmgrd
                                            # semodule -X 300 -i my-rebootmgrd.pp
**************************************************

So, please open a bug report upstream for this one.

> 
> I attach the journal for the boot (-b -0) as well as what gets written to
> the journal after running
> > printi:~ # systemctl restart NetworkManager
> 
> You can see after reboot the system is not able to start NetworkManager
> > Aug 16 12:01:12 printi systemd[1]: Failed to start Network Manager Wait Online.
> 
> After starting it from hand I have my expected access to the network.

There are lots of MAC addresses settings on boot, which is weird.
Maybe ask NetworkManager folks?
Comment 6 Yifan Jiang 2023-11-22 09:44:24 UTC
Hi Jonathan, I remember NM may throw out useful logs for the stop/start cycle, can you help on driving this further please? Thanks.
Comment 7 Jonathan Kang 2023-11-30 02:19:02 UTC
(In reply to Matthias Brugger from comment #4)
> Created attachment 870894 [details]
> reboot log
> 
> I can now see that the system rebooted, maybe it even rebooted with SELinux
> enabled, I would need to check. The problem is, that it does not get wifi
> connection and therefor the clock + date is wrong.
> 
> I attach the journal for the boot (-b -0) as well as what gets written to
> the journal after running
> > printi:~ # systemctl restart NetworkManager
> 
> You can see after reboot the system is not able to start NetworkManager
> > Aug 16 12:01:12 printi systemd[1]: Failed to start Network Manager Wait Online.
> 
> After starting it from hand I have my expected access to the network.
> 
> Guillaume any idea why this could happen?

The log shows that the wifi connection cannot be activated. But the default NetworkManager log level doesn't give enough useful information about why.

Can you add the following to /etc/NetworkManager/NetworkManager.conf, reboot, reproduce this issue and attach reboot log here?

> [logging]
> level=trace

Thanks.
Comment 8 Matthias Brugger 2023-12-01 13:08:11 UTC
Created attachment 871104 [details]
Boot log, no wifi

I observed that NM had problems to connect to my wifi and I had to restart it various times to get it connected.

I attach the bootlog.
Comment 9 Matthias Brugger 2023-12-01 13:09:17 UTC
Created attachment 871105 [details]
nmcli connection error

When running
> # nmcli connection up MIWIFI_TU6P 
> Error: Connection activation failed: Secrets were required, but not provided
Hint: use 'journalctl -xe NM_CONNECTION=a29966cf-55ac-4fd2-b282-55b7a4f7cce2 + NM_DEVICE=wlan0' to get more details.

I attach this log as well.
Comment 10 Matthias Brugger 2023-12-01 13:13:31 UTC
> # cat /etc/NetworkManager/system-connections/MIWIFI_TU6P-a29966cf-55ac-4fd2-b282-55b7a4f7cce2.nmconnection 
[connection]
id=MIWIFI_TU6P
uuid=a29966cf-55ac-4fd2-b282-55b7a4f7cce2
type=wifi
interface-name=wlan0
timestamp=1701433559

[wifi]
mode=infrastructure
ssid=MIWIFI_TU6P

[wifi-security]
auth-alg=open
key-mgmt=wpa-psk
psk=verysecretpassword

[ipv4]
method=auto

[ipv6]
addr-gen-mode=stable-privacy
method=auto

[proxy]
Comment 11 Matthias Brugger 2023-12-01 15:03:27 UTC
the only strange message I see is:

request-scan: could not get scan request result: GDBus.Error:fi.w1.wpa_supplicant1.Interface.ScanError: Scan request rejected

Although restarting wpa_supplicant.service does not fix the issue.
Comment 12 Jonathan Kang 2023-12-04 08:24:41 UTC
The logs indicate that wpa_supplicant failed to do the association.

> Dec 01 13:02:33 printi NetworkManager[814]: <warn>  [1701435753.0529] device (wlan0): Activation: (wifi) association took too long

But there is no wpa_supplicant log. They are stored inside /var/log/wpa_supplicant.log. Can you attach that file here?
Comment 13 Matthias Brugger 2023-12-04 14:58:40 UTC
Created attachment 871141 [details]
wpa_supplicant log file
Comment 14 Jonathan Kang 2023-12-05 09:27:45 UTC
(In reply to Matthias Brugger from comment #13)
> Created attachment 871141 [details]
> wpa_supplicant log file

Can you change the "ExecStart" line in /usr/lib/systemd/system/wpa_supplicant.service to

> ExecStart=/usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -u -d -t

restart the system, reproduce this issue and attach the output of "journalctl -b" here?

This enables verbose logging in wpa_supplicant directly in journalctl.
Comment 15 Jonathan Kang 2024-07-05 04:00:12 UTC
Closing this as RESOLVED NORESPONSE.

Feel free to re-open it with requested information.