Bugzilla – Bug 1210247
ssh cannot assign to requested static address, configured via nmcli
Last modified: 2023-09-21 13:28:13 UTC
On a fresh MicroOS installation, sshd fails to start when attempting to bind to a specific IPv6 address, which is assigned to Networkmanager. The /etc/ssh/sshd_config.d/local.conf file looks like the following > AddressFamily inet6 > ListenAddress <redacted> This is a dedicated management address, that has been assigned to NetworkManager via > nmcli con mod enp1s0 +ipv6.addresses "<redacted>" Of course, restarting sshd after booting resolves the issue, the problem is that it appears, that there is a race condition between sshd and NetworkManager that doesn't allow sshd to listen to a static address, which has been assigned to NetworkManager.
Hello, this issue is still present. I just checked it via the following minimal reproducer: ## Reproducer > nmcli con mod enp1s0 +ipv6.addresses fee8::8 > echo -e 'AddressFamily inet6\nListenAddress fee8::8' > /etc/ssh/sshd_config.d/bind.conf > reboot This assigns the `fee8::8` to the local network interface, and the second one makes ssh bind to this address. Upon reboot, sshd.service fails to start: > × sshd.service - OpenSSH Daemon > Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; preset: disabled) > Active: failed (Result: exit-code) since Mon 2023-05-08 13:23:49 CEST; 17s ago > Process: 773 ExecStartPre=/usr/sbin/sshd-gen-keys-start (code=exited, status=0/SUCCESS) > Process: 777 ExecStartPre=/usr/sbin/sshd -t $SSHD_OPTS (code=exited, status=0/SUCCESS) > Process: 778 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION) > Main PID: 778 (code=exited, status=255/EXCEPTION) > CPU: 33ms > > May 08 13:23:49 tumbleweed systemd[1]: Starting OpenSSH Daemon... > May 08 13:23:49 tumbleweed sshd-gen-keys-start[773]: Checking for missing server keys in /etc/ssh > May 08 13:23:49 tumbleweed sshd[778]: error: Bind to port 22 on fee8::8 failed: Cannot assign requested address. > May 08 13:23:49 tumbleweed sshd[778]: fatal: Cannot bind any address. > May 08 13:23:49 tumbleweed systemd[1]: sshd.service: Main process exited, code=exited, status=255/EXCEPTION > May 08 13:23:49 tumbleweed systemd[1]: sshd.service: Failed with result 'exit-code'. > May 08 13:23:49 tumbleweed systemd[1]: Failed to start OpenSSH Daemon. I tried 5 reboots, sshd.service fails to start in 3/5, at least in my VM. The IP address is however assigned, also if sshd.service fails: > 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 > link/ether 52:54:00:95:ba:ef brd ff:ff:ff:ff:ff:ff > inet 192.168.122.77/24 brd 192.168.122.255 scope global dynamic noprefixroute enp1s0 > valid_lft 3543sec preferred_lft 3543sec > inet6 fee8::8/128 scope site noprefixroute > valid_lft forever preferred_lft forever > inet6 fee8::2c8e:b78a:85a7:d921/64 scope site temporary dynamic > valid_lft 604744sec preferred_lft 85817sec > inet6 fee8::5054:ff:fe95:baef/64 scope site mngtmpaddr noprefixroute > valid_lft forever preferred_lft forever > inet6 fe80::5054:ff:fe95:baef/64 scope link noprefixroute > valid_lft forever preferred_lft forever Still looks like a race condition to me.
To summarize https://suse.slack.com/archives/C028VS8TM2B/p1683545556951149 there are two things necessary to allow `sshd.service` to work well with a statically assigned IPv6 address 1. `ipv6.may-fail` needs to be set to false > nmcli con mod enp1s0 ipv6.may-fail false 2. The sshd.service file needs to be edited, and After=network-online.target + Wants=network-online.target be added to the unit section With both of those effects being set, 5/5 reboots work nicely. With one of those steps not being in place, about 3/5 fail.
The discussion continues for a bit in the referenced conversation. Points that I consider important to mention: The issue is strongly related to the specific selection of a non-Loopback address in `ListenAddress fee8::8`. All https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ and https://unix.stackexchange.com/a/442199 and https://unix.stackexchange.com/a/511976 provide good explanations and ideas how to handle: If you try to bind to a specific non-loopback address and not * or 0.0.0.0 then you will run into this problem. Two ideas mentioned that I think are valid: 1. Specify both your address and loopback so that sshd can at least bind on loopback and restart eventually on the second address 2. Use IP_FREEBIND, e.g. `echo 'net.ipv4.ip_nonlocal_bind = 1' > /etc/sysctl.d/99-nonlocal_bind.conf` to allow binding to remote addresses even when they are not there (yet). I don't consider those workarounds but intrications of how the handling of a network stack is designed. If you configure ListenAddress to be a single specific address and that address is not available then sshd will fail with a fatal error telling you about it. So IMHO I consider it a an acceptable setup that in this configuration you can configured ListenAddress both on your selected address and loopback and sshd should retry until it can listen on all. Regarding "network-online.target": Also upstream systemd suggests to not rely on network-online but instead design applications properly to handle if the network is not up yet. There was one more important point my mwilck: If NetworkManager-wait-online.service is configured correctly and still the issue occurs, an upstream issue may actually be justified.
The issue also arises, when NetworkManager-wait-online.service is enabled, and both `ipv6.may-fail=false` and `ipv4.may-fail=false` are applied for the device in question. In my test VM, 5/5 reboots fail, as shown below. Here I bind ssh to the IPv4 and IPv6 address. IPv4 is assigned by DHCP, IPv6 statically via `nmcli con mod enp1s0 +ipv6.addresses fee8::8`. > May 09 14:35:34 tumbleweed sshd[778]: error: Bind to port 22 on 192.168.122.77 failed: Cannot assign requested address. > May 09 14:35:34 tumbleweed sshd[778]: error: Bind to port 22 on fee8::8 failed: Cannot assign requested address. > May 09 14:35:34 tumbleweed sshd[778]: fatal: Cannot bind any address. *** I filed https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1292 for this issue.
Three notes here, from https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1292#note_1901889 1. Using `After=network-online.target` instead of `After=network.target` in sshd.service did not help 2. Fedora 38 is having the same issue 3. On Fedora the issue is mitigated by ssh being restarted after 42s: > # sshd.service from Fedora 38 > [Service] > Restart=on-failure > RestartSec=42 So while the same issue occurs there, the impact is limited to a longer time ssh needs, because if it fails on the first attempt it will succeed on the second one. This was a mitigation that also Oliver suggested in the Slack thread.
(In reply to Felix Niederwanger from comment #5) > This was a mitigation that also Oliver suggested in the Slack thread. that's true however configuring ``` ListenAddress ::1 ListenAddress fee8::8 ``` should handle the restarting within sshd, not relying on sshd so less error messages in systemd, less relying on a specific service handler if you ever want to run sshd in a container without systemd or something ;)
> > ``` > ListenAddress ::1 > ListenAddress fee8::8 > ``` > > should handle the restarting within sshd, not relying on sshd so less error > messages in systemd, less relying on a specific service handler if you ever > want to run sshd in a container without systemd or something ;) That makes the service start successfully but because it couldn't bind to fee8::8 at startup, ssh is never going to listen on that interface. In this configuration sshd listens to ::1 but not on fee8::8 so the underlying issue still remains.
This is not an upstream bug in NetworkManager, see https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1292#note_1901913 The overall outcome is, that in the default configuration, ssh will fail to start/bind, if ListenAddress is specified. Even with a single IPv4 address, assigned via DHCP from libvirt, sshd.service fails with > error: Bind to port 22 on 192.168.122.77 failed: Cannot assign requested address * This applies to IPv4 and IPv6 addresses, both static and dynamic. * Enabling the `NetworkManager-wait-online` service doesn't help. * Adding `After=network-online.target` doesn't help either. The same issue is present on Fedora 38, but there the change of adding `After=network-online.target` to the sshd service helped (5/5 successful reboots). on openSUSE Tumbleweed the issue is still present after those changed (0/5 successful reboots). It would be great if anyone with deeper NetworkManager knowledge can have a look at this. I consider it a bug that on a server system a user cannot simply add `ListenAddress` to it's sshd configuration without locking themselves out.
(In reply to Oliver Kurz from comment #3) > The discussion continues for a bit in the referenced conversation. Points > that I consider important to mention: The issue is strongly related to the > specific selection of a non-Loopback address in `ListenAddress fee8::8`. All > https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ and > https://unix.stackexchange.com/a/442199 and > https://unix.stackexchange.com/a/511976 provide good explanations and ideas > how to handle: If you try to bind to a specific non-loopback address and not > * or 0.0.0.0 then you will run into this problem. Two ideas mentioned that I > think are valid: 1. Specify both your address and loopback so that sshd can > at least bind on loopback and restart eventually on the second address 2. > Use IP_FREEBIND, e.g. `echo 'net.ipv4.ip_nonlocal_bind = 1' > > /etc/sysctl.d/99-nonlocal_bind.conf` to allow binding to remote addresses > even when they are not there (yet). I don't consider those workarounds but > intrications of how the handling of a network stack is designed. If you > configure ListenAddress to be a single specific address and that address is > not available then sshd will fail with a fatal error telling you about it. > So IMHO I consider it a an acceptable setup that in this configuration you > can configured ListenAddress both on your selected address and loopback and > sshd should retry until it can listen on all. Regarding > "network-online.target": Also upstream systemd suggests to not rely on > network-online but instead design applications properly to handle if the > network is not up yet. > > There was one more important point my mwilck: If > NetworkManager-wait-online.service is configured correctly and still the > issue occurs, an upstream issue may actually be justified. https://systemd.io/NETWORK_ONLINE/ Upstream does not suggest not relying on it. What it says is - "It is strongly recommended not to make use of this target too liberally: for example network server software should generally not pull this in (since server software generally is happy to accept local connections even before any routable network interface is up). Its primary purpose is network client software that cannot operate without network." The standard use case for sshd is to provide secure *remote* shell access. This isn't something like PostgreSQL where it's perfectly normal to have just local access. Just like the example usage in the documentation (remote mount points) it does not make sense to have sshd start before the network is completely configured and ready. Copying sshd.service to /etc/systemd/system and adding `After=network-online.target` to sshd.service did solve the issue for me on Tumbleweed.
(In reply to mhurron from comment #9) > Copying sshd.service to /etc/systemd/system and adding > `After=network-online.target` to sshd.service did solve the issue for me on > Tumbleweed. I tested this before and now again on a fresh Tumbleweed VM (from the DVD install), and 10/10 times the sshd daemon could not start (5x IPv4 + 5x IPv6 address used, see below). As before I just copied the sshd.service file from /usr/lib/systemd/system/sshd.service to /etc/systemd/system/ and replaced the After= section, just as mhurron states in comment #9 > network6:~ # cat /etc/systemd/system/sshd.service > [Unit] > Description=OpenSSH Daemon > After=network-online.target > > [Service] > Type=notify > EnvironmentFile=-/etc/sysconfig/ssh > ExecStartPre=/usr/sbin/sshd-gen-keys-start > ExecStartPre=/usr/sbin/sshd -t $SSHD_OPTS > ExecStart=/usr/sbin/sshd -D $SSHD_OPTS > ExecReload=/bin/kill -HUP $MAINPID > KillMode=process > Restart=on-failure > RestartPreventExitStatus=255 > TasksMax=infinity > > [Install] > WantedBy=multi-user.target And the ssh configuration in /etc/ssh/sshd_config > # cat /etc/ssh/sshd_config.d/local.conf > ## I tested it either with inet6 or with inet > #AddressFamily inet6 > #ListenAddress fee8::5054:ff:fed9:af62 > AddressFamily inet > ListenAddress 192.168.122.45 > > PermitRootLogin prohibit-password As said, 5/5 times the sshd daemon would not start, neither with the assigned IPv6 nor with the IPv4 address. I also checked if a static IPv6 address makes a difference and even there 5/5 times the sshd daemon fails to start. > nmcli con mod enp1s0 +ipv6.addresses fee8::8 > nmcli con mod enp1s0 ipv6.may-fail false > vim /etc/ssh/sshd_config.d/local.conf # Replace the IPv6 address above with fee8::8 I cannot see that this issue is resolved by using `After=network-online.target`.