Bug 774502

Summary: NTP & NFS mount fail before network is up
Product: [openSUSE] openSUSE Tumbleweed Reporter: Bob Ewart <shaggyman>
Component: BasesystemAssignee: Michal Filka <mfilka>
Status: NEW --- QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: axel.braun, fcrozat, jreidinger, mcatanzaro, mfilka, mt, ncutler
Version: Current   
Target Milestone: Current   
Hardware: x86-64   
OS: openSUSE 12.2   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: dmesg output
/var/log/messages
output of supportconfig

Description Bob Ewart 2012-08-03 22:25:57 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1

NTP is configured to sync on boot-up.
There are several NFS mounts in /etc/fstab, one with a server name and the other with an ip address to the same server.

All fail before the network is fully up.  

I've tried with the system defaulting to getting it's address from DHCP and specifying the ip address manually.  

Reproducible: Always

Steps to Reproduce:
1.Configure NTP to start on bootup
2.Add NFS mounts via YAST (or manually to /etc/fstab)
3.reboot
Actual Results:  
NTP failed to access the public server.
None of the NFS mounts happened.


Expected Results:  
NTP should have set the system clock and the NFS directories should have been mounted
Comment 1 Peter Varkoly 2012-08-10 10:20:23 UTC
I can not see why is it an ntp problem?
Comment 2 Bob Ewart 2012-08-10 15:00:22 UTC
NTP is started before the network is completely up.  So it fails on trying to access a public server.
Comment 3 Peter Varkoly 2012-08-15 08:00:09 UTC
IMHO it is a bug of systemd if it starts a service which needs network befor starting the network.
Comment 4 Frederic Crozat 2012-08-16 08:26:05 UTC
systemd is supposed to wait for $network to be available before starting ntp.

please boot with systemd.log_level=debug systemd.log_target=kmsg and attach dmesg output to this bug report.
Comment 5 Bob Ewart 2012-08-16 18:28:07 UTC
Created attachment 502587 [details]
dmesg output
Comment 6 Bob Ewart 2012-08-16 18:28:51 UTC
Created attachment 502588 [details]
/var/log/messages
Comment 7 Bob Ewart 2012-08-16 18:38:11 UTC
I've added the dmesg output as requested and the contents of /var/log/messages.

In the messages file at line:

 925 r8169 driver is loaded

1022 eth0 is set up
1041

1053 NFS client starts
1060  NFS can't resolve server name

1085 sntp started
1087  can't resolve remote ntp server

1135 dhcpd starting
1154 address and route set up

It looks like dhcpd needs to complete before nfs or ntp can access the network and resolve remote servers.
Comment 8 Frederic Crozat 2012-08-17 12:38:29 UTC
from what I see, your system is configured to use ifup/ifdown (not NetworkManager) to configure your network, and network.service is being started after 3s, so it should wait until network is up.

not related to systemd, I suggest you look at outpout from systemctl status network.service (or systemd-journalctl _SYSTEMD_UNIT=network.service)
Comment 9 Marius Tomaschewski 2012-08-17 14:56:48 UTC
(In reply to comment #8)
> from what I see, your system is configured to use ifup/ifdown (not
> NetworkManager) to configure your network, and network.service is being
> started after 3s, so it should wait until network is up.

Sorry, but I don't understand what last commenter wanted to say here :-)

AFAIS, the start order is OK but you're using ifplugd. When you're using
it, change to start all network related services via network hooks, see
"man ifservices". In ifplugd mode, rcnetwork returns "done" as soon as
possible and starts the interface in the background...

Please disable it -- change /etc/sysconfig/network/ifcfg-eth0 to use:

STARTMODE=auto

When really required (I don't think so), set LINK_READY_WAIT="15" in
the same file.

Alternatively, you can also use STARTMODE=nfsroot -- it causes that
network will be persistent / a stop does not remove IP addresses.
Comment 10 Marius Tomaschewski 2012-08-17 15:41:44 UTC
Ahm... needinfo: does it work when you use STARTMODE=auto ?
Comment 11 Bob Ewart 2012-08-17 17:19:05 UTC
Yes it does.

I did a fresh install from openSUSE-KDE-LiveCD-Build0091-x86_64.iso

ifcfg-etho had
STARTMODE='ifplugd'

I added 2 nfs mounts via Yast.  The mount failed.  I tried to mount them as root and it complained that rpc.statd wasn't running.

I started rpcbind via YAST and could mount them.

When I set STARTMODE='auto' it worked.

Looks like the default STARTMODE is wrong
Comment 12 Marius Tomaschewski 2012-08-20 08:21:37 UTC
(In reply to comment #11)
> Yes it does.
> 
> I did a fresh install from openSUSE-KDE-LiveCD-Build0091-x86_64.iso
> 
> ifcfg-etho had
> STARTMODE='ifplugd'
> 
> I added 2 nfs mounts via Yast.  The mount failed.  I tried to mount them as
> root and it complained that rpc.statd wasn't running.
> 
> I started rpcbind via YAST and could mount them.
>
> When I set STARTMODE='auto' it worked.
> 
> Looks like the default STARTMODE is wrong

Well, the proposal yast2 makes depends on the view / focus.

For notebooks or "home desktops" (that are "focused" on openSUSE),
where things like nfs usually aren't in use/needed [and also the
rpcbind is disabled by default?], ifplugd is not really wrong:
  it starts and stops one or more interfaces (you can also set
  a priority, e.g. to prefer lan over wlan) on demand.

Independently of the above, this is a configuration issue and thus
I reassign it to yast2. Perhaps yast2 can be improved here.
Comment 13 Marius Tomaschewski 2012-08-20 08:28:42 UTC
(In reply to comment #9)
> Alternatively, you can also use STARTMODE=nfsroot -- it causes that
> network will be persistent / a stop does not remove IP addresses.

BTW / for completeness: This is mandatory for nfsroot, /usr on nfs and
other [iSCSI] scenarios where the filesystems or discs are started in
the initrd and accessed over the network and a "rcnetwork stop" should
not stop interfaces.
Comment 14 Michal Filka 2012-08-21 05:48:06 UTC
(In reply to comment #11)
> When I set STARTMODE='auto' it worked.
> 
> Looks like the default STARTMODE is wrong

there is several "defaults" in networking - depending on target system.

Just to be sure. Did you perform installation at laptop, rigth?
Comment 15 Bob Ewart 2012-08-21 15:29:00 UTC
No!

Standard desktop with:
Giga-byte GA-EP45-DS5 Motherboard with core i5 CPU @ 3.20 GHz
Intel 5 Series/3400 High Definition Audio
Realtek RTL8111/8168B PCI Express Gigabit Ethernet controller
4 GB memory
ATI graphics card
500 GB SATA drive on sda
1000 GB SATA drive on sdb
DVD on sr0

2 ViewSonic VA2431wm monitors at 1920x1080 (total 3840x1080)
Comment 16 Bob Ewart 2012-08-23 13:24:43 UTC
forgot to turn needinfo off
Comment 17 Michael Catanzaro 2013-01-12 17:34:23 UTC
This is the only error message I get when I start my computer:

Jan 12 10:35:41 victory-road ntp[683]: 12 Jan 10:35:41 sntp[746]: Error looking 
up bigben.cac.washington.edu: Name or service not known
Jan 12 10:35:41 victory-road ntp[683]: Unable to resolve hostname(s)
Jan 12 10:35:41 victory-road ntp[683]: Time could not be synchronized

I haven't changed any defaults, besides setting ntpd to use a public server rather than undisciplined local clock (terrible default). i.e. I am using NetworkManager. 

Is this the same bug?
Comment 18 Axel Braun 2013-01-16 17:45:38 UTC
I was just about to open a new bug on ntp, as I found this one.

I actually notice the same issue on a Laptop using WLAN and Networkmanager: After boot and login to KDE, I have to enter the password for kwallet before a network connection is up. ntp uses this time to die without further notice:
T520:/home/docb # rcntp status
redirecting to systemctl
ntp.service - LSB: Network time protocol daemon (ntpd)
          Loaded: loaded (/etc/init.d/ntp)
          Active: inactive (dead) since Tue, 15 Jan 2013 15:34:09 +0100; 1 day and 1h ago
         Process: 1498 ExecStop=/etc/init.d/ntp stop (code=exited, status=0/SUCCESS)
         Process: 804 ExecStart=/etc/init.d/ntp start (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/ntp.service

Jan 15 15:31:40 T520.axxite.internal ntpd[887]: proto: precision = 0.132 usec
Jan 15 15:31:40 T520.axxite.internal ntpd[887]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Jan 15 15:31:40 T520.axxite.internal ntpd[887]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Jan 15 15:31:40 T520.axxite.internal ntp[804]: Starting network time protocol daemon (NTPD)..done
Jan 15 15:31:40 T520.axxite.internal ntpd[887]: Listen and drop on 1 v6wildcard :: UDP 123
Jan 15 15:31:41 T520.axxite.internal ntpd[887]: Listen normally on 2 lo 127.0.0.1 UDP 123
Jan 15 15:31:41 T520.axxite.internal ntpd[887]: Listen normally on 3 lo ::1 UDP 123
Jan 15 15:31:41 T520.axxite.internal ntpd[887]: peers refreshed
Jan 15 15:31:41 T520.axxite.internal ntpd[887]: Listening on routing socket on fd #20 for interface updates
Jan 15 15:34:09 T520.axxite.internal ntp[1498]: Shutting down network time protocol daemon (NTPD)..done

After a restart, ntp works as expected (note that this is one day later!):
T520:/home/docb # rcntp restart
redirecting to systemctl
T520:/home/docb # rcntp status
redirecting to systemctl
ntp.service - LSB: Network time protocol daemon (ntpd)
          Loaded: loaded (/etc/init.d/ntp)
          Active: active (running) since Wed, 16 Jan 2013 18:20:26 +0100; 4s ago
         Process: 1498 ExecStop=/etc/init.d/ntp stop (code=exited, status=0/SUCCESS)
         Process: 8118 ExecStart=/etc/init.d/ntp start (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/ntp.service
                  └ 8150 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -i /var/lib/ntp -c /etc/ntp.conf

Jan 16 18:20:26 T520.axxite.internal ntp[8118]: Starting network time protocol daemon (NTPD)..done
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: Listen and drop on 1 v6wildcard :: UDP 123
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: Listen normally on 2 lo 127.0.0.1 UDP 123
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: Listen normally on 3 wlan0 192.168.1.16 UDP 123
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: Listen normally on 4 wlan0 fe80::120b:a9ff:feef:6c4c UDP 123
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: Listen normally on 5 lo ::1 UDP 123                                                                                    
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: peers refreshed                                                                                                        
Jan 16 18:20:26 T520.axxite.internal ntpd[8150]: Listening on routing socket on fd #22 for interface updates                   

I dont feel that this is a configuration issue, but correct me if I'm wrong. IMHO ntp should restart after each network connection (i.e. after resume from suspend)
Comment 19 Marius Tomaschewski 2013-01-17 10:13:15 UTC
(In reply to comment #18)
> I dont feel that this is a configuration issue, but correct me if I'm wrong.
> IMHO ntp should restart after each network connection (i.e. after resume from
> suspend)

Not any more:
- Fixed netconfig/ntp-runtime to not use try-restart as its start
  may block for a while, so just add servers ourself in background.
  When ntpd is not running (not yet started), the ntp init script
  will pickup our servers later at ntp start (bnc#780644).

In another words, when the network starts, and gets a lease including
an ntp server, 'echo "addserver xxx" | ntpdc' is used now to sync the
time and the ntp servers are written to /var/run/ntp/servers-netconfig.

When ntpd is already started [e.g. lease renew], ntpdc makes the sync,
otherwise the ntp init script will pick up the server list from the
/var/run/ntp/servers-netconfig file when it (re)starts.

So (at boot) first the network starts and does nothing with ntp (ntpdc
addserver fails). Then, the ntpd starts and the init script makes the
ntpdc addserver call.

Further, as I wrote in e.g. comment 9, do not use STARTMODE='ifplugd'
(On-Cable-Connect or something like this in yast2).

With STARTMODE='ifplugd', it is required to disable ntp & nfs start
(all services which can't be started without network up) at boot time
and use ifup(8) or ifservices(5) hooks to start them when network
goes up.

AFAIS, there is no dependency (Required-Start, Should-Start) to ntp
($time) in /etc/init.d/nfs at all and this is IMO the problem here.
Comment 20 Marius Tomaschewski 2013-01-17 10:21:08 UTC
When I see correctly, resume from suspend seems to be still broken,
at least when systemd is used. Going to try to fix it in bnc#781106.
But I don't think resume will ever work with nfs mounts.
Comment 21 Axel Braun 2013-01-17 13:14:57 UTC
(In reply to comment #19)
> (In reply to comment #18)
> > I dont feel that this is a configuration issue, but correct me if I'm wrong.
> > IMHO ntp should restart after each network connection (i.e. after resume from
> > suspend)
> 
> Not any more:
> - Fixed netconfig/ntp-runtime to not use try-restart as its start
>   may block for a while, so just add servers ourself in background.
>   When ntpd is not running (not yet started), the ntp init script
>   will pickup our servers later at ntp start (bnc#780644).

Hmm....this bug mentions 
openSUSE 12.2 (src):    sysconfig-0.76.4-1.4.1
as package where the solution is available. It is installed on my system, but still the ntp-bug exists...

Or is it
openSUSE 12.1 (src):    sysconfig-0.75.4-2.16.1
but not just for 12.1?
Comment 22 Marius Tomaschewski 2013-01-17 14:49:05 UTC
(In reply to comment #21)
> (In reply to comment #19)
> > (In reply to comment #18)
> > > I dont feel that this is a configuration issue, but correct me if I'm wrong.
> > > IMHO ntp should restart after each network connection (i.e. after resume from
> > > suspend)
> > 
> > Not any more:
> > - Fixed netconfig/ntp-runtime to not use try-restart as its start
> >   may block for a while, so just add servers ourself in background.
> >   When ntpd is not running (not yet started), the ntp init script
> >   will pickup our servers later at ntp start (bnc#780644).
> 
> Hmm....this bug mentions 
> openSUSE 12.2 (src):    sysconfig-0.76.4-1.4.1
> as package where the solution is available. It is installed on my system, but
> still the ntp-bug exists...

Seems, you didn't read the complete comment...

Above I just said, that ntp is not restarted any more on network
connection changes.

Further, I also said:

AFAIS, there is no dependency (Required-Start, Should-Start) to ntp
($time) in /etc/init.d/nfs at all and this is IMO the problem here.
Comment 23 Axel Braun 2013-01-17 16:33:28 UTC
(In reply to comment #22)

> Seems, you didn't read the complete comment...

Lets say, I didnt get it.... 
 
> Above I just said, that ntp is not restarted any more on network
> connection changes.
> 
> Further, I also said:
> 
> AFAIS, there is no dependency (Required-Start, Should-Start) to ntp
> ($time) in /etc/init.d/nfs at all and this is IMO the problem here.

 /etc/init.d/nfs ...relates to the classical startup sequence, no?
Sorry, I'm not an expert in systemd or init scripts, I just can see from the LSB-header if  /etc/init.d/nfs that it requires the network to be started. And that was nicely working in old times.
With systemd obviously not. So IMHO this is a problem in systemd, or am I wrong?
Comment 24 Marius Tomaschewski 2013-01-17 16:57:25 UTC
(In reply to comment #23)
>  /etc/init.d/nfs ...relates to the classical startup sequence, no?

Yes.

> Sorry, I'm not an expert in systemd or init scripts, I just can see from the
> LSB-header if  /etc/init.d/nfs that it requires the network to be started.

Network yes, but neither $time (ntp) nor $named.

And it will never work with STARTMODE='ifplugd' or NetworkManager, unless
you handle it yourself via ifservices(5) or NetworkManager dispatcher
scripts. => Use STARTMODE=auto or STARTMODE=nfsroot.

AFAIS it fails because name resolution (comment 7).
Does your /etc/resolv.conf contains the correct name server settings when
the network is completely up? Are you able to resolve the hostnames used
in fstab then (e.g. "host nfsserver")?

> And that was nicely working in old times.
> With systemd obviously not. So IMHO this is a problem in systemd, or am I
> wrong?

systemd makes much more in parallel.

You can try to double the WAIT_FOR_INTERFACES and DHCLIENT_WAIT_AT_BOOT
that are in /etc/sysconfig/network/{config,dhcp}. When these timeouts
expire for some reason, /etc/init.d/network reports 0 (success) to the
systemd even the network is not completely started.

Please attach a supportconfig (install supportutils) output.
Comment 25 Axel Braun 2013-01-18 14:56:22 UTC
Created attachment 520876 [details]
output of supportconfig

I have increased the two parameters, but no effect.
My problem deals only with ntp, systemd and Networkmanager, as I'm working on a laptop.
What I dont get so far is why ntp starts even if the network is still down. I though thats the big advantage of systemd , that it starts services depending on states and actions (network comes up -> Network depending services are started)
Comment 26 Nathan Cutler 2017-08-14 18:35:43 UTC
Moving to Tumbleweed because openSUSE 12.2 is out of maintenance. Setting needinfo on assignee - Michal, could you please update the bug?
Comment 27 Josef Reidinger 2018-08-22 13:29:16 UTC
michal - ping