Bug 971331

Summary: upgraded my 13.1 desktop, now fails to start (systemd dependency, claims an nfs drive wasn't mounted).
Product: [openSUSE] openSUSE 13.1 Reporter: Per Jessen <per>
Component: BasesystemAssignee: Franck Bui <fbui>
Status: RESOLVED WONTFIX QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: mkubecek, nfbrown, per, wolfgang
Version: Final   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Per Jessen 2016-03-16 08:19:30 UTC
I updated a 13.1 office desktop last week, I saw somebody mention a 
new 3.12 kernel.  On start-up, it now claims one of two NFS drives 
couldn't be mounted, although in fact they were both mounted.  This 
caused postfix and X not to be started.  I commented out of the mounts 
in /etc/fstab and rebooted, now it worked. 


office11:~ # systemctl status home-london.mount
home-london.mount - /home/london
   Loaded: loaded (/etc/fstab)
   Active: failed (Result: exit-code) since Wed 2016-03-16 09:00:15 CET; 1min 13s ago
    Where: /home/london
     What: fileserver:/home/london
  Process: 2712 ExecMount=/bin/mount -n -t nfs -o v3,rsize=8192,wsize=8192,fsc fileserver:/home/london /home/london (code=exited, status=32)

Mar 16 09:00:15 office11 mount[2712]: Starting rpc.statd .....done
Mar 16 09:00:15 office11 systemd[1]: home-london.mount mount process exited, code=exited status=32
Mar 16 09:00:15 office11 systemd[1]: Failed to mount /home/london.
Mar 16 09:00:15 office11 systemd[1]: Unit home-london.mount entered failed state.
Mar 16 09:00:15 office11 mount[2712]: mount.nfs: rpc.statd is not running but is required for remote locking.
Mar 16 09:00:15 office11 mount[2712]: mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
Mar 16 09:00:15 office11 mount[2712]: mount.nfs: an incorrect mount option was specified


Looks like a timing issue wrt rpc.statd ?  On one boot-up it'll be
/home/london, on another it'll be /home/barcelona.

/etc/fstab:
office11:~ # cat /etc/fstab
UUID=1c3e6959-8be2-4de3-831e-1998981c3da6 swap                 swap       defaults              0 0
/dev/disk/by-id/ata-WDC_WD800AAJS-60PSA0_WD-WCAP92208789-part2 /                    ext4       acl,user_xattr        1 1
UUID=6ad0ba66-331c-4a52-b0ea-b89a9cc88dee /home                ext4       acl,user_xattr        1 2

fileserver:/home/london  /home/london  nfs      v3,rsize=8192,wsize=8192,fsc 0 0
fileserver:/home/barcelona  /home/barcelona  nfs      v3,rsize=8192,wsize=8192,fsc 0 0

(using nfsv3 because the fileserver has a problem with v4).
Comment 1 Wolfgang Rosenauer 2016-03-16 08:35:15 UTC
Frank, does this look related to the systemd Upgrade?
Comment 2 Franck Bui 2016-03-16 08:47:59 UTC
(In reply to Wolfgang Rosenauer from comment #1)
> Frank, does this look related to the systemd Upgrade?

Hmm not sure, but it looks like more an NFS issue since the mount command started by the mount unit failed.

But really not sure.

Can you add an NFS expert in the loop ?
Comment 3 Wolfgang Rosenauer 2016-03-16 09:02:47 UTC
Hmm, just assuming for now that this is somehow related to the recent updates. Then we have the kernel still as candidate which might have changed.
Adding Michal to CC.
Per, do you still have the older kernel around and see what happens if you boot with that?
Comment 4 Per Jessen 2016-03-16 09:28:51 UTC
(In reply to Franck Bui from comment #2)
> (In reply to Wolfgang Rosenauer from comment #1)
> > Frank, does this look related to the systemd Upgrade?
> 
> Hmm not sure, but it looks like more an NFS issue since the mount command
> started by the mount unit failed.
> But really not sure.

To me it looks like an issue with rpc.statd.

> Can you add an NFS expert in the loop ?

I think that's probably Neil Brown. 

> Hmm, just assuming for now that this is somehow related to the recent
> updates. Then we have the kernel still as candidate which might have 
> changed. Adding Michal to CC.
> Per, do you still have the older kernel around and see what happens 
> if you boot with that?

Yep, already tried with the previous one (3.11.10-29), same result.

It's a clearly a race - sometime it works with two rpc.statd:

# systemctl status home-london.mount
home-london.mount - /home/london
   Loaded: loaded (/etc/fstab)
   Active: active (mounted) since Wed 2016-03-16 10:18:59 CET; 1min 34s ago
    Where: /home/london
     What: fileserver:/home/london
  Process: 2725 ExecMount=/bin/mount -n -t nfs -o v3,rsize=8192,wsize=8192,fsc fileserver:/home/london /home/london (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/home-london.mount
           └─2794 /usr/sbin/rpc.statd --no-notify

Mar 16 10:18:59 office11 rpc.statd[2794]: Version 1.2.8 starting
Mar 16 10:18:59 office11 rpc.statd[2794]: Flags: TI-RPC
Mar 16 10:18:59 office11 mount[2725]: Starting rpc.statd .....done
Mar 16 10:18:59 office11 systemd[1]: Mounted /home/london.

# systemctl status home-barcelona.mount
home-barcelona.mount - /home/barcelona
   Loaded: loaded (/etc/fstab)
   Active: active (mounted) since Wed 2016-03-16 10:18:59 CET; 1min 54s ago
    Where: /home/barcelona
     What: fileserver:/home/barcelona
  Process: 2721 ExecMount=/bin/mount -n -t nfs -o v3,rsize=8192,wsize=8192,fsc fileserver:/home/barcelona /home/barcelona (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/home-barcelona.mount
           └─2789 /usr/sbin/rpc.statd --no-notify

Mar 16 10:18:59 office11 rpc.statd[2789]: Version 1.2.8 starting
Mar 16 10:18:59 office11 rpc.statd[2789]: Flags: TI-RPC
Mar 16 10:18:59 office11 mount[2721]: Starting rpc.statd .....done
Mar 16 10:18:59 office11 systemd[1]: Mounted /home/barcelona.
Comment 5 Per Jessen 2016-03-16 09:41:27 UTC
Uh, this doesn't appear to be an issue in the updates.  I think the NFS-mounted /home/barcelona was added at some point, but that machine (office11) never rebooted.  I have just tried on another 13.1 desktop (office20, systemd 208, kernel 3.11.10-29-desktop), same problem. 

Thinking out loud - rpc.statd is needed because these are nfsv3 mounts, but rpc.statd is started by mount, not by systemd.  I presume there's nothing guaranteeing any specific sequencing of two mounts, so there's plenty of opportunity for a race.

I'll try it on 13.2 and Leap421 too.
Comment 6 Per Jessen 2016-03-16 09:46:25 UTC
(In reply to Per Jessen from comment #5)
> 
> I'll try it on 13.2 and Leap421 too.

Well, both 13.2 and Leap421 have a /usr/lib/systemd/system/rpc-statd.service.
Comment 7 Neil Brown 2016-03-17 23:32:44 UTC
Please try the fix mentioned in comments 12 and 13 of bug 936321
Comment 8 Tomáš Chvátal 2018-04-12 13:44:49 UTC
This version of openSUSE changed to end-of-life (EOL [1]) status. As such
it is no longer maintained, which means that it will not receive any
further security or bug fix updates.
As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
openSUSE, or consider the bug still valid, please feel free to reopen this
bug against that version, or open a new ticket.

Thank you for reporting this bug and we are sorry it could not be fixed
during the lifetime of the release.

[1] https://en.opensuse.org/Lifetime