Bug 1164313

Summary: [Build 20200218] yast2 nfs-client fails, cannot start rpc-statd.service
Product: [openSUSE] openSUSE Tumbleweed Reporter: Jonathan Rivrain <jrivrain>
Component: YaST2Assignee: YaST Team <yast-internal>
Status: CONFIRMED --- QA Contact: Jiri Srain <jsrain>
Severity: Normal    
Priority: P5 - None CC: guillaume.gardet, jlopez, jreidinger, jrivrain
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
URL: https://openqa.opensuse.org/tests/1179190/modules/yast2_nfs_client/steps/49
See Also: http://bugzilla.suse.com/show_bug.cgi?id=1161687
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: nfs.conf

Description Jonathan Rivrain 2020-02-19 13:52:20 UTC
## Observation

extract of y2logs:

2020-02-19 07:28:44 <1> susetest(2415) [Ruby] callbacks/libstorage_callback.rb:76 Error details. Message: Mounting 10.0.2.101:/srv/nfs at /tmp/nfs/client. What: command '/bin/mount -t nfs '10.0.2.101:/srv/nfs' '/tmp/nfs/client'' failed:

stderr:
Job for rpc-statd.service failed because a timeout was exceeded.
See "systemctl status rpc-statd.service" and "journalctl -xe" for details.
: config error at /etc/nfs.conf:12: error loading included config
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Protocol not supported

exit code:
32.

Could be related to https://bugzilla.opensuse.org/show_bug.cgi?id=969152 and/or https://bugzilla.opensuse.org/show_bug.cgi?id=1161687.

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-yast2_nfs_v3_client@64bit fails in
[yast2_nfs_client](https://openqa.opensuse.org/tests/1179190/modules/yast2_nfs_client/steps/49)

## Test suite description
NFS multi-machine test suite. Maintainer jrivrain@suse.com


## Reproducible

Fails since (at least) Build [20200120](https://openqa.opensuse.org/tests/1151100)


## Expected result

Last good: [20200119](https://openqa.opensuse.org/tests/1150542) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=yast2_nfs_v3_client&version=Tumbleweed)
Comment 1 Arvin Schnell 2020-02-19 15:54:01 UTC
The SUT has yast2-nfs-client 4.2.6 installed, so the fix for bug #1161687,
which could be related, is not included (since it is only in 4.2.7).

Please recheck with yast2-nfs-client 4.2.7.
Comment 2 Jonathan Rivrain 2020-02-20 15:48:57 UTC
I just tried with 4.2.7, it still happens. 

The problem mentioned in https://bugzilla.suse.com/show_bug.cgi?id=1161687 is different to what is happening right now. as we can see in openqa, we already no longer see the problem with rpcbind and yast2-nfs-client 4.2.6. So, that what was added in 4.2.7 did not solve the previous problem, as it got solved earlier. I can try to track at what point exactly it got fixed.

Right now, the problem seems to be on the server side: https://openqa.opensuse.org/tests/1179131#step/yast2_nfs_server/68 in that screenshot we see that "firewall is not configurable". Then on client we see "Remote system error - Conection refused" in output of "systemctl status rpc.statd". 
Locally, if I disable the firewall on my nfs server, the error does not occur anymore. Is this normal ? Are we doing something wrong in our test ?
Comment 3 Jonathan Rivrain 2020-02-20 15:53:21 UTC
Sorry, I missed that https://openqa.opensuse.org/tests/1179131#step/yast2_nfs_server/31, we actually disable the firewall before.
Comment 4 Jonathan Rivrain 2020-02-26 17:25:07 UTC
I tried with 4.2.7, it does the same. What I can say so far:

- bsc#1161687 is already fixed (?) with 4.2.6, as we can see in openqa,
- the current bug happens with both 4.2.6 and 4.2.7,
- On server side, rpc-statd service is started and port is opened, firewalld is disabled on both side (in my manual tests, bare-metal TW server, Qemu vm with bridged networking for client)

journalctl -u rpc.statd:

Feb 26 09:08:32 dhcp9 systemd[1]: Starting NFS status monitor for NFSv2/3 locking....
Feb 26 09:08:32 dhcp9 rpc.statd[2491]: : config error at /etc/nfs.conf:12: error loading included config
Feb 26 09:08:33 dhcp9 rpc.statd[2509]: Version 2.4.3 starting
Feb 26 09:08:33 dhcp9 rpc.statd[2509]: Flags: TI-RPC
Feb 26 09:08:33 dhcp9 rpc.statd[2509]: Failed to register (statd, 1, udp): svc_reg() err: RPC: Remote system error - Connection refused
Feb 26 09:08:33 dhcp9 rpc.statd[2509]: Failed to register (statd, 1, tcp): svc_reg() err: RPC: Remote system error - Connection refused
Feb 26 09:08:33 dhcp9 rpc.statd[2509]: Failed to register (statd, 1, udp6): svc_reg() err: RPC: Remote system error - Connection refused
Feb 26 09:08:33 dhcp9 rpc.statd[2509]: Failed to register (statd, 1, tcp6): svc_reg() err: RPC: Remote system error - Connection refused
Feb 26 09:08:33 dhcp9 rpc.statd[2509]: failed to create RPC listeners, exiting
Feb 26 09:08:33 dhcp9 systemd[1]: rpc-statd.service: Failed to parse PID from file /run/rpc.statd.pid: Invalid argument
Feb 26 09:10:02 dhcp9 systemd[1]: rpc-statd.service: start operation timed out. Terminating.
Feb 26 09:10:02 dhcp9 systemd[1]: rpc-statd.service: Failed with result 'timeout'.
Feb 26 09:10:02 dhcp9 systemd[1]: Failed to start NFS status monitor for NFSv2/3 locking..
Comment 5 Josef Reidinger 2020-02-28 09:15:12 UTC
Jonathat - I think root of issue is that "config error at /etc/nfs.conf:12: error loading included config"

Can you please attach /etc/nfs.conf and also file included at line 12 ?
Comment 6 Jonathan Rivrain 2020-02-28 17:09:08 UTC
(In reply to Josef Reidinger from comment #5)
> Jonathat - I think root of issue is that "config error at /etc/nfs.conf:12:
> error loading included config"
> 
> Can you please attach /etc/nfs.conf and also file included at line 12 ?

At line 12 we have "include = /etc/nfs.conf.local" a file that does not exist. but if we comment the line 12, the problem still occurs. The error message is the same except the part about that line.
Comment 7 Jonathan Rivrain 2020-02-28 17:16:39 UTC
Created attachment 831574 [details]
nfs.conf

Rectification, the line 12 is include = -/etc/nfs.conf.local.
Comment 14 José Iván López González 2020-06-11 12:39:24 UTC
It seems we still have some issue because rpc.stand is not running, see https://openqa.opensuse.org/tests/1284963:

/var/log/YaST2/y2log:2020-06-02 13:02:36 <3> susetest(2429) [libstorage] SystemCmd.cc:97 stderr:
/var/log/YaST2/y2log:2020-06-02 13:02:36 <3> susetest(2429) [libstorage] SystemCmd.cc:97 Job for rpc-statd.service failed because a timeout was exceeded.
/var/log/YaST2/y2log:2020-06-02 13:02:36 <3> susetest(2429) [libstorage] SystemCmd.cc:97 See "systemctl status rpc-statd.service" and "journalctl -xe" for details.
/var/log/YaST2/y2log:2020-06-02 13:02:36 <3> susetest(2429) [libstorage] SystemCmd.cc:97 mount.nfs: rpc.statd is not running but is required for remote locking.
/var/log/YaST2/y2log:2020-06-02 13:02:36 <3> susetest(2429) [libstorage] SystemCmd.cc:97 mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
/var/log/YaST2/y2log:2020-06-02 13:02:36 <3> susetest(2429) [libstorage] SystemCmd.cc:97 mount.nfs: Protocol not supported

Maybe this could be slightly related to https://bugzilla.suse.com/show_bug.cgi?id=1161687.

Tracking in our trello board for planning it.
Comment 31 Oliver Kurz 2021-03-06 06:05:57 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_client
https://openqa.opensuse.org/tests/1656876

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 32 Oliver Kurz 2021-03-21 06:05:02 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_client
https://openqa.opensuse.org/tests/1674386

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 33 Oliver Kurz 2021-04-05 05:03:30 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_client
https://openqa.opensuse.org/tests/1689175

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 34 Oliver Kurz 2021-04-19 05:03:37 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_client
https://openqa.opensuse.org/tests/1704370

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 35 openQA Review 2021-05-03 05:17:59 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_client
https://openqa.opensuse.org/tests/1722351

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 36 Oliver Kurz 2021-05-17 06:23:25 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_server
https://openqa.opensuse.org/tests/1742904

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 37 openQA Review 2021-06-01 05:16:55 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_server
https://openqa.opensuse.org/tests/1747425

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 38 openQA Review 2021-06-15 05:18:47 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_server
https://openqa.opensuse.org/tests/1785038

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed
Comment 39 Oliver Kurz 2021-06-29 06:24:29 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_server
https://openqa.opensuse.org/tests/1808974

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The label in the openQA scenario is removed
Comment 40 openQA Review 2021-07-25 00:02:35 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_server
https://openqa.opensuse.org/tests/1853703

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The label in the openQA scenario is removed
Comment 41 openQA Review 2021-08-20 00:17:07 UTC
This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_nfs_v3_server
https://openqa.opensuse.org/tests/1879753

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The label in the openQA scenario is removed