Bug 1213268 - [leap15.4][zfs] 'ztest' fails sporadically with error "ztest: attach (/tmp/ztest.12a 285212672, /tmp/ztest.spares.1 330825728, 1) returned 95, expected 0"
Summary: [leap15.4][zfs] 'ztest' fails sporadically with error "ztest: attach (/tmp/zt...
Status: RESOLVED INVALID
Alias: None
Product: openSUSE Distribution
Classification: openSUSE
Component: Other (show other bugs)
Version: Leap 15.4
Hardware: Other openSUSE Leap 15.5
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: E-mail List
QA Contact: E-mail List
URL: https://openqa.opensuse.org/tests/342...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-07-13 04:49 UTC by Richard Fan
Modified: 2023-07-13 08:29 UTC (History)
1 user (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments
serial_console_logs (812.96 KB, text/plain)
2023-07-13 05:39 UTC, Richard Fan
Details
Test run logs (2.39 MB, text/plain)
2023-07-13 05:40 UTC, Richard Fan
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Fan 2023-07-13 04:49:28 UTC
The issue can be seen sporadically. but I can reproduce the issue in past few weeks via openQA automation run.

Bug description:

"ztest" reports below errors which are not expected:

ztest: attach (/tmp/ztest.12a 285212672, /tmp/ztest.spares.1 330825728, 1) returned 95, expected 0
/usr/sbin/ztest(+0xdff3)[0x56371f86fff3]
/lib64/libpthread.so.0(+0x168c0)[0x7f2daa1a38c0]
/lib64/libc.so.6(gsignal+0x10d)[0x7f2da9fe0c6b]
/lib64/libc.so.6(abort+0x177)[0x7f2da9fe2305]
/usr/sbin/ztest(+0xd146)[0x56371f86f146]
/usr/sbin/ztest(+0x1552b)[0x56371f87752b]
/usr/sbin/ztest(+0x1315e)[0x56371f87515e]
/lib64/libpthread.so.0(+0xa6ea)[0x7f2daa1976ea]
/lib64/libc.so.6(clone+0x3f)[0x7f2daa0ad94f]
child died with signal 6
nff7n-3-

===================================================================
The latest failure as below:

##Steps to reproduce the issue:

1. install the packages
#zypper -n addrepo -fG https://download.opensuse.org/repositories/filesystems/15.4/filesystems.repo 

# zypper -n refresh

#zypper -n install zfs
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 8 NEW packages are going to be installed:
  libnvpair3 libuutil3 libzfs4 libzfs_core3 libzpool5 zfs zfs-kmp-default zfs-sudo

8 new packages to install.
Overall download size: 20.2 MiB. Already cached: 0 B. After the operation, additional 138.1 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving: libnvpair3-2.1.12-lp154.1.6.x86_64 (Filesystem tools and FUSE-related packages (15.4)) (1/8),  49.2 KiB    
Retrieving: libnvpair3-2.1.12-lp154.1.6.x86_64.rpm [..done]
Retrieving: libuutil3-2.1.12-lp154.1.6.x86_64 (Filesystem tools and FUSE-related packages (15.4)) (2/8),  43.3 KiB    
Retrieving: libuutil3-2.1.12-lp154.1.6.x86_64.rpm [.done]
Retrieving: zfs-kmp-default-2.1.12_k5.14.21_150400.24.66-lp154.1.6.x86_64 (Filesystem tools and FUSE-related packages (15.4)) (3/8),  17.9 MiB    
Retrieving: zfs-kmp-default-2.1.12_k5.14.21_150400.24.66-lp154.1.6.x86_64.rpm [.......................done (8.0 MiB/s)]
Retrieving: libzfs_core3-2.1.12-lp154.1.6.x86_64 (Filesystem tools and FUSE-related packages (15.4)) (4/8),  68.1 KiB    
Retrieving: libzfs_core3-2.1.12-lp154.1.6.x86_64.rpm [..done (668.8 KiB/s)]
Retrieving: libzpool5-2.1.12-lp154.1.6.x86_64 (Filesystem tools and FUSE-related packages (15.4)) (5/8),   1.2 MiB    
Retrieving: libzpool5-2.1.12-lp154.1.6.x86_64.rpm [......done (3.0 MiB/s)]
Retrieving: libzfs4-2.1.12-lp154.1.6.x86_64 (Filesystem tools and FUSE-related packages (15.4)) (6/8), 185.0 KiB    
Retrieving: libzfs4-2.1.12-lp154.1.6.x86_64.rpm [..done]
Retrieving: zfs-2.1.12-lp154.1.6.x86_64 (Filesystem tools and FUSE-related packages (15.4)) (7/8), 703.5 KiB    
Retrieving: zfs-2.1.12-lp154.1.6.x86_64.rpm [....done (3.6 MiB/s)]
Retrieving: zfs-sudo-2.1.12-lp154.1.6.noarch (Filesystem tools and FUSE-related packages (15.4)) (8/8),  19.9 KiB    
Retrieving: zfs-sudo-2.1.12-lp154.1.6.noarch.rpm [.done]


2. run “ztest"
 ztest; echo nff7n-$?-

verifying concrete vdev 0, metaslab 0 of 15 ...
verifying concrete vdev 0, metaslab 1 of 15 ...
verifying concrete vdev 0, metaslab 2 of 15 ...
verifying concrete vdev 0, metaslab 3 of 15 ...
snip here [please see attached file for detail logs]
loading concrete vdev 6, metaslab 12 of 15 ...
loading concrete vdev 6, metaslab 13 of 15 ...
loading concrete vdev 6, metaslab 14 of 15 ...

17.9M completed (  18MB/s) estimated time remaining: 0hr 00min 05sec        
37.0M completed (  18MB/s) estimated time remaining: 0hr 00min 04sec        
47.8M completed (  16MB/s) estimated time remaining: 0hr 00min 04sec        
59.3M completed (  14MB/s) estimated time remaining: 0hr 00min 03sec        
69.3M completed (  13MB/s) estimated time remaining: 0hr 00min 03sec        
78.1M completed (  12MB/s) estimated time remaining: 0hr 00min 02sec        
95.6M completed (  13MB/s) estimated time remaining: 0hr 00min 01sec        ztest: attach (/tmp/ztest.12a 285212672, /tmp/ztest.spares.1 330825728, 1) returned 95, expected 0
/usr/sbin/ztest(+0xdff3)[0x56371f86fff3]
/lib64/libpthread.so.0(+0x168c0)[0x7f2daa1a38c0]
/lib64/libc.so.6(gsignal+0x10d)[0x7f2da9fe0c6b]
/lib64/libc.so.6(abort+0x177)[0x7f2da9fe2305]
/usr/sbin/ztest(+0xd146)[0x56371f86f146]
/usr/sbin/ztest(+0x1552b)[0x56371f87752b]
/usr/sbin/ztest(+0x1315e)[0x56371f87515e]
/lib64/libpthread.so.0(+0xa6ea)[0x7f2daa1976ea]
/lib64/libc.so.6(clone+0x3f)[0x7f2daa0ad94f]
child died with signal 6
nff7n-3-


Expected result:
ztest returns 0


------------------------------------------------------------------------

##openQA Observation

openQA test in scenario opensuse-15.4-DVD-Updates-x86_64-extra_tests_filesystem@64bit fails in
[zfs](https://openqa.opensuse.org/tests/3423681/modules/zfs/steps/39)

## Test suite description
Maintainer: QE Core

Filesystem related tests, for example snapper and btrfs features.


## Reproducible

Fails since (at least) Build [20230713-1](https://openqa.opensuse.org/tests/3423456)


## Expected result

Last good: [20230712-5](https://openqa.opensuse.org/tests/3423039) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD-Updates&machine=64bit&test=extra_tests_filesystem&version=15.4)
Comment 1 Richard Fan 2023-07-13 04:54:10 UTC
I failed to upload the log files for the time being,please wait for my notification.
Comment 2 Richard Fan 2023-07-13 05:39:28 UTC
Created attachment 868171 [details]
serial_console_logs
Comment 3 Richard Fan 2023-07-13 05:40:01 UTC
Created attachment 868172 [details]
Test run logs
Comment 4 Richard Fan 2023-07-13 05:40:56 UTC
(In reply to Richard Fan from comment #1)
> I failed to upload the log files for the time being,please wait for my
> notification.

Logs are uploaded, more logs can be found at https://openqa.opensuse.org/tests/3423681#downloads
Comment 5 Andreas Stieger 2023-07-13 07:09:56 UTC
You reported this against the maintenance component of Leap 15.4. ZFS is not in the distribution last time I checked. How do you envision the resolution of this bug?
Comment 6 Richard Fan 2023-07-13 07:20:22 UTC
(In reply to Andreas Stieger from comment #5)
> You reported this against the maintenance component of Leap 15.4. ZFS is not
> in the distribution last time I checked. How do you envision the resolution
> of this bug?

Thanks, then let me try to use component 'other' and some experts can assign it to right one.
Comment 7 Andreas Stieger 2023-07-13 07:30:43 UTC
What I was getting at: Leap 15.4 is on the way out and the package is not in any distribution. So this was never working, and needs to work in TW first. So I am wondering why you are running this?
Comment 8 Richard Fan 2023-07-13 07:42:56 UTC
(In reply to Andreas Stieger from comment #7)
> What I was getting at: Leap 15.4 is on the way out and the package is not in
> any distribution. So this was never working, and needs to work in TW first.
> So I am wondering why you are running this?

I will un-schedule the test in leap 15.4/15.5 update, Thanks for your kindly comments.
Comment 9 Marcus Meissner 2023-07-13 08:29:12 UTC
should not have been added to Leap openqa, invalid