Bug 1215160

Summary: "wipe" reports negative percentage in progress info for large target device
Product: [openSUSE] openSUSE Distribution Reporter: Joachim Wagner <jo4su>
Component: BasesystemAssignee: Lubos Kocman <lubos.kocman>
Status: NEW --- QA Contact: E-mail List <qa-bugs>
Severity: Minor    
Priority: P5 - None CC: ailiopoulos, jo4su
Version: Leap 15.5   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE Leap 15.5   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Joachim Wagner 2023-09-08 10:18:06 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0
Build Identifier: 

Running "wipe" on a 1.81 TiB disk, it shows negative percentages after moving on from "0%". This suggests that the calculations for the progress information are made with an unsuitable numeric type, overflows are not detected and wipe is not ready for large files and devices.

Furthermore, running to "-5%" and then checking how much data of the 1.81 was overwritten, shows that it's not just a flipped sign as the absolute value moves faster than expected, completing a percentage point equivalent to 18.4 GiB in a few seconds even though the target device's write speed in only about 110 MiB/s at best.

Reproducible: Always

Steps to Reproduce:
1. create or select large file or device that can be overwritten and deleted for testing (my tests: 1995527224832 bytes and 2000395780096 bytes)
2. run "wipe" on the file (I added -D to keep the file for inspection)
3. observe progress information for a minute
4. ctrl-c at -7%
5. repeat wipe with "-z"
6. ctrl-c at -5%
7. run "hexdump -C myfileordevice | head" to see how much data was zeroed
Actual Results:  
/dev/bcache2: -2%

Only 10.1 GiB, i.e. 0.5% of the 1.81 TiB device, were zeroed when "-5%" was displayed by "wipe".

Expected Results:  
Percentage in range 0 to 100.

For the 1.81 TiB device, between 83.4 GiB (4.5%) and 111.2 GiB (6%) should be zeroed if the displayed percentage just has the wrong sign.

Likely, wipe cannot handle modern day file and device sizes properly as indicated by open issues on https://sourceforge.net/p/wipe/bugs/ and 
https://bugzilla.opensuse.org/show_bug.cgi?id=1117493
Comment 1 Andreas Stieger 2023-09-08 11:14:24 UTC
Wipe was last touched upstream 2009-11-01, so not surprised here. Consider it abandoned. What do you expect the distribution to do? The tool itself is not as secure or useful as you may think for modern storage devices, setups, and file systems. For the operations that are snake oil, better solutions exist- including "cp /dev/{urandom,zero} ...", along with pv/pipemeter for a progress bar.
Comment 2 Joachim Wagner 2023-09-12 12:07:36 UTC
Good question Andreas.

I suppose that for seeking this bug to be fixed in the current wipe tool shipped with the distribution it would be more useful for me to try to revive discussion in the referenced upstream bug and maybe even contribute a patch there. However, I am posting here to open discussion about what the openSUSE community thinks is the best way forward. Options include

 * Removing the tool from the distribution as it does not function correctly

 * Replacing the tool with a working tool with the same command name and that supports all its command line options

 * Replacing the tool with a working tool with a different command line and possibly incompatible command line options

 * Replacing the tool with a non-functional tool with the same command name that only prints instructions how to wipe files and devices manually and returns an error code

 * Patching the upstream tool

 * Implementing a new tool from scratch with the same command name and that supports all its command line options

Current version 2.3.1 of the wipe tool has 4076 lines of C code. Reading wipe's man page, I think a re-implementation in Python would be much shorter and more easily maintained. Files can be opened with O_SYNC also in Python and the MT PRNG is natively available. I don't understand though the sentence about using the "mand" mount option, or why file locking would be desirable at all during wiping.
Comment 3 Anthony Iliopoulos 2023-09-22 08:48:24 UTC
the modern replacement is the `shred` tool (standard part of coreutils).
Comment 4 Chenzi Cao 2023-10-10 05:47:36 UTC
Hi Joachim, would you please have a try whether the replacement tool of `shred` work? Thanks.
Comment 5 Joachim Wagner 2023-10-16 14:49:56 UTC
The progress output of `shred -n 1 -v` on a device of similar size looks right. See below.

Still, something needs to be done about `wipe`, e.g. wrapping it in a 3-line script that outputs a warning message that it is not working for large files (and then runs the command with the supplied args) or removing the package altogether.


``` shell
IMG=test2tb-nocow.img
GiB=2048
BYTES=$((1024*1024*1024*GiB))
touch $IMG
chattr +C $IMG
fallocate -l $BYTES $IMG
losetup --find --show test2tb-nocow.img
shred -n 1 -v /dev/loop0
```

Output after 5 minutes (RAID 10 on 4 HDDs):
`shred: /dev/loop0: pass 1/1 (random)...90GiB/2.0TiB 4%`

This matches the expected speed and progress.
Comment 6 Andreas Stieger 2023-10-16 15:08:02 UTC
Or we just stop shipping the obsolete stuff
Comment 7 Joachim Wagner 2023-10-18 10:00:53 UTC
Yes, removing the tool was option 1 of comment 2.

Clearing "needinfo" as I assume this is for "does shred work?".

For the user experience of a user looking to install "wipe" as they read somewhere they should use it, just not having it in the repo is not great either. It would be nice to get a search result with an explanation why the tool is not available and what the recommended replacement is. Could there be a package that doesn't install any files and only displays some info on stdout and, before install, in the package's description field?
Comment 8 Andreas Stieger 2023-10-18 10:02:20 UTC
We could just have the shred package provide and obsolete wipe.
Comment 9 Joachim Wagner 2023-10-18 10:37:03 UTC
Shred is missing some functionality to emulate wipe. Most likely to be missed by users, I think, will be -r (recursive) and -i (interactive). Default with -r is to skip special files (-N). That's also not supported by shred, which assumes the user specifies each file individually and only the intended files.

Sure, a script can also emulate -r, -N and -i but then we are doing more programming than just mapping command line options. If we don't support those (and print an error message instead), we are getting close to option 4 of comment 2 (print instructions and exit).
Comment 10 Anthony Iliopoulos 2023-10-18 10:52:39 UTC
(In reply to Joachim Wagner from comment #9)
> Shred is missing some functionality to emulate wipe. Most likely to be
> missed by users, I think, will be -r (recursive) and -i (interactive).
> Default with -r is to skip special files (-N). That's also not supported by
> shred, which assumes the user specifies each file individually and only the
> intended files.

It will certainly not match one to one (unless someone contributes to upstream wipe and fix the pending issues, or to shred to add the missing ones).

While this doesn't work "out of the box" without a helper script, all the above missing functionality can be easily replaced by something like:

> find /path -type f -ok shred {} \;
Comment 11 Joachim Wagner 2023-10-18 11:06:24 UTC
Certainly it is not a lot of work but clearly more work than replying here.

If I tried it as a shell script, it would look horrible as my shell scripting skills are just good enough to make it work.

Would Python be acceptable for a system tool like this? I'd be doing a much better job in Python and I see that there is at least one other system tool using Python: bcache-status. (See also bug 1216370.)
Comment 12 Anthony Iliopoulos 2023-10-18 11:19:00 UTC
(In reply to Joachim Wagner from comment #11)
> Certainly it is not a lot of work but clearly more work than replying here.
> 
> If I tried it as a shell script, it would look horrible as my shell
> scripting skills are just good enough to make it work.
> 
> Would Python be acceptable for a system tool like this? I'd be doing a much
> better job in Python and I see that there is at least one other system tool
> using Python: bcache-status. (See also bug 1216370.)

In general I think it would be good to minimize dependencies especially in the base system.

I don't think I have any better recommendation than comment #8.

The `wipe` tool appears unmaintained for over a decade now, and there may be bigger problems lurking. I would recommend making it obsolete instead of continuing to artificially support it, and giving to the users the illusion that this is being maintained. Users of `wipe` should have been moved to `shred` a long time ago.

If there is such desire to keep any of wipe features or interface compatibility, the best way would be to contribute to upstream coreutils/shred directly.