Bug 112902

Created attachment 47492 [details]
Screenshot of failure

Created attachment 47493 [details]
YaST2 logs

Created attachment 47494 [details]
dmesg of affected system

ntfsresize displays the following error message:
---------------------------------------------------------------------------
ntfsresize v1.11.1
ERROR(5): Opening '/dev/hda1' as NTFS failed: Input/output error
Apparently you have a corrupted NTFS. Please run the filesystem checker
on Windows by invoking chkdsk /f. Don't forget the /f (force) parameter,
it's important! You probably also need to reboot Windows to take effect.
Then you can try ntfsresize again. No modification was made to your NTFS.
---------------------------------------------------------------------------

There is not much YaST2 can do about such a case. Unfortunately this condition
is not detected by the ntfsresize -i call YaST2 executes to check if the 
filesystem is resizable. 

Such things can happen and there is not much YaST2 can do in such a case
except aborting.

The partition is clean.

When I boot into the installation system and do a

   $ ntfsresize -i /dev/hda1

it reports that everything is fine. Though, once YaST wants to resize the partition the reported error 
occurs.

Did you even read my comment? I already wrote that the "ntfsresize -i" call 
does not detect the problem.
The error only happens during the real resize call. So either ntfsresize is
buggy (then there is not much YaST2 can do) or it is right and you have a bad
block in your ntfs aread.

(In reply to comment #6)
> Did you even read my comment?

Yes.

> I already wrote that the "ntfsresize -i" call 
> does not detect the problem.

comment below.

> The error only happens during the real resize call. So either ntfsresize is
> buggy (then there is not much YaST2 can do) or it is right and you have a bad
> block in your ntfs aread.

  * booting the system with freshly checked NTFS partition
  * $ ntfsresize -i /dev/hda1 -> no error
  * YaST resize call -> error
  * $ ntfsresize -i /dev/hda1 -> error

Actually ntfsresize does recognize the error.

Or am I missing something?

It is normal behaviour of ntfsresize (if not called with -i option) to set the
dirty flag of the ntfs filesystem. As far as I know, the problem that happened 
in your attached y2log file has nothing to do with the dirty flag, but I am not
an expert in ntfs or ntfsresie at all. 
An error with ntfsresize -i was not in y2log file you attached. As already said
I cannot rule out a bug in ntfsresize (but doubt it since dmsg also displays
warning from ntfs.

(In reply to comment #8)
> It is normal behaviour of ntfsresize (if not called with -i option) to set the
> dirty flag of the ntfs filesystem.

OK. I was supposing something like that.

> As far as I know, the problem that happened 
> in your attached y2log file has nothing to do with the dirty flag, but I am not
> an expert in ntfs or ntfsresie at all. 
> An error with ntfsresize -i was not in y2log file you attached. As already said
> I cannot rule out a bug in ntfsresize (but doubt it since dmsg also displays
> warning from ntfs.

Yes, most likely the original bug report was based on a installation where the dirty flag was already 
set. However, my further attemps were with a clean partition (no more errors in dmesg, too).

I'm just anxious that the behavior I encountered might apply to all systems where hda1 is a NTFS 
partition filling up the whole harddrive...

I dount that the original bug report was caused by a set dirty flag.
A set dirty flag is detected by ntfsresize -i and YaST2 should refuse to
resize such a filesystem. In the original bug report ntfsresize -i does not
show any problems. Additionally a set dirty flag does not normally cause
kernel warnings AFAIK.

So what is going on there?  
Do I understand you right, that there may be systems out there where we cannot  
resize whatever we do? 
Can we at least add a useful error-message or the unexperienced user? 
AJ: I think we have a problem. 

Of course we cannot resize broken ntfs filesystems, we never could.

For the unexperienced user the message "resize failed" should tell him
what hee needs to know. Nevertheless I added the stderr output of then#failed
ntfsresize command to the error popup, but doubt that the unexperienced user 
is helped by that in any way.

We are telling everyone that he can use both Windows and Linux at the same    
time on his machine. That he can resize his NTFS-Partition.    
I have here a machine where this is clearly NOT true.    
Even after defragmentation and chkdsk runs it is not possible to resize the  
NTFS partition.  
After both runs (defrag and chkdsk) the NTFS SHOULD NOT BE BROKEN.  
So please explain what is broken. The machine is available.  
Andreas: Regarding our press-releases that often states 'NTFS resizer' I  
regard this as major. 

I see no way YaST2 could handle anything else in this case.
We check if the filesystem can be resized with "ntfsresize -i" this suceeds.
Afterwards the call to ntfsresize fails, what should YaST2 do except report
the error. 

I am neither competent nor reponsible for checking if ntfsresize is buggy or 
the ntfs filesystem is broken or the kernel might have a proble reading
certain disk blocks or whatever. Am am maintainer of yast2-storage, not of
ntfsresize. 

Fact is: YaST2 calls a utility program (in this case ntfsresize) and this
program fails (unexpectedly since the same program told before it is able
to resize the very same filesystem). So YaST2 reports the error and is done 
with it. IMHO this is perfectly the right thing to do. 

Carl-Daniel, any comments from your side?

Maintainer of ntfsresize is adrian, ok. But that useless error-window is 
comming from YaST. 

I was trying to install SUSE on a Toshiba Tecra A4 and received the following  
error when proceeding with a custom partition setup that required Windows to  
be shrunk:  
  
"Error  
  
Storage modification failed.  
  
System error code was: -3027  
  
  
Failure occurred during following action: shrinking partition /dev/sda1 to  
77.0 GB.  
  
[OK]"  
  
Then the install continued after I pressed OK. However, it froze before it  
even asked me for the second disk.  

Alex Radu: please attach y2logs and tell us which beta you are using.

Szaka, can you shed some light on this?

Thomas: The "ntfsresize -f -i" output says:
"Please make a test run using both the -n and -s options before real resizing!"
Any chance we can do that? It should catch the errors above.

The interesting error during mount is:
"NTFS driver 2.1.23 [Flags: R/W MODULE].
NTFS volume version 3.1.
NTFS-fs error (device hda1): ntfs_check_logfile(): The two restart pages in
$LogFile do not match.
NTFS-fs warning (device hda1): load_system_files(): Failed to load $LogFile. 
Will not be able to remount read-write.  Mount in Windows."

The error during ntfsresize is:
"ntfsresize v1.11.1
ERROR(5): Opening '/dev/hda1' as NTFS failed: Input/output error
Apparently you have a corrupted NTFS. Please run the filesystem checker
on Windows by invoking chkdsk /f. Don't forget the /f (force) parameter,
it's important! You probably also need to reboot Windows to take effect.
Then you can try ntfsresize again. No modification was made to your NTFS."

As a side note, beta4 has an updated ntfsresize, but I doubt this will help here.

Stefan: Can you please (in that order without other steps in between):
- boot Windows and run "chkdsk /f c:" in a command window
- shutdown windows (not hibernate, not reboot)
- turn machine off
- boot into windows again (it MUST display checking the disk)
- shutdown windows (not hibernate, not reboot)
- turn machine off
- boot from a beta3 cd into recovery mode
- run
ntfsclone --save-image --output - /dev/hda1 | gzip -c | ssh host 'cat >
backup.img.gz'
- if possible (might fail due to lack of space, can be circumvented by monting a
big filesystem over nfs), also run
ntfsclone --metadata --output ntfsmeta.img /dev/hda1
tar -cjSf ntfsmeta.img.tar.bz2 ntfsmeta.img

And please provide exact Windows version (including service packs etc.).

Thomas: Could you have a look at first paragraph of comment #21?

Currently we execute "ntfsresize -i" during partition detection since is
is reasonably fast. Adding a "ntfsresize -n" in this area would considerable
delay detection is ntfs partitions are present. Additionally we would not get
much farther. The only thing we would win is that we could tell the user
that he is not able to install Linux on his machine. 

Which is essentially the same as we tell him now, just one step later.
Fact is, with this version of ntfsresize we cannot install Linux on such
a system. 

Making startup considerable slower on every machine with ntfs on it, just to
give the error message about not being able to install linux some minutes 
earlier is not worth rewriting large portions of resize handling
after release candidate 1.

Thomas: How about "ntfsresize -n" before real resize? That would just need twice
the time for real resizing, no additional overhead. However, I also agree with
you that "ntfsresize -i" should detect that problem.

Szaka: Any idea why "ntfsresize -i" found no errors, but the real resize failed?

You would just get the same error message from "ntfsresize -n" that you get 
now from real call to nfsresize, so I see nothing we win by adding a 
"ntfsresize -n" before every "ntfsresize".

We get the information that resize failed from the real resize call anyway.

ok, preparing the information requested in comment 22. Still looking for some 
storage. 

(From Szakacsits Szabolcs via mail):

Hello,

Bugzilla doesn't work for me, "reply", etc does nothing (no input box where to
write the comments), so below is my answer.

ntfsresize 1.11.2 must be more verbose about this issue and should write this:

 "Apparently the NTFS journal file is unclean. Please shutdown Windows\n"
 "properly before using this software. If it wouldn't help then please\n"
 "report it to linux-ntfs-dev@lists.sf.net. Thank you.\n";

When the latest ntfsprogs and kernel driver open NTFS for write then they check
the NTFS journal file and report the errors and refuse to progress.

Because the -i and the -i -n modes are ready-only mounts thus no journal file
check is done so no this error message. Well, I also think this is confusing.

Only a few people reported this issue so far and the problem is indeed that that
the NTFS journal file is not correct. This check is in the ntfs library and
ntfsresize wouldn't need it since if all the real stuff is consistent then it
simply resets (repairs) the journal (the corruptions are thrown away).

Why the NTFS journal is corrupted? User didn't shutdown the computer or rebooted
Windows only once after 'chkdsk /f'. Apparently chkdsk repairs NTFS (if it's the
boot/system partition) in __two__ reboots!!!

Two solutions for this problem:

  1) run ntfsfix before ntfsresize (ntfsfix resets the journal file and
     ntfsresize will accept that)

  2) boot Windows, run 'chkdsk /f' and reboot Windows __TWICE__.

Please note, that never waste time for defrag. It's completely pointless, not
needed at all for a very long time. Repairing NTFS damages, caused by the
Windows NTFS driver, is much more important because otherwise the ntfsresize
sanity checks won't pass and resizing will be refused.

I've already started adding code to repair NTFS during resizing and quite
probably I'll continue to do so (the sanity checks catching NTFS corruptions
indeed confuse many people). This case also seems to be a potential good candinate.

> ------- Additional Comments From kernel01@hailfinger.org  2005-08-29 08:00 MST
-------
> As a side note, beta4 has an updated ntfsresize, but I doubt this will help here.


More exact error message. Perhaps if user boots Windows then resizing could be ok.

> Stefan: Can you please (in that order without other steps in between):
> - boot Windows and run "chkdsk /f c:" in a command window
> - shutdown windows (not hibernate, not reboot)
> - turn machine off
> - boot into windows again (it MUST display checking the disk)
> - shutdown windows (not hibernate, not reboot)
> - turn machine off
> - boot from a beta3 cd into recovery mode
> - run
> ntfsclone --save-image --output - /dev/hda1 | gzip -c | ssh host 'cat >
> backup.img.gz'
> - if possible (might fail due to lack of space, can be circumvented by monting a
> big filesystem over nfs), also run
> ntfsclone --metadata --output ntfsmeta.img /dev/hda1
> tar -cjSf ntfsmeta.img.tar.bz2 ntfsmeta.img


The header of the NTFS journal file will have CHKD (meaning that "repair file
with chkdsk next time") and that causes libntfs refuse to open the volume for
write access.

Let me know if you need more info or something. Also feel free to add the above
comments to bugzilla or tell me how to use it and I would do myself.
Never had this bugzilla usability problem before ...

    Szaka

Ok. The interesting fact is, that the NTFS-partition might be damaged when we 
received the Laptop. So the image on that Laptop had the fault - meaning that 
perhaps all units of that model are affected. Luckily Panasonic doesn't have a 
huge market share :) 
So you suggest that we try again with beta4? 
We can exchange the disk and store it for later tests, too, if that helps, and 
so test beta4 with another harddisk on that Laptop. 
Perhaps we should at least make an SDB-articel for all those running into the 
same problem. 

Yes, it is possible that the master NTFS image, used for imaging, was not
perfectly correct.

But there are many ways NTFS to be incorrect. One of the most often ones is when
the Windows NTFS driver leaks blocks. The fix for this is chkdsk /f and reboot
twice. If user reboots only once then he can get the error described above
(journal is not ok yet).

chkdsk /f, rebooting Windows twice and clean shutdown must always leave NTFS in
a clean state.

If it doesn't then ntfsfix before ntfsresize or we must solve this in ntfsresize. 

Created attachment 48322 [details]
ntfsmeta.img.tar.bz2

Windowsversion is  
XP Prof Version 2002  SP2 
 

Thanks the file but unfortunately the ntfsmeta.img is corrupted, quite probably
because it was packed by one of the many buggy tar versions.

Tar's -S option is broken for 4+ GB files for at least 9 months. It badly 
corrupts those files, sometimes only saving 5-10 bytes in the tar archive. 
This was fixed 5 months ago in the CVS but still no new official tar 
version. The tar maintainer wrote he will release a fixed version, whenever 
he feels so.

An alternative packing is just using bzip2 (gzip is too inefficient space-wise
for sparse files):

 	ntfsclone --metadata --output ntfsmeta.img <partition>
 	bzip2 ntfsmeta.img

This is unfortunately can be very slow (from several to tens of minutes) 
but it must always work.

Unpacking is a bit tricky, otherwise bzip2 would create a non-sparse file:

  bunzip2 -c ntfsmeta.img.bz2 | cp --sparse=always /proc/self/fd/0 ntfsmeta.img

Using only bzip2 instead of tar+bzip2 has also a size overhead, perhaps 
40-60%.

I will probably also add special image format support for ntfsclone 
metadata images too, so the broken tar usage can be avoided once and for all.

BTW, here is when I reported this problem on bug-tar and the fix is in the reply.
http://lists.gnu.org/archive/html/bug-tar/2005-03/msg00004.html

Created attachment 48417 [details]
ntfsmeta-data, second try

The file again, this time only bzip was used

This image unpacked fine but there isn't anything wrong with it. It can be
perfectly loopback mounted, resized, etc.

BTW, actually if ntfsclone --metadata works then the image can't have corrupt
journal since ntfsclone read-write mounts the clonned image in a 2nd pass to
zero out unused, unimportant metadata and resident files so the image will be
much better compressable. However if this happens then the metadata image would
be still perfectly usable for analyses: just the compressed image will be
somewhat bigger.

BTW2, inconsistent NTFS, corrupted and left inconsistent by the Windows NTFS
driver, is quite normal and expected to be met here and there. But chkdsk /f and
two Windows reboot should cure these and all other cases, I'm aware of, get
fixed by ntfsresize.

Is there a possibility to force a chkdsk /f on the problematic drive from Linux?

I'm thinking of a procedure like this:
- ntfsresize fails
- yast calls ntfs_force_fsck_on_next_boot
- yast tells the user "Your windows partition was corrupt. Please boot into
windows once to fix it. After that, you can resize the windows partition."
- user boots windows
- windows automatically runs chkdsk /f on the drive
- user reboots and everything is fine.

IIRC ntfsfix does something similar, but if it just could set a flag "check this
disk" and nothing else, we could be sure nothing bad will happen.

As a side question, will forcing fsck with the method above possibly damage a
hibernated windows partition?

> Is there a possibility to force a chkdsk /f on the problematic drive from
> Linux?

ntfsfix. In some cases there wouldn't be even need to boot Windows
afterwards.

But there are several different error scenarios and you just can't run
always ntfsfix unconditionally because sometimes it doesn't make sense
and could even cause damage.

> I'm thinking of a procedure like this:
> - ntfsresize fails

Actually ntfsresize detects inconsistencies and existing problems. It's
intentional, it has a built in mini-fsck. When you say it's a failure then
people think your tools are crap and damages NTFS. No. It's the state of
the current NTFS which is crap and Linux detects it correctly meanwhile
even Microsoft isn't able to do so. I think this is hardly a failure but a
strength to vigilantly protect user's data ;-)

> - yast calls ntfs_force_fsck_on_next_boot
> - yast tells the user "Your windows partition was corrupt. Please boot into
> windows once to fix it. After that, you can resize the windows partition."

Unconditinally running ntfs_force_fsck_on_next_boot without user's
confirmation is bad.

Unconditionally running it even with user's confirmation is also bad
because it depends on the failure scenario when you can run it, a.k.a.
when it make sense.

> - user boots windows
> - windows automatically runs chkdsk /f on the drive
> - user reboots and everything is fine.

It's not. After running chkdsk /f, user needs to reboot Windows twice. So
Windows needs to be rebooted altogether three times before he/she could boot
Linux. Why three times?

  1st time: to run chkdsk /f (schedule the fs check)
  2nd time: boot time visible chkdsk, 1st pass
  3rd time: boot time unvisible chkdsk, 2nd pass

If the user doesn't follow this exactly then the NTFS journal file won't be
clean and read-write mount will be refused by all Linux-NTFS progs code
(kernel driver and the ntfsprogs tools).

> IIRC ntfsfix does something similar, but if it just could set a flag
> "check this disk" and nothing else, we could be sure nothing bad will
> happen.

And it also resets the NTFS journal file.

> As a side question, will forcing fsck with the method above possibly
> damage a hibernated windows partition?

Well, technically ntfsfix won't because NTFS is in a consistent state.
Windows will destroy it next boot time because it will unconditinally apply
the hibernation file to a different consistent state. Windows hibernation
(note, this is not the much more popular standby!) leaves NTFS consistent
because if there is a problem with resume then user could just delete the
hibernation file and will have a clean filesystem.

If I have some time in the weekend then I'll add the corrupt journal support.

Problem appeared here as well. With Beta 4.

Adding YaST2 logs.

Will this be fixed until RC1 ? If not I think we at least should change the
error message to something understandable, I don't think that "System eror code
was: -3027" will tell anybody besides programmers what is going on. 

If not being getting this fixed in ntfsresize, can the message be change to a
more understandable error message for customers, somthing like 

"Resizing of the ntfs partition failed. Try to defrag the partition first and
run a fislestystem check under with windows before retrying"

?

Created attachment 48576 [details]
YaST2

YaST2

Created attachment 48577 [details]
dmesg

dmesg

Better error message would be: "Could not resize the NTFS partition because it
is corrupt. Please boot into windows, run chkdsk /f on the partition, reboot
into windows TWICE, then try resizing again."

Szaka, what do you think of the above error message suggestion for ntfsresize
returning an error? And what message should be given if ntfsresize offers less
shrinking than yast expected? Is there any case left where defragmenting can
help, like a badly fragmented $UsnJrnl?

One reboot into windows and running a ntfs defrag and the resizing of the
partition did not cause any problems here anymore.

> One reboot into windows and running a ntfs defrag and the resizing of the
> partition did not cause any problems here anymore.

Only boot into Windows would have been enough. Your log exactly says what I
predicted above:

  Apparently the NTFS journal file is unclean. Please shutdown Windows
  properly before using this software. If it wouldn't help then please
  report it to linux-ntfs-dev@lists.sf.net. Thank you.

When I added this message to ntfsresize, nobody knew yet that chkdsk fixes NTFS
in two reboot stages so Windows needs two reboots after doing a chkdsk /f (so if
ntfsresize detects corruption then user needs to reboot Windows altogether
__THREE__ times).

Ntfsresize also doesn't require defragmentation. This is mentioned in bold at
the top of the manual and the FAQ. The SUSE Guide is incorrect about this, users
needlessly do this step.

> Szaka, what do you think of the above error message suggestion for ntfsresize
> returning an error? 

There are over a hundred different reasons why resizing can be refused but yes,
the most often reason (error category) is corrupt NTFS. 

The second most popular is disks having bad sectors. To support this, ntfsresize
 must be used with the --bad-sectors option (more in the manual). 

Well, ntfsresize's exit() value should indicate the error category so front-ends
could easily decide and/or advice to the users what to do ...

BTW, did Anton forward my comments to you? It's also in the linux-ntfs-dev
archive: http://marc.theaimsgroup.com/?l=linux-ntfs-dev&m=112499337513443&w=2

> And what message should be given if ntfsresize offers less shrinking 
> than yast expected? 

YaST can't expect less than the used space, right? Otherwise resizing would be
destructive (unless ntfsresize would compress the filesystem -- NTFS supports
transparent compression). 

Currently the only situation when ntfsresize may offer less shrinking than the
used space is if the first sector (actually extent) of the MFT (NTFS superblock)
should be moved. Since this is placed always in the very beginning of the
partitions thus this case almost never happens. At least nobody ever reported to
me over ther last three years. If one ever will then I'll add the needed support.

> Is there any case left where defragmenting can help, like a badly fragmented
> $UsnJrnl?

Frankly speaking, defragmentation was always a non-issue. NTFS defragmenters
don't defrag partition, they defrag only most files but not all (some they don't
touch at all). Or in other words, they don't compact everything into the
beginning of the partition, they only move the different file parts close
together. Somewhere, anywhere. So ntfsresize supports moving most things since
end of 2003 and everything (except the above sceanario) since version 1.11.2.

The only thing for which defrag can have either a good OR bad impact is
shrinking to the absolute minimum value (used space). The NTFS superblock is
very dynamic and during heavy modifications, relocations it may grow (needs more
disk space) so it may not be possible to relocate everything needed to the less
space than one originally had. No guarantee that defrag helps or makes the
chance worse regarding this issue. Anyway, it also doesn't make much sense to do
this because Windows needs about 50-100 MB free space left to boot safely. When
I checked YaST last time, it correctly reserved some free space for this.

Regarding comment 37: I'll try to resize the filesystem now with beta4, let's 
see if it is working now. 

Thomas, please try to improve the YaST message in worst case. I do not see 
that I can help here in any way. 

The message will now also include the stderr output of ntfsresize, 
no idea if that will help. Otherwise there is not much I can do.