Bug 858233 - Patch openSUSE-2013-1034 breaks xfs_growfs
Patch openSUSE-2013-1034 breaks xfs_growfs
Classification: openSUSE
Product: openSUSE 12.3
Classification: openSUSE
Component: Kernel
x86-64 openSUSE 12.3
: P5 - None : Normal (vote)
: ---
Assigned To: E-mail List
E-mail List
Depends on:
  Show dependency treegraph
Reported: 2014-01-10 11:20 UTC by wouter koersen
Modified: 2015-02-19 05:34 UTC (History)
8 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Note You need to log in before you can comment on or make changes to this bug.
Description wouter koersen 2014-01-10 11:20:50 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36

Recently I migrated all my filesystems to XFS. This works just fine, but I found I could not grow a filesystem while using kernel 3.7.10-1.24.1-desktop. After downgrading to 3.7.10-1.16-desktop xfs_growfs works.

Symtoms while using kernel 3.7.10-1.24.1-desktop:

xfs_growfs error message:
xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Cannot allocate memory

Message in system log:
Jan 07 12:55:58 linux-zvf1.ka.haagnet.net kernel: XFS (dm-5): _xfs_buf_find: Block out of range: block 0x4000001, EOFS 0x4000000

Reproducible: Always

Steps to Reproduce:
1.create a logical volume
2.create an xfs filesystem on the volume
3.enlarge the logical volume
4.run xfs_growfs on the filesystem
Actual Results:  
xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Cannot allocate memory

error in syslog:
Jan 07 12:55:58 linux-zvf1.ka.haagnet.net kernel: XFS (dm-5): _xfs_buf_find: Block out of range: block 0x4000001, EOFS 0x4000000

Expected Results:  
Enlarged filesystem
No errors

Works with kernel 3.7.10-1.16-desktop
Comment 1 Paul Neuwirth 2014-01-24 14:51:12 UTC
Same problem here, same kernel.
corresponding entries in syslog:
2014-01-24T15:37:28.888724+01:00 alpha kernel: [1121931.525327] XFS (dm-7): _xfs_buf_find: Block out of range: block 0xc580001, EOFS 0x8200000
2014-01-24T15:37:28.888758+01:00 alpha kernel: [1121931.525336] XFS (dm-7): _xfs_buf_find: Block out of range: block 0xc580001, EOFS 0x8200000
Comment 2 Urs Beyerle 2014-02-03 14:24:24 UTC
I had to reboot into the old 3.7.10-1.16-default kernel to grow the xfs filesystem. A fix for this problem is highly appreciated!
Comment 3 Robert Boßecker 2014-02-10 15:37:39 UTC
Unfortunately also the kernel 3.7.10-1.28.1 compiled on Tue Feb 04, still contains this bug.
Still stuck with 3.7.10-1.16.
Comment 4 Takashi Iwai 2014-03-17 10:35:25 UTC
Does openSUSE 13.1 work for you instead?
Comment 5 wouter koersen 2014-03-17 13:56:27 UTC
(In reply to comment #4)
> Does openSUSE 13.1 work for you instead?

Sorry, I don't have 13.1 yet. If you can't reproduce the error using the steps described in the bug report it seems like it's fixed. The XFS version on 12.3 is: xfs-1.1.2-5.1.1.x86_64 (in rpm -qi).
Comment 6 Robert Boßecker 2014-03-17 18:11:01 UTC
openSUSE 13.1 is at the moment not an option due to other issues, but the xfs_growfs works on 13.1.

This xfs-"Bug" was also seen in SLES11-SP2 and SLES11-SP3 kernels but got fixed.
SR# 10858518821
Comment 7 Robert Boßecker 2014-03-17 18:14:31 UTC
The package mentioned by wouter koersen "xfs-1.1.2-5.1.1" is not related to the xfs-filesystem.

The bug is also not affected by the xfsprogs package, it is just kernel-version related.
Comment 8 Takashi Iwai 2014-03-19 08:48:59 UTC
OK, since it's not seen in the recent systems, let's close the bug.  Please reopen if you still encounter the same problem again.  Thanks.
Comment 9 Urs Beyerle 2014-03-19 10:48:54 UTC
The problem does still exist in openSUSE 12.3. It was never fixed. To summarize again: 

- kernel 3.7.10-1.16 works
- kernel 3.7.10-1.28 does not work

Since it seems to be fixed in SLES11-SP2 and SLES11-SP3 kernels, I would be very interested to see what was the patch there?

I have reopened the bug again.
Comment 10 Robert Boßecker 2014-03-19 10:52:42 UTC
The current situation is not a solution.
To prevent the problem only the workaround to downgrade the kernel to version 3.7.10-1.16 gives a system which is "maintainable".

Every time installing updates the kernel and the corresponding packages need to be downgraded manually.
Comment 11 Takashi Iwai 2014-03-19 11:36:28 UTC
OK, thanks for updating.

Looking at the log, this seems like the result of the security fix
commit eb178619f930fa2ba2348de332a1ff1c66a31424
   xfs: fix _xfs_buf_find oops on blocks beyond the filesystem end

The upstream 3.7.x reverted this patch.

commit c3793e0d94af2071c6f3842dcdb5ea08bd011354
Author: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date:   Thu Feb 14 11:22:53 2013 -0800

    Revert: xfs: fix _xfs_buf_find oops on blocks beyond the filesystem end
    This reverts commit a56040731e5b00081c6d6c26b99e6e257a5d63d7 which was
    commit eb178619f930fa2ba2348de332a1ff1c66a31424 upstream.
    It has been reported to cause problems:
So, we should do it, too.

I reverted the affecting patch now in kernel git openSUSE-12.3 branch.
Please try OBS Kernel:openSUSE-12.3 repo later.  If the problem still persists with the newer kernel with the commit 60c70ed216, please reopen.  Thanks.
Comment 12 Marcus Meissner 2014-03-19 13:02:13 UTC
original security bug 858233
Comment 13 Mark Tinguely 2014-03-19 20:47:58 UTC
I would not remove the patch!

 Revert: xfs: fix _xfs_buf_find oops on blocks beyond the filesystem end

it opens you up to NULL pointer dereferences on huge block numbers.

xfs_growfs() cannot call xfs_buf_find(). In earlier kernels it did via xfs_buf_get(). The commit, fd23683c, in v3.7-rc1-42-gfd23683 changed those to direct reads.
Comment 18 Takashi Iwai 2014-03-20 07:27:28 UTC
(In reply to comment #13)
> I would not remove the patch!
>  Revert: xfs: fix _xfs_buf_find oops on blocks beyond the filesystem end
> it opens you up to NULL pointer dereferences on huge block numbers.
> xfs_growfs() cannot call xfs_buf_find(). In earlier kernels it did via
> xfs_buf_get(). The commit, fd23683c, in v3.7-rc1-42-gfd23683 changed those to
> direct reads.

You mean 3.8-rc1, right?  If it were 3.7-rc1, it should have been already fixed :)

I backported that commit and the previous one to apply cleanly, and revert the revert.  The test kernel packages are being built in OBS home:tiwai:bnc858233 repo.  Please test whether this fixes the issue.

Once when confirmed, I'll push this to git tree.
Comment 19 Mark Tinguely 2014-03-20 13:28:47 UTC
Yes, my mistake, I was using "git describe" on the patch from a OSS kernel and it does not always match up with the real releases. I did look on the linux-3.7-stable version on kernel.org and the  commit "fd23683c" had not yet made it into Linux-3.7-stable, so yes this patch would be first seen in Linux-3.8.

You can check to see which patches were used is SLES11SP3, but these look

author	Dave Chinner <dchinner@redhat.com>	2012-04-23 05:58:55 (GMT)
committer	Ben Myers <bpm@sgi.com>	2012-05-14 21:20:51 (GMT)
commit	7ca790a507a9288ebedab90a8e40b9afa8e4e949 (patch)
tree	70ad7655655e3f5e021b3c80f5c102f4399bbd68 /fs/xfs/xfs_fsops.c
parent	a8acad70731e7d0585f25f33f8a009176f001f70 (diff)
xfs: kill xfs_read_buf()
xfs_read_buf() is effectively the same as xfs_trans_read_buf() when called outside a transaction context. The error handling is slightly different in that xfs_read_buf stales the errored buffer it gets back, but there is probably good reason for xfs_trans_read_buf() for doing this. Hence update xfs_trans_read_buf() to the same error handling as xfs_read_buf(), and convert all the callers of xfs_read_buf() to use the former function. We can then remove xfs_read_buf(). Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Mark Tinguely <tinguely@sgi.com> Signed-off-by: Ben Myers <bpm@sgi.com> 

author	Dave Chinner <dchinner@redhat.com>	2012-11-12 11:53:59 (GMT)
committer	Ben Myers <bpm@sgi.com>	2012-11-13 22:40:43 (GMT)
commit	fd23683c3b1ab905cba61ea2981c156f4bf52845 (patch)
tree	1ea38537f14d0ff537ff15a97cedd244a18890e9 /fs/xfs/xfs_fsops.c
parent	b64f3a390d3477517cbff7d613e551705540769b (diff)
xfs: growfs: use uncached buffers for new headers
When writing the new AG headers to disk, we can't attach write verifiers because they have a dependency on the struct xfs-perag being attached to the buffer to be fully initialised and growfs can't fully initialise them until later in the process. The simplest way to avoid this problem is to use uncached buffers for writing the new headers. These buffers don't have the xfs-perag attached to them, so it's simple to detect in the write verifier and be able to skip the checks that need the xfs-perag. This enables us to attach the appropriate buffer ops to the buffer and hence calculate CRCs on the way to disk. IT also means that the buffer is torn down immediately, and so the first access to the AG headers will re-read the header from disk and perform full verification of the buffer. This way we also can catch corruptions due to problems that went undetected in growfs. Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by Rich Johnston <rjohnston@sgi.com> Signed-off-by: Ben Myers <bpm@sgi.com>
Comment 20 Takashi Iwai 2014-03-20 13:46:09 UTC
Thanks.  As mentioned in comment 18, I backported the commit fd23683c3b.  But this patch requires another clean up patch b64f3a390d3 (xfs: use btree block initialisation functions in growfs), so I took it, too, instead of taking SP3 one.  Unmodified patches are much easier to maintain in the case like this (between 3.7 and 3.8).

Let's see whether the test package works.
Comment 22 wouter koersen 2014-03-21 12:17:06 UTC
I'm running a kernel from the http://download.opensuse.org/repositories/home:/tiwai:/bnc858233/standard/ repo and xfs_growfs works for me.

Linux linux-zvf1.ka.haagnet.net 3.7.10-1-desktop #1 SMP PREEMPT Thu Mar 20 07:12:37 UTC 2014 (95b5819) x86_64 x86_64 x86_64 GNU/Linux.
Comment 23 Takashi Iwai 2014-03-21 12:54:31 UTC
Thanks for checking.  Now the commits are merged to git tree openSUSE-12.3 branch.  You can use the fixed kernel from OBS Kernel:openSUSE-12.3 repo later.
Comment 24 wouter koersen 2014-03-21 14:37:40 UTC
Thanks for fixing this bug. I'm impressed by the bug fixing process.
Comment 25 Urs Beyerle 2014-03-21 14:57:53 UTC
Just to confirm. The problem is fixed in kernel 3.7.10-95.1.g60c70ed from Kernel:openSUSE-12.3:

* Wed Mar 19 2014 tiwai@suse.de
- Delete patches.fixes/xfs-fix-xfs_buf_find-oops-on-blocks-beyond-the-filesystem-end.
  It turned out that this patch causes regressions (bnc#858233)
  The upstream 3.7.x also reverted it in the end (commit c3793e0d94af2).
- commit 60c70ed

I'm looking forward to see it in update/12.3. Thanks!
Comment 26 Takashi Iwai 2014-03-21 15:00:19 UTC
Note that there are other commits again making this patch back but with more proper fixes.  It was committed today, so the package should be available in tomorrow or later.
Comment 27 Urs Beyerle 2014-03-24 08:54:00 UTC
Bug still fixed with kernel 3.7.10-97.gd1d38c4 from Kernel:openSUSE-12.3:

* Thu Mar 20 2014 tiwai@suse.de
- xfs: growfs: use uncached buffers for new headers (bnc#858233).
- xfs: use btree block initialisation functions in growfs
- commit 4ade8a9

Comment 28 Takashi Iwai 2014-03-24 08:59:31 UTC
Thanks for confirmation!
Comment 29 Swamp Workflow Management 2014-05-19 12:06:29 UTC
openSUSE-SU-2014:0677-1: An update that solves 16 vulnerabilities and has 10 fixes is now available.

Category: security (important)
Bug References: 733022,811746,833968,837111,851426,852652,852967,858233,858638,858869,858870,858872,860835,862145,863335,864025,866102,868653,869414,869898,871148,871252,871325,873717,875690,875798
CVE References: CVE-2013-4254,CVE-2013-4579,CVE-2013-6885,CVE-2014-0101,CVE-2014-0196,CVE-2014-0691,CVE-2014-1438,CVE-2014-1444,CVE-2014-1445,CVE-2014-1446,CVE-2014-1690,CVE-2014-1737,CVE-2014-1738,CVE-2014-1874,CVE-2014-2523,CVE-2014-2672
Sources used:
openSUSE 12.3 (src):    kernel-docs-3.7.10-1.32.2, kernel-source-3.7.10-1.32.1, kernel-syms-3.7.10-1.32.1