Bug 116531

Summary: Reiserfs easily deadlocks on large machines under stress
Product: [openSUSE] SUSE LINUX 10.0 Reporter: Andreas Kleen <ak>
Component: KernelAssignee: Chris L Mason <mason>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P5 - None CC: aj, jeffm, sitnikov
Version: RC 2   
Target Milestone: ---   
Hardware: x86-64   
OS: All   
Whiteboard: released:kernel:sles8 affected:sles8 applied:sles8
Found By: Other Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Andreas Kleen 2005-09-12 15:38:49 UTC 
I can easily deadlock reiserfs by running reaim on a 16 core
machine. Similar oops have been seen on a 8 CPU machine under
normal abuild load.

I'm setting this to critical because it points imho to a serious
problem in 10.0 when the standard file system is not reliable.

Problem happens with ACLs disabled too.

Standard oops (see my post on linux-fsdevel for more). Also see
bug #105377
  
NMI Watchdog detected LOCKUP on CPU13CPU 13 
Modules linked in:
Pid: 3601, comm: reaim Not tainted 2.6.13
RIP: 0010:[<ffffffff80419f7d>] <ffffffff80419f7d>{.text.lock.spinlock+20}
RSP: 0018:ffff81013ac1dd00  EFLAGS: 00000082
RAX: 0000000000000000 RBX: ffffffff804c1820 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff81007da86670 RDI: ffffffff804c1828
RBP: ffff81006f021970 R08: 0000000000000000 R09: ffff81013ac1dd68
R10: ffff81023ffa3d90 R11: 0000000000000246 R12: ffffffff804c1828
R13: ffff8101387fd870 R14: ffff81013ac1dd10 R15: 00007fffff90e350
FS:  00002aaaaaf3b0a0(0000) GS:ffffffff805ffe80(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaaf1f7e8 CR3: 0000000136a36000 CR4: 00000000000006a0
Process reaim (pid: 3601, threadinfo ffff81013ac1c000, task ffff8101387fd870)
Stack: 0000000000000286 ffffffff80417b5b 0000000000000000 ffff8101387fd870 
       ffffffff80131fe0 0000000000000000 0000000000000000 ffff810137ef3dd0 
       ffff81006f021970 ffff81006f021970 
Call Trace:<ffffffff80417b5b>{__down+75} <ffffffff80131fe0>{default_wake_function+0}
       <ffffffff80419b19>{__down_failed+53}
<ffffffff8041a0e6>{.text.lock.kernel_lock+25}
       <ffffffff801c249c>{reiserfs_link+44} <ffffffff8018d6d5>{vfs_link+245}
       <ffffffff8019132b>{sys_link+187} <ffffffff80419bc7>{__up_wakeup+53}
       <ffffffff80197cd1>{dput+33} <ffffffff80190d27>{sys_unlink+295}
       <ffffffff8017ceea>{fd_install+58} <ffffffff8010d95e>{system_call+126}
       

Code: f3 90 80 3f 00 7e f9 e9 90 fd ff ff f3 90 80 3f 00 7e f9 e9 
console shuts up ...
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!
 <0>Rebooting in 30 seconds..SESC[0mESC[1m^@ESC[01;00H^@
Comment 1 Chris L Mason 2005-09-12 17:00:07 UTC 
Our current theory is this is due to a lock inversion problem between the BKL
and the inode semaphore.  I'm building a kernel on bellini now to try.

If this doesn't work, I'll use Andi's patch to get a stack trace of all cpus, it
should lead us quickly to the deadlock.
Comment 2 Andreas Kleen 2005-09-12 17:03:23 UTC 
If you wait a bit the current kernel has the patch applied and reaim
is running. So eventually it should deadlock and you should get a full
backtrace.
Comment 3 Chris L Mason 2006-02-16 00:27:14 UTC 
Andi, is this still an issue?
Comment 4 Andreas Kleen 2006-02-16 12:17:47 UTC 
Probably not. I let avitrano run reaim tests for a few days on reiserfs
and there weren't any deadlocks

(well it eventually hung, but that was probably a different problem)

Let's close this one for now. I assume it's already fixed.
Comment 5 Andreas Kleen 2006-02-16 12:18:55 UTC 
I should add avitrano had only half as many cores as the 16 core system
where I saw the problem.

Perhaps it would be a good idea to run some testing on a larger core 
system again. But I assume that will happen during SLES10 testing anyways.
Comment 6 Andrew Sitnikov 2006-05-08 13:35:31 UTC 
when i use stress (http://weather.ou.edu/%7Eapw/projects/stress/, running as: stress -v -c 8 -i 4 -m 32 -d 16) it crash my server (2x Dual Core Opteron, 16G RAM) after 2-6 hours with reiserfs. with one of result:
 - 100% disk usage
 - io precesses died in "Uninterruptible sleep"
 - fs crash

if i use xfs server "stable" about 4 days (stress test stoped manualy)

P.S. i used some version 2.6.16 and 2.6.15 (from factory)
Comment 7 Jeff Mahoney 2006-05-08 14:04:13 UTC 
Andrew -

This bug has been marked resolved and the lock inversion problem has been addressed. If you're seeing ReiserFS problems with a recent release kernel, please open a separate bug.
Comment 8 Andrew Sitnikov 2006-05-08 15:52:43 UTC 
done, see https://bugzilla.novell.com/show_bug.cgi?id=173558
Comment 9 Anja Stock 2007-10-01 12:02:23 UTC 
setting whiteboard
Comment 10 Klaus Wagner 2007-10-23 17:48:13 UTC 
Just for the record:

Patch:  patches.common/reiserfs-bkl-inversion

included, enabled, and released in:

  SLES-8 SP4 kernel update 2.4.21-325
  dated Oct 04, 2007 & released Oct 12, 2007.

Setting Whiteboard Status for SLES8 --> released