Bugzilla – Bug 116531
Reiserfs easily deadlocks on large machines under stress
Last modified: 2007-10-23 17:48:13 UTC
I can easily deadlock reiserfs by running reaim on a 16 core machine. Similar oops have been seen on a 8 CPU machine under normal abuild load. I'm setting this to critical because it points imho to a serious problem in 10.0 when the standard file system is not reliable. Problem happens with ACLs disabled too. Standard oops (see my post on linux-fsdevel for more). Also see bug #105377 NMI Watchdog detected LOCKUP on CPU13CPU 13 Modules linked in: Pid: 3601, comm: reaim Not tainted 2.6.13 RIP: 0010:[<ffffffff80419f7d>] <ffffffff80419f7d>{.text.lock.spinlock+20} RSP: 0018:ffff81013ac1dd00 EFLAGS: 00000082 RAX: 0000000000000000 RBX: ffffffff804c1820 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff81007da86670 RDI: ffffffff804c1828 RBP: ffff81006f021970 R08: 0000000000000000 R09: ffff81013ac1dd68 R10: ffff81023ffa3d90 R11: 0000000000000246 R12: ffffffff804c1828 R13: ffff8101387fd870 R14: ffff81013ac1dd10 R15: 00007fffff90e350 FS: 00002aaaaaf3b0a0(0000) GS:ffffffff805ffe80(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002aaaaaf1f7e8 CR3: 0000000136a36000 CR4: 00000000000006a0 Process reaim (pid: 3601, threadinfo ffff81013ac1c000, task ffff8101387fd870) Stack: 0000000000000286 ffffffff80417b5b 0000000000000000 ffff8101387fd870 ffffffff80131fe0 0000000000000000 0000000000000000 ffff810137ef3dd0 ffff81006f021970 ffff81006f021970 Call Trace:<ffffffff80417b5b>{__down+75} <ffffffff80131fe0>{default_wake_function+0} <ffffffff80419b19>{__down_failed+53} <ffffffff8041a0e6>{.text.lock.kernel_lock+25} <ffffffff801c249c>{reiserfs_link+44} <ffffffff8018d6d5>{vfs_link+245} <ffffffff8019132b>{sys_link+187} <ffffffff80419bc7>{__up_wakeup+53} <ffffffff80197cd1>{dput+33} <ffffffff80190d27>{sys_unlink+295} <ffffffff8017ceea>{fd_install+58} <ffffffff8010d95e>{system_call+126} Code: f3 90 80 3f 00 7e f9 e9 90 fd ff ff f3 90 80 3f 00 7e f9 e9 console shuts up ... <0>Kernel panic - not syncing: Aiee, killing interrupt handler! <0>Rebooting in 30 seconds..SESC[0mESC[1m^@ESC[01;00H^@
Our current theory is this is due to a lock inversion problem between the BKL and the inode semaphore. I'm building a kernel on bellini now to try. If this doesn't work, I'll use Andi's patch to get a stack trace of all cpus, it should lead us quickly to the deadlock.
If you wait a bit the current kernel has the patch applied and reaim is running. So eventually it should deadlock and you should get a full backtrace.
Andi, is this still an issue?
Probably not. I let avitrano run reaim tests for a few days on reiserfs and there weren't any deadlocks (well it eventually hung, but that was probably a different problem) Let's close this one for now. I assume it's already fixed.
I should add avitrano had only half as many cores as the 16 core system where I saw the problem. Perhaps it would be a good idea to run some testing on a larger core system again. But I assume that will happen during SLES10 testing anyways.
when i use stress (http://weather.ou.edu/%7Eapw/projects/stress/, running as: stress -v -c 8 -i 4 -m 32 -d 16) it crash my server (2x Dual Core Opteron, 16G RAM) after 2-6 hours with reiserfs. with one of result: - 100% disk usage - io precesses died in "Uninterruptible sleep" - fs crash if i use xfs server "stable" about 4 days (stress test stoped manualy) P.S. i used some version 2.6.16 and 2.6.15 (from factory)
Andrew - This bug has been marked resolved and the lock inversion problem has been addressed. If you're seeing ReiserFS problems with a recent release kernel, please open a separate bug.
done, see https://bugzilla.novell.com/show_bug.cgi?id=173558
setting whiteboard
Just for the record: Patch: patches.common/reiserfs-bkl-inversion included, enabled, and released in: SLES-8 SP4 kernel update 2.4.21-325 dated Oct 04, 2007 & released Oct 12, 2007. Setting Whiteboard Status for SLES8 --> released