Bug 1219346

Summary: zram_submit_bio WARNING due to bcachefs REQ_OP_FLUSH
Product: [openSUSE] openSUSE Tumbleweed Reporter: David Disseldorp <ddiss>
Component: Kernel:FilesystemsAssignee: Kernel Filesystems <kernel-fs>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None    
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description David Disseldorp 2024-01-30 12:22:58 UTC
I ran into the following WARNING while doing a bcachefs fstests run against tumbleweed's 6.7.1-2-default kernel:

generic/068       [  395.418059][  T263] run fstests generic/068 at 2024-01-30 12:03:57
[  395.638334][T22189] bcachefs (zram0): going read-write
[  395.707629][T22212] bcachefs (zram1): mounting version 1.3: rebalance_work
[  395.712533][T22212] bcachefs (zram1): initializing new filesystem
[  395.717313][T22212] bcachefs (zram1): going read-write
[  395.722198][T22212] bcachefs (zram1): initializing freespace
[  397.884529][T22254] bcachefs (zram1): going read-write
[  398.093884][T22267] ------------[ cut here ]------------
[  398.095352][T22267] WARNING: CPU: 1 PID: 22267 at drivers/block/zram/zram_drv.c:1948 zram_submit_bio+0x1bd/0x980 [zram]
[  398.097604][T22267] Modules linked in: ext4 mbcache jbd2 loop dm_flakey dm_mod bcachefs lz4_compress xor lz4hc_compress libcrc32c raid6_pq zram virtio_blk
[  398.100412][T22267] CPU: 1 PID: 22267 Comm: fsstress Tainted: G        W          6.7.1-2-default #1 openSUSE Tumbleweed d50116cfdb1b14a701e904c894d8f1c040bf1146
[  398.103313][T22267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-3-gd478f380-rebuilt.opensuse.org 04/01/2014
[  398.105681][T22267] RIP: 0010:zram_submit_bio+0x1bd/0x980 [zram]
[  398.106940][T22267] Code: ff ff 83 44 24 14 01 45 31 c9 45 85 f6 0f 85 e8 fe ff ff 4c 8b 64 24 18 e9 25 07 00 00 0f b6 d0 83 fa 03 74 31 83 fa 09 74 2c <0f> 0b 48 8b 44 20
[  398.110897][T22267] RSP: 0018:ffffb8a3822c7ca8 EFLAGS: 00010297
[  398.112119][T22267] RAX: 0000000000000002 RBX: ffff9ba30095e800 RCX: ffff9ba30dbfb210
[  398.113774][T22267] RDX: 0000000000000002 RSI: 000000000002d23e RDI: ffff9ba30dbfb210
[  398.115360][T22267] RBP: ffff9ba31ad5c000 R08: 0000000000000001 R09: 0000000000004001
[  398.116957][T22267] R10: 000056356b09a2f1 R11: 000000000000038f R12: ffff9ba31ad5c000
[  398.118548][T22267] R13: 0000000000000000 R14: 0000000000000000 R15: ffff9ba31acce800
[  398.120117][T22267] FS:  00007fa7c7ee6040(0000) GS:ffff9ba437d00000(0000) knlGS:0000000000000000
[  398.121925][T22267] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  398.123307][T22267] CR2: 00007fa7c7c00000 CR3: 000000011cdc8000 CR4: 00000000000006f0
[  398.124906][T22267] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  398.126508][T22267] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  398.128094][T22267] Call Trace:
[  398.128752][T22267]  <TASK>
[  398.129364][T22267]  ? zram_submit_bio+0x1bd/0x980 [zram 306d1de59f59b3565665180c68ecbb123a2737f1]
[  398.131148][T22267]  ? __warn+0x81/0x130
[  398.131997][T22267]  ? zram_submit_bio+0x1bd/0x980 [zram 306d1de59f59b3565665180c68ecbb123a2737f1]
[  398.133849][T22267]  ? report_bug+0x171/0x1a0
[  398.134786][T22267]  ? handle_bug+0x3c/0x80
[  398.135650][T22267]  ? exc_invalid_op+0x17/0x70
[  398.136583][T22267]  ? asm_exc_invalid_op+0x1a/0x20
[  398.137579][T22267]  ? zram_submit_bio+0x1bd/0x980 [zram 306d1de59f59b3565665180c68ecbb123a2737f1]
[  398.138979][T22267]  ? __schedule+0x3e5/0x14b0
[  398.139577][T22267]  ? bio_associate_blkg_from_css+0xf8/0x340
[  398.140325][T22267]  ? queue_work_on+0x3b/0x50
[  398.140907][T22267]  __submit_bio+0xb3/0x170
[  398.141471][T22267]  submit_bio_noacct_nocheck+0x147/0x380
[  398.142192][T22267]  bch2_inode_flush_nocow_writes_async+0xc9/0x140 [bcachefs 87a9241f310455d8dd26e5519b093c57d3f4361d]
[  398.143652][T22267]  bch2_flush_inode.part.0+0xb3/0xd0 [bcachefs 87a9241f310455d8dd26e5519b093c57d3f4361d]
[  398.144954][T22267]  bch2_fsync+0x82/0xa0 [bcachefs 87a9241f310455d8dd26e5519b093c57d3f4361d]
[  398.146115][T22267]  __x64_sys_fdatasync+0x52/0x90
[  398.146742][T22267]  do_syscall_64+0x64/0xe0
Comment 1 David Disseldorp 2024-01-30 12:28:02 UTC
From a quick glance it looks as though bcachefs is submitting one or more REQ_OP_FLUSH bios:

55) void bch2_inode_flush_nocow_writes_async(struct bch_fs *c,
56)                                          struct bch_inode_info *inode,                               
57)                                          struct closure *cl)                                         
58) {
59)         struct nocow_flush *bio;
60)         struct bch_dev *ca;
61)         struct bch_devs_mask devs;
62)         unsigned dev;
63)
64)         dev = find_first_bit(inode->ei_devs_need_flush.d, BCH_SB_MEMBERS_MAX);
65)         if (dev == BCH_SB_MEMBERS_MAX)
66)                 return;
67)
68)         devs = inode->ei_devs_need_flush;
69)         memset(&inode->ei_devs_need_flush, 0, sizeof(inode->ei_devs_need_flush));
70)
71)         for_each_set_bit(dev, devs.d, BCH_SB_MEMBERS_MAX) {         
72)                 rcu_read_lock();
73)                 ca = rcu_dereference(c->devs[dev]);
74)                 if (ca && !percpu_ref_tryget(&ca->io_ref))
75)                         ca = NULL;
76)                 rcu_read_unlock();
77)
78)                 if (!ca)
79)                         continue;
80)
81)                 bio = container_of(bio_alloc_bioset(ca->disk_sb.bdev, 0,    
82)                                                     REQ_OP_FLUSH,                                               
83)                                                     GFP_KERNEL,                                                 
84)                                                     &c->nocow_flush_bioset),                                    
85)                                    struct nocow_flush, bio);                                   
86)                 bio->cl                 = cl;
87)                 bio->ca                 = ca;
88)                 bio->bio.bi_end_io      = nocow_flush_endio;                
89)                 closure_bio_submit(&bio->bio, cl);
90)         }
91) }

which makes it through to zram and is unhandled by zram_submit_bio():

1932) static void zram_submit_bio(struct bio *bio)
1933) {
1934)     struct zram *zram = bio->bi_bdev->bd_disk->private_data;
1935) 
1936)     switch (bio_op(bio)) {
1937)     case REQ_OP_READ:
1938)             zram_bio_read(zram, bio);
1939)             break;
1940)     case REQ_OP_WRITE:
1941)             zram_bio_write(zram, bio);
1942)             break;
1943)     case REQ_OP_DISCARD:
1944)     case REQ_OP_WRITE_ZEROES:
1945)             zram_bio_discard(zram, bio);
1946)             break;
1947)     default:
1948)             WARN_ON_ONCE(1);
1949)             bio_endio(bio);
1950)     }
1951) }
Comment 2 David Disseldorp 2024-02-01 10:00:44 UTC
(In reply to David Disseldorp from comment #1)
> From a quick glance it looks as though bcachefs is submitting one or more
> REQ_OP_FLUSH bios:

This is a bcachefs bug: it should be using the REQ_PREFLUSH flag instead of REQ_OP_FLUSH. It's fixed in mainline v6.8-rc2 via 3e44f325f6f75 ("bcachefs: fix incorrect usage of REQ_OP_FLUSH").

I've submitted the fix for Tumbleweed inclusion via https://kerncvs.suse.de/gitweb/?p=kernel-source.git;a=shortlog;h=refs/heads/users/ddiss/stable/for-next

Closing...