Looka like btrfs support is broken.

It forgets to pass the required '@/'.

><rescue> mount -vo subvol=var/spool,ro /dev/sda2 /sysroot/
[  113.852047] BTRFS info (device sda2): disk space caching is enabled
[  113.852869] BTRFS: has skinny extents
mount: mount(2) failed: No such file or directory
><rescue> mount -vo subvol=@var/spool,ro /dev/sda2 /sysroot/
[  116.082642] BTRFS info (device sda2): disk space caching is enabled
[  116.083476] BTRFS: has skinny extents
mount: mount(2) failed: No such file or directory
><rescue> mount -vo subvol=@/var/spool,ro /dev/sda2 /sysroot/
[  123.995343] BTRFS info (device sda2): disk space caching is enabled
[  123.996238] BTRFS: has skinny extents
mount: /dev/sda2 mounted on /sysroot.

openSUSE-13.2rc1 does not have @/ in fstab, while SLE12 has it.

I ran into this problem while using virt-v2v, and started a thread about it here: https://www.redhat.com/archives/libguestfs/2014-April/msg00071.html

The outcome of that thread made me believe this is specifically due to the SUSE naming convention of /@. As upstream didn't seem to be interested in solving the issue, I worked around it in virt-v2v.

The problem itself can be seen with guestfish:

xen124:/var/lib/xen/images/sles12 # guestfish
><fs> add_drive /var/lib/xen/images/sles12/disk0.raw
><fs> launch
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ --:--
><fs> inspect-os 
/dev/sda2
btrfsvol:/dev/sda2/@
><fs> 

Since inspect-os returns two possible root devices, other tools end up believing this is a multi-boot system. Since these are actually the same partition, my virt-v2v patch checks this, and drops the 'btrfsvol:...' entry so the v2v process only sees one /dev/sd* entry.

In order for the rest of these tools to not encounter the same problem, we'll have to strip out the extra entry somewhere earlier in the tool chain (e.g. inside guestfs).

For the record, here's my virt-v2v patch:

Index: virt-v2v-0.9.1/v2v/virt-v2v.pl
===================================================================
--- virt-v2v-0.9.1.orig/v2v/virt-v2v.pl
+++ virt-v2v-0.9.1/v2v/virt-v2v.pl
@@ -693,6 +693,14 @@ sub inspect_guest

     @roots = sort @roots;

+    # If using btrfs, it is possible to have the same partition listed using
+    # two different names (e.g. '/dev/sda2' and 'btrfsvol:/dev/sda2/@').
+    # In this case, filter out the duplicate 'btrfsvol' entry.
+    if ((@roots == 2) && ("btrfsvol:$roots[0]/@" eq "$roots[1]")) {
+        @roots = grep(/^btrfsvol/, @roots);
+    }
+
+
     my $root_dev;

     if(@roots == 0) {

http://dist.suse.de/ibs/home:/olh:/sle12/SLE_12/ contains updated guestfs packages with this change:

diff --git a/daemon/btrfs.c b/daemon/btrfs.c
index 7a4d43d..ceea593 100644
--- a/daemon/btrfs.c
+++ b/daemon/btrfs.c
@@ -356,6 +356,7 @@ do_btrfs_subvolume_list (const mountable_t *fs)
     ADD_ARG (argv, i, str_btrfs);
     ADD_ARG (argv, i, "subvolume");
     ADD_ARG (argv, i, "list");
+    ADD_ARG (argv, i, "-a");
     ADD_ARG (argv, i, fs_buf);
     ADD_ARG (argv, i, NULL);
 
diff --git a/src/listfs.c b/src/listfs.c
index ffb0adc..db127ea 100644
--- a/src/listfs.c
+++ b/src/listfs.c
@@ -176,9 +183,12 @@ check_with_vfs_type (guestfs_h *g, const char *device, struct stringsbuf *sb)
 
     for (size_t i = 0; i < vols->len; i++) {
       struct guestfs_btrfssubvolume *this = &vols->val[i];
+      char *btrfssubvolume_path = this->btrfssubvolume_path;
+      guestfs___trace(g, "XXX %s(%u) device %s btrfssubvolume_path %s", __func__, __LINE__, device, btrfssubvolume_path);
+      if (STREQ(btrfssubvolume_path, "<FS_TREE>/@")) btrfssubvolume_path = "/@";
       guestfs___add_sprintf (g, sb,
                              "btrfsvol:%s/%s",
-                             device, this->btrfssubvolume_path);
+                             device, btrfssubvolume_path);
       guestfs___add_string (g, sb, "btrfs");
     }

(In reply to Olaf Hering from comment #6)
> http://dist.suse.de/ibs/home:/olh:/sle12/SLE_12/ contains updated guestfs
> packages with this change:

I updated my server with these 3 packages from this repo;
guestfs-data-1.26.9-153.olh_sle12.1.x86_64.rpm
guestfs-tools-1.26.9-153.olh_sle12.1.x86_64.rpm
libguestfs0-1.26.9-153.olh_sle12.1.x86_64.rpm

The output from running 'virt-ls -a sles12.qcow2 /tmp' was;

"libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path <FS_TREE>/@
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/boot/grub2/i386-pc
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/boot/grub2/x86_64-efi
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/home
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/opt
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/srv
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/tmp
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/usr/local
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/crash
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/lib/mailman
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/lib/named
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/lib/pgsql
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/log
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/opt
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/spool
libguestfs: trace: XXX check_with_vfs_type(180) device /dev/sda2 btrfssubvolume_path @/var/tmp
virt-ls: multi-boot operating systems are not supported

If using guestfish '-i' option, remove this option and instead
use the commands 'run' followed by 'list-filesystems'.
You can then mount filesystems you want by hand using the
'mount' or 'mount-ro' command.

If using guestmount '-i', remove this option and choose the
filesystem(s) you want to see by manually adding '-m' option(s).
Use 'virt-filesystems' to see what filesystems are available.

If using other virt tools, multi-boot operating systems won't work
with these tools.  Use the guestfish equivalent commands
(see the virt tool manual page)."

So it still fails. The image is a Xen sles12 PV QCOW2 image.

(In reply to Olaf Hering from comment #6)
> +    ADD_ARG (argv, i, "-a");

Instead of "-a" the "-o" option to "print only subvolumes bellow specified <path>" seems to work better. I does not print the root subvolume nor does it print its <FS_TREE> prefix. I wonder how reliable it will be in the future.

(In reply to Olaf Hering from comment #8)
> (In reply to Olaf Hering from comment #6)
> > +    ADD_ARG (argv, i, "-a");
> 
> Instead of "-a" the "-o" option to "print only subvolumes bellow specified
> <path>" seems to work better. I does not print the root subvolume nor does
> it print its <FS_TREE> prefix. I wonder how reliable it will be in the
> future.

Yes, the option -o appears to fix it. rpm Release 155+ will have this change.

David,

is the "-o" option the correct fix, is it reliable longterm?

https://www.redhat.com/archives/libguestfs/2014-October/msg00132.html

The -o option compares the 'top level id' of the subvolumes, so it's possible to miss some subvolumes if they are created with a different subvolume mounted at that time. It's not true in your case, as all are created either with @ mounted or /.

I think that -a is the right way to go, as it should print the full paths regardless of other conditions (snapshot, subvolume, top level, default subvol). What seems like a bug is that the listing in commen 7 lacks the FS_TREE prefix.

(In reply to David Sterba from comment #11)
> I think that -a is the right way to go, as it should print the full paths
> regardless of other conditions (snapshot, subvolume, top level, default
> subvol). What seems like a bug is that the listing in commen 7 lacks the
> FS_TREE prefix.

There is a FS_TREE prefix?

In any case, the proper fix for this bug is to detect if a given disk partition and a subvolume listed with 'btrfs subvolume list -a $mnt' are the same. If such support is missing from btrfs, it has to be added.

For the time being we will use the -o flag until the subvolume handling in SLE12 is done properly.

Thanks. MR accepted but it might take a few weeks until QA gets to this package. Let me know if the impact of these bugs justify a higher priority.

Only FYI:

I think there are some misunderstandings about '/@' in
https://www.redhat.com/archives/libguestfs/2014-April/msg00080.html
that reads (excerpt):
------------------------------------------------------------------------------
From: Mike Latimer <mlatimer suse com>
...
On Monday, April 28, 2014 10:02:54 AM Matthew Booth wrote:
...
> It looks like /dev/sda2 is both the root of the btrfs filesystem
> and a subvolume called '@'.
...
> Is '@' btrfs magic that means the root?
...
It looks like @ is just a naming convention that
we have chosen for the root of the btrfs volume.
------------------------------------------------------------------------------

I think both of you Mike Latimer and Matthew Booth do not
understand what '/@' actually is in SLE12.

As far as I understand it it is as follows:

The '/@' is just the btrfs default subvolume.
Any btrfs has a default subvolume.
The btrfs default subvolume is the one that gets mounted by default
when no subvolume is specified when mounting a btrfs.
Usually the default subvolume is the btrfs root (called "FS_TREE" by btrfs).
But for SLE12 we use '/@' as btrfs default subvolume:

On SLE12 we have:
------------------------------------------------------------------------------
# btrfs subvolume get-default /
ID 257 gen 2260 top level 5 path @

# findmnt -t btrfs | cut -b-61
TARGET                   SOURCE                              
/                        /dev/sda2[/@]                       
|-/.snapshots            /dev/sda2[/@/.snapshots]            
|-/var/spool             /dev/sda2[/@/var/spool]             
|-/var/tmp               /dev/sda2[/@/var/tmp]               
|-/var/lib/named         /dev/sda2[/@/var/lib/named]         
|-/var/opt               /dev/sda2[/@/var/opt]               
|-/usr/local             /dev/sda2[/@/usr/local]             
|-/tmp                   /dev/sda2[/@/tmp]                   
|-/var/lib/pgsql         /dev/sda2[/@/var/lib/pgsql]         
|-/home                  /dev/sda2[/@/home]                  
|-/var/crash             /dev/sda2[/@/var/crash]             
|-/opt                   /dev/sda2[/@/opt]                   
|-/var/log               /dev/sda2[/@/var/log]               
|-/boot/grub2/i386-pc    /dev/sda2[/@/boot/grub2/i386-pc]    
|-/var/lib/mailman       /dev/sda2[/@/var/lib/mailman]       
|-/srv                   /dev/sda2[/@/srv]                   
`-/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] 
------------------------------------------------------------------------------

On openSUSE 13.2 we have
------------------------------------------------------------------------------
# btrfs subvolume get-default /
ID 5 (FS_TREE)

# findmnt -t btrfs | cut -b-59
TARGET                   SOURCE                            
/                        /dev/sda2                         
|-/.snapshots            /dev/sda2[/.snapshots]            
|-/var/spool             /dev/sda2[/var/spool]             
|-/var/tmp               /dev/sda2[/var/tmp]               
|-/var/log               /dev/sda2[/var/log]               
|-/var/opt               /dev/sda2[/var/opt]               
|-/var/lib/mailman       /dev/sda2[/var/lib/mailman]       
|-/var/crash             /dev/sda2[/var/crash]             
|-/var/lib/named         /dev/sda2[/var/lib/named]         
|-/var/lib/pgsql         /dev/sda2[/var/lib/pgsql]         
|-/opt                   /dev/sda2[/opt]                   
|-/tmp                   /dev/sda2[/tmp]                   
|-/usr/local             /dev/sda2[/usr/local]             
|-/boot/grub2/i386-pc    /dev/sda2[/boot/grub2/i386-pc]    
|-/srv                   /dev/sda2[/srv]                   
`-/boot/grub2/x86_64-efi /dev/sda2[/boot/grub2/x86_64-efi]
------------------------------------------------------------------------------

To mount the btrfs filesystem root (i.e. the btrfs root subvolume)
on SLE12 one can use something like:
------------------------------------------------------------------------------
# mkdir /tmp/btrfsroot

# mount -t btrfs -o subvolid=0 /dev/sda2 /tmp/btrfsroot

# findmnt -t btrfs | cut -b-61
TARGET                   SOURCE                              
/                        /dev/sda2[/@]                       
|-/.snapshots            /dev/sda2[/@/.snapshots]            
|-/var/spool             /dev/sda2[/@/var/spool]             
|-/var/opt               /dev/sda2[/@/var/opt]               
|-/var/log               /dev/sda2[/@/var/log]               
|-/var/tmp               /dev/sda2[/@/var/tmp]               
|-/var/crash             /dev/sda2[/@/var/crash]             
|-/var/lib/named         /dev/sda2[/@/var/lib/named]         
|-/var/lib/pgsql         /dev/sda2[/@/var/lib/pgsql]         
|-/home                  /dev/sda2[/@/home]                  
|-/tmp                   /dev/sda2[/@/tmp]                   
| `-/tmp/btrfsroot       /dev/sda2                           
|-/srv                   /dev/sda2[/@/srv]                   
|-/var/lib/mailman       /dev/sda2[/@/var/lib/mailman]       
|-/usr/local             /dev/sda2[/@/usr/local]             
|-/boot/grub2/x86_64-efi /dev/sda2[/@/boot/grub2/x86_64-efi] 
|-/opt                   /dev/sda2[/@/opt]                   
`-/boot/grub2/i386-pc    /dev/sda2[/@/boot/grub2/i386-pc]

# ls /tmp/btrfsroot
@

# ls /tmp/btrfsroot/@
bin  boot  dev  etc  home  lib  lib64  mnt  opt  proc  root  run
sbin  selinux  .snapshots  srv  sys  tmp  usr  var

# ls /
bin  boot  dev  etc  home  lib  lib64  mnt  opt  proc  root  run
sbin  selinux  .snapshots  srv  sys  tmp  usr  var
------------------------------------------------------------------------------

Caution:
Do not confuse the btrfs root subvolume with the btrfs default subvolume
and do not confuse the btrfs filesystem root with what is mounted at
the root directory '/' or with what is mounted at another mount point.
With btrfs any subvolume can be mounted anywhere and as far as I know
only "findmnt" tells you what btrfs subvolume is mounted where.
My personal experience is that without findmnt I am lost.


Disclaimer:
I am really not a btrfs expert. Only by bad fortune I had to learn
something about btrfs the hard way - by trial and error :-(
which means I am not a real btrfs expert - only one who had
learned something about it...

To make it even more clear:

The '@' is just a name of a btrfs subvolume.
As far as I know '@' has no special meaning in btrfs.

The SLE12 btrfs default subvolume could also have been
named '/defltsubvol' or whatever one likes.

As far as I remember when you boot from a SLE12 snapshot
the btrfs default subvolume gets changed to the name of the
btrfs snapshot subvolume.

This means there is no such thing as THE btrfs default subvolume
for SLE12 (i.e. the btrfs default subvolume is not fixed).

The '/@' is just the btrfs default subvolume that is used
by default on SLE12 (note the two 'default' !) because another
btrfs subvolume could also be set as btrfs default subvolume.

(In reply to Johannes Meixner from comment #16)
> Caution:
> Do not confuse the btrfs root subvolume with the btrfs default subvolume
> and do not confuse the btrfs filesystem root with what is mounted at
> the root directory '/' or with what is mounted at another mount point.

Very interesting. Thanks for taking the time to explain that.

> With btrfs any subvolume can be mounted anywhere and as far as I know
> only "findmnt" tells you what btrfs subvolume is mounted where.
> My personal experience is that without findmnt I am lost.

The problem is, findmnt is used to examine mounted filesystems. In this case, the guestfs tools are trying to find the root filesystem in a virtual disk image. As the image is examined, partitions containing a btrfs filesystem are mounted by the physical partition device (e.g. /dev/sda2). As this will mount the 'default' subvolume, findmnt won't help us determine if we found the real root filesystem.

I just tested this, and if the default btrfs subvolume is not the root filesystem the only tool that appears to report the real root filesystem is `btrfs subvolume list -a`. Even if a strange subvolume is mounted (I tested @/tmp), `btrfs subvolume list -a` still reports all subvolumes on the device (with their full path):

# btrfs subvolume list -a btrfsroot/
ID 257 gen 39795 top level 5 path <FS_TREE>/@
ID 258 gen 18604 top level 257 path <FS_TREE>/@/boot/grub2/i386-pc
ID 259 gen 18604 top level 257 path <FS_TREE>/@/boot/grub2/x86_64-efi
ID 260 gen 18604 top level 257 path <FS_TREE>/@/home
ID 261 gen 18604 top level 257 path <FS_TREE>/@/opt
ID 262 gen 18604 top level 257 path <FS_TREE>/@/srv
ID 263 gen 39799 top level 257 path <FS_TREE>/@/tmp
ID 264 gen 18604 top level 257 path <FS_TREE>/@/usr/local
ID 265 gen 18604 top level 257 path <FS_TREE>/@/var/crash
ID 266 gen 18604 top level 257 path <FS_TREE>/@/var/lib/mailman
ID 267 gen 18604 top level 257 path <FS_TREE>/@/var/lib/named
ID 268 gen 18604 top level 257 path <FS_TREE>/@/var/lib/pgsql
ID 269 gen 39793 top level 257 path <FS_TREE>/@/var/log
ID 270 gen 18604 top level 257 path <FS_TREE>/@/var/opt
ID 271 gen 39793 top level 257 path <FS_TREE>/@/var/spool
ID 272 gen 39793 top level 257 path <FS_TREE>/@/var/tmp

In the above output, the lowest ID/level number, or shortest path corresponds to the root filesystem.

Is it safe to assume that subvolid=0 always equals the highest level subvolume - and therefore the root filesystem? If so, specifying that option during the mounting of the partition overrides the default setting and would ensure we get the correct subvolume.

# mount -t btrfs -o subvolid=0 /dev/disk/by-id/dm-name-loop0p2 btrfsroot/
# ls btrfsroot/
@
# ls btrfsroot/@/
bin  boot  data  data2  dev  etc  home  lib  lib64  mnt  opt  proc  root  run  sbin  selinux  srv  sys  tmp  usr  var

(In reply to Johannes Meixner from comment #17)
> The '/@' is just the btrfs default subvolume that is used
> by default on SLE12 (note the two 'default' !) because another
> btrfs subvolume could also be set as btrfs default subvolume.

Other than when booting from a snapshot, do you have any idea how common it is to change the default subvolume to something other than the root of the filesystem?

(In reply to Olaf Hering from comment #13)
> For the time being we will use the -o flag until the subvolume handling in
> SLE12 is done properly.

Given the details from Johannes, using the -o flag doesn't work if the default subvolume is not the root of the filesystem. In my test, @/tmp doesn't have any subvolumes, so -o doesn't have anything to report:

# btrfs subvolume list -o btrfsroot/
# btrfs subvolume list -a btrfsroot/
ID 257 gen 39795 top level 5 path <FS_TREE>/@
ID 258 gen 18604 top level 257 path <FS_TREE>/@/boot/grub2/i386-pc
ID 259 gen 18604 top level 257 path <FS_TREE>/@/boot/grub2/x86_64-efi
ID 260 gen 18604 top level 257 path <FS_TREE>/@/home
ID 261 gen 18604 top level 257 path <FS_TREE>/@/opt
ID 262 gen 18604 top level 257 path <FS_TREE>/@/srv
ID 263 gen 39799 top level 257 path <FS_TREE>/@/tmp
ID 264 gen 18604 top level 257 path <FS_TREE>/@/usr/local
ID 265 gen 18604 top level 257 path <FS_TREE>/@/var/crash
ID 266 gen 18604 top level 257 path <FS_TREE>/@/var/lib/mailman
ID 267 gen 18604 top level 257 path <FS_TREE>/@/var/lib/named
ID 268 gen 18604 top level 257 path <FS_TREE>/@/var/lib/pgsql
ID 269 gen 39793 top level 257 path <FS_TREE>/@/var/log
ID 270 gen 18604 top level 257 path <FS_TREE>/@/var/opt
ID 271 gen 39793 top level 257 path <FS_TREE>/@/var/spool
ID 272 gen 39793 top level 257 path <FS_TREE>/@/var/tmp

I'm not if this is a situation we will encounter, but if we want to ensure we always find the root filesystem, we should probably choose the root subvolume from the full list. (We can't just rely on the partition itself either.)

Mike Latimer,
regarding your comment#18:

What exactly do you mean with "root filesystem"?

Whatever is mounted at the moint point '/'
or the root of a filesystem?


I have an idea how common it is to change the default subvolume
to something other than the root of the filesystem:

In SLE12 the btrfs default subvolume is by default
something other than the root of the btrfs filesystem:
-------------------------------------------------------------------------
# btrfs subvolume get-default /
ID 257 gen 2972 top level 5 path @
-------------------------------------------------------------------------


FYI:

Note that wht is currently set as btrfs default subvolume
can be different to the subvolume that was used at the time
when this btrfs filesystem was mounted without '-o subvolid=...'
-------------------------------------------------------------------------
# btrfs subvolume list -a /
ID 257 gen 2979 top level 5 path <FS_TREE>/@
...
ID 264 gen 2800 top level 257 path @/usr/local
...

# btrfs subvolume get-default /
ID 257 gen 2990 top level 5 path @

# btrfs subvolume set-default 264 /

# btrfs subvolume get-default /
ID 264 gen 2992 top level 257 path usr/local
-------------------------------------------------------------------------
This would result that the next time when this btrfs filesystem will be
mounted without '-o subvolid=...' then the btrfs subvolume with ID 264
would be used.
I assume my above example may result that now my system can no longer
be booted therefore I change it back to what it was before:
-------------------------------------------------------------------------
# btrfs subvolume set-default 257 /

# btrfs subvolume get-default /
ID 257 gen 2990 top level 5 path @
-------------------------------------------------------------------------

Also note that one same btrfs filesystem can be mounted
multiple times (which is also the default on SLE12) because
on a SLE12 default system there is one single btrfs filesystem
that has multiple subvolumes and each of its subvolumes is
mounted at a different mount point by default - and even more
the admin can mount each of its subvolumes additionally at any
arbitrary mount point he likes:
---------------------------------------------------------------------------
# btrfs subvolume list -a /
ID 257 gen 2979 top level 5 path <FS_TREE>/@
ID 258 gen 2810 top level 257 path @/boot/grub2/i386-pc
ID 259 gen 2751 top level 257 path @/boot/grub2/x86_64-efi
ID 260 gen 2970 top level 257 path @/home
ID 261 gen 2969 top level 257 path @/opt
ID 262 gen 2813 top level 257 path @/srv
ID 263 gen 2985 top level 257 path @/tmp
ID 264 gen 2800 top level 257 path @/usr/local
ID 265 gen 2751 top level 257 path @/var/crash
ID 266 gen 2751 top level 257 path @/var/lib/mailman
ID 267 gen 2751 top level 257 path @/var/lib/named
ID 268 gen 2751 top level 257 path @/var/lib/pgsql
ID 269 gen 2981 top level 257 path @/var/log
ID 270 gen 2751 top level 257 path @/var/opt
ID 271 gen 2989 top level 257 path @/var/spool
ID 272 gen 2985 top level 257 path @/var/tmp
ID 275 gen 2979 top level 257 path @/.snapshots
ID 279 gen 131 top level 275 path <FS_TREE>/@/.snapshots/1/snapshot
ID 280 gen 132 top level 275 path <FS_TREE>/@/.snapshots/2/snapshot

# mkdir /tmp/usrlocal

# mount -t btrfs -o subvolid=264 /dev/sda2 /tmp/usrlocal

# echo hello >/tmp/usrlocal/hello

# ls /usr/local
bin  games  hello  include  lib  lib64  man  mysubsubvol  sbin  share  src

# ls /tmp/usrlocal
bin  games  hello  include  lib  lib64  man  mysubsubvol  sbin  share  src

# cat /usr/local/hello
hello

# mkdir /tmp/btrfsdefaultsubvol

# mount -t btrfs /dev/sda2 /tmp/btrfsdefaultsubvol

# echo hello2 >/tmp/btrfsdefaultsubvol/hello2

# cat /hello2
hello2

# findmnt -at btrfs | cut -b-63
TARGET                      SOURCE                             
/                           /dev/sda2[/@]                      
...
|-/usr/local                /dev/sda2[/@/usr/local]
...
|-/tmp                      /dev/sda2[/@/tmp]                  
| |-/tmp/usrlocal           /dev/sda2[/@/usr/local]            
| `-/tmp/btrfsdefaultsubvol /dev/sda2[/@]                      
---------------------------------------------------------------------------

Regarding comment#18:
"Is it safe to assume that subvolid=0 always equals
 the highest level subvolume"

I did not find really explicite and authoritative info.

I only found:

---------------------------------------------------------------------------
# man 8 btrfs-subvolume
...
Also every btrfs filesystem has a default subvolume as its initially
top-level subvolume, whose subvolume id is 5(FS_TREE).
---------------------------------------------------------------------------

https://btrfs.wiki.kernel.org/index.php/SysadminGuide
---------------------------------------------------------------------------
A btrfs filesystem has a default subvolume, which is initially set to be
the top-level subvolume. It is the default subvolume which is mounted
if no subvol or subvolid option is passed to mount.

Changing the default subvolume with btrfs subvolume default will make
the top level of the filesystem inaccessible, except by use of the
subvolid=0 mount option.
---------------------------------------------------------------------------

Acordingly one finds both "-o subvolid=0" and "-o subvolid=5"
to be used to mount the btrfs top-level/root subvolume.

Personally I prefer "-o subvolid=0" because the '0' looks more
like 'root' and I assume the ID 5 might change but right now
"-o subvolid=0" does the right thing regardless when the
top-level/root subvolume us actually '5' so that I assume
that if the ID 5 changes, "-o subvolid=0" would still do the
right thing to mount the top-level/root subvolume.

You may Google for "subvolumes: default and IDs".

Again:
I am really not a btrfs expert. Only by bad fortune I had to learn
something about btrfs the hard way - by trial and error :-(

(In reply to Johannes Meixner from comment #21)
> Regarding comment#18:
> "Is it safe to assume that subvolid=0 always equals
>  the highest level subvolume"
> 
> I did not find really explicite and authoritative info.

Thanks for all that info. I will see how to incorporate that into the btrfs handling in guestfs.

I think I have a similar kind of issue:

According to comment#18 you (i.e. the guestfs tools) are trying
to find out something about a btrfs filesystem in a virtual disk image.

I also need to find out the "real reality" about an existing btrfs
because I need to re-create it, see bug#908854 and
https://github.com/rear/rear/issues/497

I also think that I will have to mount the btrfs root subvolume
to be able to find out the "real reality" about a btrfs.

Please keep me informed what general approach you use
and what problems you may detect.

(In reply to Johannes Meixner from comment #23)
> Please keep me informed what general approach you use
> and what problems you may detect.

I think the approach should be to handle a btrfs partition not as a filesystem but as a container of filesystems. Right now the tool just mounts it, checks if its a root parition, then parses fstab to find other filesystems and subvolumes. Later it tries to map that partition and the referenced subvolumes, which fails if the "root partition" happens to have an unexpected subvolume entry.

Perhaps guestfs should create a map of available subvolumes, the default subvolume and check which of them is a root filesystem. Then see which other subvolumes are referenced by a given root filesystem.

It will take some time until I can start working on this bug.

The current workaround of using the "-o" option is just a bandaid.

SUSE-RU-2015:1313-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (low)
Bug References: 900346,900530,906692,908632,916567
CVE References: 
Sources used:
SUSE Linux Enterprise Software Development Kit 12 (src):    libguestfs-1.26.9-2.8.1
SUSE Linux Enterprise Server 12 (src):    libguestfs-1.26.9-2.8.1

possible fix:

https://www.redhat.com/archives/libguestfs/2016-February/msg00227.html

New libguestfs in SLES 12 SP2 comes with a new -m option for virt-ls. The problem is addressed with such a call:

virt-ls -a <VM disk image> -m /dev/sda2:/:subvol=@/.snapshots/3/snapshot /tmp

The list of available snapshots is available with:
virt-filesystems -a -VM disk image>

If omitting the subvol option to -m, the default subvolume will be used.

Apart from a clearer doc and more helpful error messages, nothing can be done to distinguish a SLES 12 BTRFS snapshot from the default one.

@Charles, any concern with it not being backported to SLES 12 and SLES 12 SP1?

(In reply to Cédric Bosdonnat from comment #30)
> New libguestfs in SLES 12 SP2 comes with a new -m option for virt-ls. The
> problem is addressed with such a call:
> 
> virt-ls -a <VM disk image> -m /dev/sda2:/:subvol=@/.snapshots/3/snapshot /tmp
> 
> The list of available snapshots is available with:
> virt-filesystems -a -VM disk image>
> 
> If omitting the subvol option to -m, the default subvolume will be used.
> 
> Apart from a clearer doc and more helpful error messages, nothing can be
> done to distinguish a SLES 12 BTRFS snapshot from the default one.
> 
> @Charles, any concern with it not being backported to SLES 12 and SLES 12
> SP1?

No problem. SLES 12 is now LTSS and SP1 hasn't had any customer complaints.

Fixed in SLES 12 SP2.