Bugzilla – Bug 865682
Cant start xen domU using drbd block device
Last modified: 2017-11-04 12:02:17 UTC
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.90 Safari/537.1 domU config file: ------------------ bootloader="pygrub" on_reboot = 'restart' on_crash = 'restart' name = "test" memory = "384" vcpus=1 cpus="1-15" audio=1 soundhw='es1370' vif = [ 'mac=00:16:3E:32:E3:EE, vifname=test.0, bridge=br400', 'mac=00:16:3E:34:42:DF, vifname=test.1, bridge=br600', 'mac=00:16:3E:5A:CD:3F, vifname=test.2, bridge=br300' ] disk = [ 'drbd:test.root,xvda1,w', 'drbd:test.swap,xvda2,w', ] ----------------- Reproducible: Always Steps to Reproduce: 1. Create DRBD devices 2. Create Xen guest config file specifying drbd device 3. xl -vvvvvvv create <domU> Actual Results: xl -vvvvvv create test Parsing config from test libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x22666a0: create: how=(nil) callback=(nil) poller=0x2262ef0 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvda1, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda1, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvda2, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda2, using backend phy libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=phy libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=(null), uses script=... assuming phy backend libxl: debug: libxl.c:2604:libxl__device_disk_local_initiate_attach: locally attaching PHY disk test.root libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x22666a0: inprogress: poller=0x2262ef0, flags=i libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x2263fa8 wpath=/local/domain/39 token=3/0: register slotnum=3 libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x22666a0: progress report: ignored libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.39.out libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0 libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.39.d libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: test.root libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x2263fa8 wpath=/local/domain/39 token=3/0: event epath=/local/domain/39 libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.39.log libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [11949] exited with error status 1 libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x2263fa8 wpath=/local/domain/39 token=3/0: deregister slotnum=3 libxl: error: libxl_create.c:900:domcreate_rebuild_done: cannot (re-)build domain: -3 libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x22666a0: complete, rc=-3 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x22666a0: destroy xc: debug: hypercall buffer: total allocations:27 total releases:27 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:23 misses:2 toobig:2 Expected Results: Expected that DomU guest starts up. This same setup works on SLES 11.2 OpenSUSE 13.1 with all patches applied as of 2014-02-25
Might be relevent http://lists.xen.org/archives/html/xen-devel/2014-02/msg01895.html
This bug was submitted after that patch was applied. Guest creation still fails with that patch applied.
I'll look into this here.
Can I get some additional details about your drbd setup? A full supportconfig (https://www.suse.com/communities/conversations/free_tools/supportconfig-linux/) would be helpful. Using the 13.1 version of drbd (8.3.11-8.1.3), drbd doesn't seem to want to start at all... There is at least one report of this (bnc#851536), that was resolved by moving to the Factory version of drbd. Before I do that, I want to see if you encountered this, or worked around it another way.
Yes, the drbd tools are too old to work. Which is a bug in and of itself. I installed this one and it works for me: http://download.opensuse.org/repositories/home:/espe:/stidev/openSUSE_13.1/x86_64/drbd-8.4.3-6.1.x86_64.rpm
Correction: This is the package I installed. Not the one in the previous post. http://download.opensuse.org/repositories/home:/smarty12/openSUSE_13.1/x86_64/drbd-8.4.3-5.8.x86_64.rpm
(In reply to comment #6) > Correction: > > This is the package I installed. Not the one in the previous post. > > http://download.opensuse.org/repositories/home:/smarty12/openSUSE_13.1/x86_64/drbd-8.4.3-5.8.x86_64.rpm Thank you. That package gets me into a working drbd state, but I am still encountering segfaults that might be the root of your problem as well. With the drbd resource active, can you get me the output of the following command: # /sbin/drbdadm sh-dev test.root On my test machine, that command segfaults, which in turn causes the entire block attach to fail. If the above command is successful, add 'set -x' to the beginning of /etc/xen/scripts/block-drbd and try to bring up the domain again. After it fails, grab the /var/log/xen/xen-hotplug.log file and attach it to this bug.
I can save you the trouble. With the definition disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ] The block-drbd script is never called If the disk definition is this: (completely bypassing drbd) the guest starts up. But only if I make the drbd device primary before starting the guest: disk = [ 'phy:/dev/drbd/by-res/test.root/0,sda1,w', 'phy:/dev/drbd/by-res/test.swap/0,sda2,w' ] This is the only way I can get block-drbd script to run, but the guest doesn't start up. So it looks like xl is having trouble parsing the stuff after the drbd: directive. Maybe it is expecting /some/sort/of/path instead of just the test.root drbd device. disk = [ 'drbd:/dev/drbd/by-res/test.root/0,sda1,w', 'drbd:/dev/drbd/by-res/test.swap/0,sda2,w', ] xl -vvvvv create test Parsing config from test libxl: debug: libxl_create.c:1230:do_domain_create: ao 0xd2c530: create: how=(nil) callback=(nil) poller=0xd2ca80 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda1 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda1, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda1, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda2 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda2, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda2, using backend phy libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=phy libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=(null), uses script=... assuming phy backend libxl: debug: libxl.c:2604:libxl__device_disk_local_initiate_attach: locally attaching PHY disk /dev/drbd/by-res/test.root/0 libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_create.c:1243:do_domain_create: ao 0xd2c530: inprogress: poller=0xd2ca80, flags=i libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0xd2cde8 wpath=/local/domain/4 token=3/0: register slotnum=3 libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0xd2c530: progress report: ignored libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.4.out libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0 libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.4.d libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /dev/drbd/by-res/test.root/0 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xd2cde8 wpath=/local/domain/4 token=3/0: event epath=/local/domain/4 libxl: debug: libxl_bootloader.c:634:bootloader_finished: bootloader completed libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained kernel /var/run/xen/bootloader.4.d/boot_kernel.PlyuY_ libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained ramdisk /var/run/xen/bootloader.4.d/boot_ramdisk.YImqKq libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained args ro root=/dev/sda1 console=xvc0 libxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader execution successful libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0xd2cde8 wpath=/local/domain/4 token=3/0: deregister slotnum=3 domainbuilder: detail: xc_dom_allocate: cmdline="ro root=/dev/sda1 console=xvc0", features="(null)" libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 1 path /var/run/xen/bootloader.4.d/boot_kernel.PlyuY_ domainbuilder: detail: xc_dom_kernel_mem: called domainbuilder: detail: xc_dom_malloc : 9705 kB domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x1f4308 -> 0x97a7c0 domainbuilder: detail: xc_dom_ramdisk_mem: called domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ... domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ... domainbuilder: detail: loader probe OK xc: detail: elf_parse_binary: phdr: paddr=0xffffffff80200000 memsz=0x2f4d70 xc: detail: elf_parse_binary: phdr: paddr=0xffffffff804f4d80 memsz=0x14c510 xc: detail: elf_parse_binary: phdr: paddr=0xffffffff80642000 memsz=0xc08 xc: detail: elf_parse_binary: phdr: paddr=0xffffffff80644000 memsz=0x11be8c xc: detail: elf_parse_binary: memory: 0xffffffff80200000 -> 0xffffffff8075fe8c xc: detail: elf_xen_parse_note: GUEST_OS = "linux" xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6" xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0" xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000 xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0xffffffff80000000 xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff80200000 xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80206000 xc: detail: elf_xen_parse_note: FEATURES = "writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel" xc: detail: elf_xen_parse_note: LOADER = "generic" xc: detail: elf_xen_addr_calc_check: addresses: xc: detail: virt_base = 0xffffffff80000000 xc: detail: elf_paddr_offset = 0xffffffff80000000 xc: detail: virt_offset = 0x0 xc: detail: virt_kstart = 0xffffffff80200000 xc: detail: virt_kend = 0xffffffff8075fe8c xc: detail: virt_entry = 0xffffffff80200000 xc: detail: p2m_base = 0xffffffffffffffff domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff80200000 -> 0xffffffff8075fe8c domainbuilder: detail: xc_dom_mem_init: mem 384 MB, pages 0x18000 pages, 4k each domainbuilder: detail: xc_dom_mem_init: 0x18000 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64 domainbuilder: detail: xc_dom_malloc : 768 kB domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_alloc_segment: kernel : 0xffffffff80200000 -> 0xffffffff80760000 (pfn 0x200 + 0x560 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x200+0x560 at 0x7f8b45d8c000 xc: detail: elf_load_binary: phdr 0 at 0x7f8b45d8c000 -> 0x7f8b46080d70 xc: detail: elf_load_binary: phdr 1 at 0x7f8b46080d80 -> 0x7f8b461cd290 xc: detail: elf_load_binary: phdr 2 at 0x7f8b461ce000 -> 0x7f8b461cec08 xc: detail: elf_load_binary: phdr 3 at 0x7f8b461d0000 -> 0x7f8b4620d0a8 domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0xffffffff80760000 -> 0xffffffff80d33000 (pfn 0x760 + 0x5d3 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x760+0x5d3 at 0x7f8b457b9000 domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x26182c -> 0x5d2a10 domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0xffffffff80d33000 -> 0xffffffff80df3000 (pfn 0xd33 + 0xc0 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xd33+0xc0 at 0x7f8b4924a000 domainbuilder: detail: xc_dom_alloc_page : start info : 0xffffffff80df3000 (pfn 0xdf3) domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xffffffff80df4000 (pfn 0xdf4) domainbuilder: detail: xc_dom_alloc_page : console : 0xffffffff80df5000 (pfn 0xdf5) domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffff80ffffff, 8 table(s) domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xffffffff80df6000 -> 0xffffffff80e01000 (pfn 0xdf6 + 0xb pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xdf6+0xb at 0x7f8b4923f000 domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xffffffff80e01000 (pfn 0xe01) domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xffffffff80e02000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0xffffffff81000000 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: arch_setup_bootearly: doing nothing domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64 domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x18000 domainbuilder: detail: clear_page: pfn 0xdf5, mfn 0xc7ed44 domainbuilder: detail: clear_page: pfn 0xdf4, mfn 0x43770c domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xdf3+0x1 at 0x7f8b493da000 domainbuilder: detail: start_info_x86_64: called domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff80206000 pfn=0x206 domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 10548 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 0 bytes domainbuilder: detail: domU mmap : 12284 kB domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0x7f1d7 domainbuilder: detail: shared_info_x86_64: called domainbuilder: detail: vcpu_x86_64: called domainbuilder: detail: vcpu_x86_64: cr3: pfn 0xdf6 mfn 0x43770b domainbuilder: detail: launch_vm: called, ctxt=0x7fff8dfcd4b0 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda1 spec.backend=phy libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda1, uses script=... assuming phy backend libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0xd2d928 wpath=/local/domain/0/backend/vbd/4/2049/state token=3/1: register slotnum=3 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda2 spec.backend=phy libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda2, uses script=... assuming phy backend libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0xd2f348 wpath=/local/domain/0/backend/vbd/4/2050/state token=2/2: register slotnum=2 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xd2d928 wpath=/local/domain/0/backend/vbd/4/2049/state token=3/1: event epath=/local/domain/0/backend/vbd/4/2049/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/2049/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0xd2d928 wpath=/local/domain/0/backend/vbd/4/2049/state token=3/1: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0xd2d928: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block-drbd add libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/2049/state token=3/1: empty slot libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xd2f348 wpath=/local/domain/0/backend/vbd/4/2050/state token=2/2: event epath=/local/domain/0/backend/vbd/4/2050/state libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/2050/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xd2f348 wpath=/local/domain/0/backend/vbd/4/2050/state token=2/2: event epath=/local/domain/0/backend/vbd/4/2050/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/2050/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0xd2f348 wpath=/local/domain/0/backend/vbd/4/2050/state token=2/2: deregister slotnum=2 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0xd2f348: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block-drbd add ++ dirname /etc/xen/scripts/block-drbd + dir=/etc/xen/scripts + . /etc/xen/scripts/block-common.sh ++ dirname /etc/xen/scripts/block-drbd +++ dirname /etc/xen/scripts/block-drbd + dir=/etc/xen/scripts + . /etc/xen/scripts/block-common.sh +++ dirname /etc/xen/scripts/block-drbd ++ dir=/etc/xen/scripts ++ . /etc/xen/scripts/xen-hotplug-common.sh +++ '[' -n '' ']' ++++ dirname /etc/xen/scripts/block-drbd ++ dir=/etc/xen/scripts ++ . /etc/xen/scripts/xen-hotplug-common.sh +++ '[' -n '' ']' +++ dir=/etc/xen/scripts +++ . /etc/xen/scripts/hotplugpath.sh ++++ dirname /etc/xen/scripts/block-drbd ++++ SBINDIR=/usr/sbin ++++ BINDIR=/usr/bin ++++ LIBEXEC=/usr/lib/xen/bin ++++ LIBDIR=/usr/lib64 ++++ SHAREDIR=/usr/share ++++ PRIVATE_BINDIR=/usr/lib64/xen/bin ++++ XENFIRMWAREDIR=/usr/lib/xen/boot ++++ XEN_CONFIG_DIR=/etc/xen ++++ XEN_SCRIPT_DIR=/etc/xen/scripts ++++ XEN_LOCK_DIR=/var/lock ++++ XEN_RUN_DIR=/var/run/xen ++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging +++ . /etc/xen/scripts/logging.sh +++ dir=/etc/xen/scripts +++ . /etc/xen/scripts/hotplugpath.sh ++++ SBINDIR=/usr/sbin ++++ BINDIR=/usr/bin ++++ LIBEXEC=/usr/lib/xen/bin ++++ LIBDIR=/usr/lib64 ++++ SHAREDIR=/usr/share ++++ PRIVATE_BINDIR=/usr/lib64/xen/bin ++++ XENFIRMWAREDIR=/usr/lib/xen/boot ++++ XEN_CONFIG_DIR=/etc/xen ++++ XEN_SCRIPT_DIR=/etc/xen/scripts ++++ XEN_LOCK_DIR=/var/lock ++++ XEN_RUN_DIR=/var/run/xen ++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging +++ . /etc/xen/scripts/logging.sh +++ . /etc/xen/scripts/xen-script-common.sh +++ . /etc/xen/scripts/xen-script-common.sh ++++ set -e +++ . /etc/xen/scripts/locking.sh ++++ set -e +++ . /etc/xen/scripts/locking.sh ++++ LOCK_BASEDIR=/var/run/xen-hotplug ++++ LOCK_BASEDIR=/var/run/xen-hotplug +++ exec +++ exec libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block-drbd add [12393] exited with error status 1 libxl: error: libxl_device.c:1021:device_hotplug_child_death_cb: script: /etc/xen/scripts/block-drbd failed; error detected. libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block-drbd add [12392] exited with error status 1 libxl: error: libxl_device.c:1021:device_hotplug_child_death_cb: script: /etc/xen/scripts/block-drbd failed; error detected. libxl: error: libxl_create.c:932:domcreate_launch_dm: unable to add disk devices libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0xd30db8 wpath=/local/domain/0/backend/vbd/4/2049/state token=2/3: register slotnum=2 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0xd31008 wpath=/local/domain/0/backend/vbd/4/2050/state token=3/4: register slotnum=3 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xd30db8 wpath=/local/domain/0/backend/vbd/4/2049/state token=2/3: event epath=/local/domain/0/backend/vbd/4/2049/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/2049/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0xd30db8 wpath=/local/domain/0/backend/vbd/4/2049/state token=2/3: deregister slotnum=2 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0xd30db8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block-drbd remove libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/2049/state token=2/3: empty slot libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xd31008 wpath=/local/domain/0/backend/vbd/4/2050/state token=3/4: event epath=/local/domain/0/backend/vbd/4/2050/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/2050/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0xd31008 wpath=/local/domain/0/backend/vbd/4/2050/state token=3/4: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0xd31008: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block-drbd remove libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/2050/state token=3/4: empty slot ++ dirname /etc/xen/scripts/block-drbd ++ dirname /etc/xen/scripts/block-drbd + dir=/etc/xen/scripts + . /etc/xen/scripts/block-common.sh +++ dirname /etc/xen/scripts/block-drbd + dir=/etc/xen/scripts + . /etc/xen/scripts/block-common.sh +++ dirname /etc/xen/scripts/block-drbd ++ dir=/etc/xen/scripts ++ . /etc/xen/scripts/xen-hotplug-common.sh +++ '[' -n '' ']' ++++ dirname /etc/xen/scripts/block-drbd ++ dir=/etc/xen/scripts ++ . /etc/xen/scripts/xen-hotplug-common.sh +++ '[' -n '' ']' ++++ dirname /etc/xen/scripts/block-drbd +++ dir=/etc/xen/scripts +++ . /etc/xen/scripts/hotplugpath.sh ++++ SBINDIR=/usr/sbin ++++ BINDIR=/usr/bin ++++ LIBEXEC=/usr/lib/xen/bin ++++ LIBDIR=/usr/lib64 ++++ SHAREDIR=/usr/share ++++ PRIVATE_BINDIR=/usr/lib64/xen/bin ++++ XENFIRMWAREDIR=/usr/lib/xen/boot ++++ XEN_CONFIG_DIR=/etc/xen ++++ XEN_SCRIPT_DIR=/etc/xen/scripts ++++ XEN_LOCK_DIR=/var/lock ++++ XEN_RUN_DIR=/var/run/xen ++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging +++ . /etc/xen/scripts/logging.sh +++ . /etc/xen/scripts/xen-script-common.sh ++++ set -e +++ . /etc/xen/scripts/locking.sh ++++ LOCK_BASEDIR=/var/run/xen-hotplug +++ dir=/etc/xen/scripts +++ . /etc/xen/scripts/hotplugpath.sh ++++ SBINDIR=/usr/sbin ++++ BINDIR=/usr/bin ++++ LIBEXEC=/usr/lib/xen/bin ++++ LIBDIR=/usr/lib64 ++++ SHAREDIR=/usr/share ++++ PRIVATE_BINDIR=/usr/lib64/xen/bin ++++ XENFIRMWAREDIR=/usr/lib/xen/boot ++++ XEN_CONFIG_DIR=/etc/xen ++++ XEN_SCRIPT_DIR=/etc/xen/scripts +++ exec ++++ XEN_LOCK_DIR=/var/lock ++++ XEN_RUN_DIR=/var/run/xen ++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging +++ . /etc/xen/scripts/logging.sh +++ . /etc/xen/scripts/xen-script-common.sh ++++ set -e +++ . /etc/xen/scripts/locking.sh ++++ LOCK_BASEDIR=/var/run/xen-hotplug +++ exec libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block-drbd remove [12446] exited with error status 1 libxl: error: libxl_device.c:1021:device_hotplug_child_death_cb: script: /etc/xen/scripts/block-drbd failed; error detected. libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block-drbd remove [12448] exited with error status 1 libxl: error: libxl_device.c:1021:device_hotplug_child_death_cb: script: /etc/xen/scripts/block-drbd failed; error detected. libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0xd2c530: complete, rc=-3 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0xd2c530: destroy xc: debug: hypercall buffer: total allocations:239 total releases:239 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:233 misses:2 toobig:4
(In reply to comment #8) > I can save you the trouble. > > With the definition > disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ] > The block-drbd script is never called That's not what is happening in my environment. The block-drdb script is executing just fine. However, you do need the fix Jim mentioned in comment #1. With that in place, the block-attach is still failing - but due to drbdadm segfaulting in two places. If I replace those commands with the hard-coded correct response (in my system), the disk attaches properly and the VM is started. (At least in my environment.) > If the disk definition is this: (completely bypassing drbd) the guest starts > up. But only if I make the drbd device primary before starting the guest: > disk = [ 'phy:/dev/drbd/by-res/test.root/0,sda1,w', > 'phy:/dev/drbd/by-res/test.swap/0,sda2,w' ] This makes sense, as Xen would not even know about drbd with this setup. > This is the only way I can get block-drbd script to run, but the guest doesn't > start up. > So it looks like xl is having trouble parsing the stuff after the drbd: > directive. Maybe it is expecting /some/sort/of/path instead of just the > test.root drbd device. One difference between your environment and mine is that I am not using drbd for root. This changes the startup a bit, but still leaves me in a situation where xvdb1 cannot be attached. It might be interesting to change your disk configuration to something like: disk = [ 'phy:/dev/drbd/by-res/test.root/0,sda1,w', 'drbd:test.swap,xvda2,w', ] Just to see if the system logs end up closer to mine... Either way, the key to tracking this down is a debug xen-hotpug log. Please add 'set -x' at the beginning of /etc/xen/scripts/block-drbd, try the test again (with 'drbd:test.root' syntax), and attach /var/log/xen/xen-hotplug.log after it fails. If there is nothing in this log when using the correct syntax, try it again using your /dev/drbd/... syntax and send me the debug log from that run.
Okay so with this disk definition: disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ] And the patch applied: vhost1:/etc/xen/scripts # diff block-drbd block-drbd.orig 1c1 < #!/bin/bash -x --- > #!/bin/bash 251c251 < drbd|phy) --- > drbd) 279c279 < drbd|phy) --- > drbd) =========== xl -vvvvvv create test Parsing config from test libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x1bef510: create: how=(nil) callback=(nil) poller=0x1befa80 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda1 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda1, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda1, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda2 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda2, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda2, using backend phy libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=phy libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=(null), uses script=... assuming phy backend libxl: debug: libxl.c:2604:libxl__device_disk_local_initiate_attach: locally attaching PHY disk test.root libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x1bef510: inprogress: poller=0x1befa80, flags=i libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x1befde8 wpath=/local/domain/7 token=3/0: register slotnum=3 libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x1bef510: progress report: ignored libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.7.out libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0 libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.7.d libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: test.root libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1befde8 wpath=/local/domain/7 token=3/0: event epath=/local/domain/7 libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.7.log libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [12772] exited with error status 1 libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x1befde8 wpath=/local/domain/7 token=3/0: deregister slotnum=3 libxl: error: libxl_create.c:900:domcreate_rebuild_done: cannot (re-)build domain: -3 libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x1bef510: complete, rc=-3 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x1bef510: destroy xc: debug: hypercall buffer: total allocations:27 total releases:27 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:23 misses:2 toobig:2 ============ cat /var/log/xen/xen-hotplug.log +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ set ++++ grep '^LC_' ++++ set ++++ cut -d= -f1 ++++ grep '^LC_' ++++ cut -d= -f1 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug add XENBUS_PATH=backend/vbd/4/2049 +++ local level=debug +++ unset LC_CTYPE +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: add XENBUS_PATH=backend/vbd/4/2049 +++ trap sigerr ERR +++ log debug add XENBUS_PATH=backend/vbd/4/2050 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: add XENBUS_PATH=backend/vbd/4/2050 ++ findCommand add ++ for arg in '"$@"' ++ expr index add = ++ findCommand add ++ for arg in '"$@"' ++ expr index add = ++ command=add ++ return ++ command=add ++ return ++ '[' add '!=' add ']' ++ XENBUS_PATH=backend/vbd/4/2049 ++ '[' add '!=' add ']' ++ XENBUS_PATH=backend/vbd/4/2050 ++ xenstore_read_default backend/vbd/4/2050/type MISSING ++ xenstore_read_default backend/vbd/4/2049/type MISSING ++ xenstore-read backend/vbd/4/2050/type ++ xenstore-read backend/vbd/4/2049/type + t=phy + t=phy + case "$command" in + case "$command" in ++ xenstore_read_default backend/vbd/4/2049/physical-device MISSING ++ xenstore-read backend/vbd/4/2049/physical-device ++ xenstore_read_default backend/vbd/4/2050/physical-device MISSING ++ xenstore-read backend/vbd/4/2050/physical-device ++ echo MISSING ++ echo MISSING + phys=MISSING + phys=MISSING + '[' MISSING '!=' MISSING ']' + '[' MISSING '!=' MISSING ']' + '[' -n phy ']' + '[' -n phy ']' ++ xenstore_read backend/vbd/4/2049/params ++ xenstore_read backend/vbd/4/2050/params +++ xenstore-read backend/vbd/4/2049/params +++ xenstore-read backend/vbd/4/2050/params ++ local v=/dev/drbd/by-res/test.root/0 ++ local v=/dev/drbd/by-res/test.swap/0 ++ '[' /dev/drbd/by-res/test.root/0 '!=' '' ']' ++ '[' /dev/drbd/by-res/test.swap/0 '!=' '' ']' ++ echo /dev/drbd/by-res/test.root/0 ++ echo /dev/drbd/by-res/test.swap/0 + p=/dev/drbd/by-res/test.swap/0 + p=/dev/drbd/by-res/test.root/0 ++ xenstore_read backend/vbd/4/2050/mode ++ xenstore_read backend/vbd/4/2049/mode +++ xenstore-read backend/vbd/4/2050/mode +++ xenstore-read backend/vbd/4/2049/mode ++ local v=w ++ '[' w '!=' '' ']' ++ echo w ++ local v=w ++ '[' w '!=' '' ']' ++ echo w + mode=w + case $t in + drbd_resource=/dev/drbd/by-res/test.swap/0 + mode=w + case $t in + drbd_resource=/dev/drbd/by-res/test.root/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.swap/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.root/0 dev/drbd/by-res/test.swap/0 is not a valid number dev/drbd/by-res/test.root/0 is not a valid number + drbd_role= ++ sigerr ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ _xenstore_write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error + drbd_role= ++ log debug 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ local level=debug ++ sigerr ++ shift ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ _xenstore_write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ log debug 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' ++ local level=debug ++ shift ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' ++ xenstore-write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error ++ xenstore-write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ exit 1 ++ exit 1 +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ set +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games ++++ grep '^LC_' +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ cut -d= -f1 ++++ set ++++ grep '^LC_' ++++ cut -d= -f1 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug remove XENBUS_PATH=backend/vbd/4/2049 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: remove XENBUS_PATH=backend/vbd/4/2049 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug remove XENBUS_PATH=backend/vbd/4/2050 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: remove XENBUS_PATH=backend/vbd/4/2050 ++ findCommand remove ++ for arg in '"$@"' ++ expr index remove = ++ command=remove ++ return ++ findCommand remove ++ '[' remove '!=' add ']' ++ for arg in '"$@"' ++ '[' remove '!=' remove ']' ++ expr index remove = ++ XENBUS_PATH=backend/vbd/4/2049 ++ xenstore_read_default backend/vbd/4/2049/type MISSING ++ command=remove ++ return ++ xenstore-read backend/vbd/4/2049/type ++ '[' remove '!=' add ']' ++ '[' remove '!=' remove ']' ++ XENBUS_PATH=backend/vbd/4/2050 ++ xenstore_read_default backend/vbd/4/2050/type MISSING ++ xenstore-read backend/vbd/4/2050/type + t=phy + case "$command" in + case $t in ++ xenstore_read backend/vbd/4/2049/params +++ xenstore-read backend/vbd/4/2049/params + t=phy + case "$command" in + case $t in ++ xenstore_read backend/vbd/4/2050/params +++ xenstore-read backend/vbd/4/2050/params ++ local v=/dev/drbd/by-res/test.root/0 ++ '[' /dev/drbd/by-res/test.root/0 '!=' '' ']' ++ echo /dev/drbd/by-res/test.root/0 + p=/dev/drbd/by-res/test.root/0 + drbd_resource=/dev/drbd/by-res/test.root/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.root/0 ++ local v=/dev/drbd/by-res/test.swap/0 ++ '[' /dev/drbd/by-res/test.swap/0 '!=' '' ']' ++ echo /dev/drbd/by-res/test.swap/0 + p=/dev/drbd/by-res/test.swap/0 + drbd_resource=/dev/drbd/by-res/test.swap/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.swap/0 dev/drbd/by-res/test.root/0 is not a valid number + drbd_role= ++ sigerr ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ _xenstore_write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ log debug 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' ++ local level=debug ++ shift ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' dev/drbd/by-res/test.swap/0 is not a valid number + drbd_role= ++ sigerr ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ _xenstore_write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error ++ log debug 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ local level=debug ++ shift ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ xenstore-write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ xenstore-write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ exit 1 ++ exit 1
(In reply to comment #10) > Okay so with this disk definition: > disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ] > > And the patch applied: Ok, good starting point. ;) > + case $t in > + drbd_resource=/dev/drbd/by-res/test.root/0 > ++ /sbin/drbdadm role /dev/drbd/by-res/test.swap/0 > ++ /sbin/drbdadm role /dev/drbd/by-res/test.root/0 > dev/drbd/by-res/test.swap/0 is not a valid number > dev/drbd/by-res/test.root/0 is not a valid number > + drbd_role= > ++ sigerr This 'sigerr' is in the place I am talking about (caused by drbdadm). However, it looks like this log is from an earlier configuration - while using the /dev/drbd/by-res... syntax. The drbd_resource should match what is in the drbd config above (e.g. test.root or test.swap). It could be that this block-drbd script was never executed after changing the syntax, or there could be something else strange going on. Can you clear the file by echoing "" into it, then try the test again (using the test.root syntax)?
Yeah block-drbd isn't executing with: disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ] hotplug log is empty, block-drbd doesn't get called.
(In reply to comment #12) > Yeah block-drbd isn't executing with: > disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ] > > hotplug log is empty, block-drbd doesn't get called. Interesting... I can see the same behavior (no block-drbd execution) if I use the drdb device as my root device. Going back to comment #10, I can see the same error reported in your logs: > libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - > consult logfile /var/log/xen/bootloader.7.log Can you post the contents of bootloader.7.log here? Also, you are saying this same configuration (of having a root device on a drbd device, with the 'drbd:test.root' syntax) was working previous to openSUSE 13.1?
After playing around with this a bit more, we should see what happens when we attach the device to dom0. Delete /var/log/xen/xen-hotplug.log, then execute the following command: xl block-attach 0 'drbd:test.root,sda1,w' This will put debug information into the xen-hotplug.log file that should help us track this down.
(In reply to comment #14) > After playing around with this a bit more, we should see what happens when we > attach the device to dom0. Delete /var/log/xen/xen-hotplug.log, then execute > the following command: > > xl block-attach 0 'drbd:test.root,sda1,w' Opps - change the sda1 to xvda1: xl block-attach 0 'drbd:test.root,xvda1,w' as this will create the specified device in dom0.
cat /var/log/xen/bootloader.7.log Traceback (most recent call last): File "/usr/lib64/xen/bin/pygrub", line 863, in <module> part_offs = get_partition_offsets(file) File "/usr/lib64/xen/bin/pygrub", line 106, in get_partition_offsets image_type = identify_disk_image(file) File "/usr/lib64/xen/bin/pygrub", line 48, in identify_disk_image fd = os.open(file, os.O_RDONLY) OSError: [Errno 2] No such file or directory: 'test.root'
xl block-attach 0 'drbd:test.root,sda,w' ++ dirname /etc/xen/scripts/block-drbd + dir=/etc/xen/scripts + . /etc/xen/scripts/block-common.sh +++ dirname /etc/xen/scripts/block-drbd ++ dir=/etc/xen/scripts ++ . /etc/xen/scripts/xen-hotplug-common.sh +++ '[' -n '' ']' ++++ dirname /etc/xen/scripts/block-drbd +++ dir=/etc/xen/scripts +++ . /etc/xen/scripts/hotplugpath.sh ++++ SBINDIR=/usr/sbin ++++ BINDIR=/usr/bin ++++ LIBEXEC=/usr/lib/xen/bin ++++ LIBDIR=/usr/lib64 ++++ SHAREDIR=/usr/share ++++ PRIVATE_BINDIR=/usr/lib64/xen/bin ++++ XENFIRMWAREDIR=/usr/lib/xen/boot ++++ XEN_CONFIG_DIR=/etc/xen ++++ XEN_SCRIPT_DIR=/etc/xen/scripts ++++ XEN_LOCK_DIR=/var/lock ++++ XEN_RUN_DIR=/var/run/xen ++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging +++ . /etc/xen/scripts/logging.sh +++ . /etc/xen/scripts/xen-script-common.sh ++++ set -e +++ . /etc/xen/scripts/locking.sh ++++ LOCK_BASEDIR=/var/run/xen-hotplug +++ exec # xl block-list 0 Vdev BE handle state evt-ch ring-ref BE-path 2048 0 0 5 109 8 /local/domain/0/backend/vbd/0/2048 cat /var/log/xen/xen-hotplug.log +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ set ++++ grep '^LC_' ++++ cut -d= -f1 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug add XENBUS_PATH=backend/vbd/0/2048 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: add XENBUS_PATH=backend/vbd/0/2048 ++ findCommand add ++ for arg in '"$@"' ++ expr index add = ++ command=add ++ return ++ '[' add '!=' add ']' ++ XENBUS_PATH=backend/vbd/0/2048 ++ xenstore_read_default backend/vbd/0/2048/type MISSING ++ xenstore-read backend/vbd/0/2048/type + t=phy + case "$command" in ++ xenstore_read_default backend/vbd/0/2048/physical-device MISSING ++ xenstore-read backend/vbd/0/2048/physical-device ++ echo MISSING + phys=MISSING + '[' MISSING '!=' MISSING ']' + '[' -n phy ']' ++ xenstore_read backend/vbd/0/2048/params +++ xenstore-read backend/vbd/0/2048/params ++ local v=test.root ++ '[' test.root '!=' '' ']' ++ echo test.root + p=test.root ++ xenstore_read backend/vbd/0/2048/mode +++ xenstore-read backend/vbd/0/2048/mode ++ local v=w ++ '[' w '!=' '' ']' ++ echo w + mode=w + case $t in + drbd_resource=test.root ++ /sbin/drbdadm role test.root + drbd_role=Primary/Secondary + drbd_lrole=Primary ++ /sbin/drbdadm sh-dev test.root + drbd_dev=/dev/drbd37 + '[' Primary '!=' Primary ']' + dev=/dev/drbd37 ++ xenstore_read backend/vbd/0/2048/frontend-id +++ xenstore-read backend/vbd/0/2048/frontend-id ++ local v=0 ++ '[' 0 '!=' '' ']' ++ echo 0 + FRONTEND_ID=0 ++ xenstore_read_default /local/domain/0/vm unknown ++ xenstore-read /local/domain/0/vm ++ echo unknown + FRONTEND_UUID=unknown + claim_lock block + mkdir -p /var/run/xen-hotplug + _setlockfd block + local i + (( i = 0 )) + (( i < 0 )) + _lockdict[$i]=block + let _lockfd=200+i + _lockfile=/var/run/xen-hotplug/block + local rightfile + true + eval 'exec 200<>/var/run/xen-hotplug/block' ++ exec + flock -x 200 ++ perl -e ' open STDIN, "<&200" or die $!; my $fd_inum = (stat STDIN)[1]; die $! unless defined $fd_inum; my $file_inum = (stat $ARGV[0])[1]; print "y\n" if $fd_inum eq $file_inum; ' /var/run/xen-hotplug/block + rightfile=y + '[' xy = xy ']' + break + check_device_sharing /dev/drbd37 w + local dev=/dev/drbd37 ++ canonicalise_mode w ++ local mode=w ++ expr index w w ++ expr index w '!' ++ echo w + local mode=w + local result + '[' xw = 'x!' ']' ++ check_sharing /dev/drbd37 w ++ local dev=/dev/drbd37 ++ local mode=w +++ device_major_minor /dev/drbd37 +++ stat -L -c %t:%T /dev/drbd37 ++ local devmm=93:25 ++ local file ++ toskip='^$' +++ cat /proc/mounts +++ grep -v '^$' +++ cut -f 1 -d ' ' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e rootfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e devtmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e devpts ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e /dev/mapper/system-root ']' +++ device_major_minor /dev/mapper/system-root +++ stat -L -c %t:%T /dev/mapper/system-root ++ local d=fd:0 ++ '[' fd:0 = 93:25 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e proc ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e sysfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e securityfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e pstore ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e systemd-1 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e mqueue ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e debugfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e /dev/sda1 ']' +++ device_major_minor /dev/sda1 +++ stat -L -c %t:%T /dev/sda1 ++ local d=8:1 ++ '[' 8:1 = 93:25 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e /dev/mapper/vgdomu-backup ']' +++ device_major_minor /dev/mapper/vgdomu-backup +++ stat -L -c %t:%T /dev/mapper/vgdomu-backup ++ local d=fd:2 ++ '[' fd:2 = 93:25 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e none ']' ++ local base_path=backend/vbd +++ xenstore-list backend/vbd ++ for dom in '$(xenstore-list "$base_path")' +++ xenstore-list backend/vbd/0 ++ for dev in '$(xenstore-list "$base_path/$dom")' +++ xenstore_read_default backend/vbd/0/2048/physical-device '' +++ xenstore-read backend/vbd/0/2048/physical-device +++ echo '' ++ d= ++ '[' '' = 93:25 ']' ++ echo ok + result=ok + '[' ok '!=' ok ']' + write_dev /dev/drbd37 + local mm ++ device_major_minor /dev/drbd37 ++ stat -L -c %t:%T /dev/drbd37 + mm=93:25 + '[' -z 93:25 ']' + xenstore_write backend/vbd/0/2048/physical-device 93:25 + _xenstore_write backend/vbd/0/2048/physical-device 93:25 + log debug 'Writing backend/vbd/0/2048/physical-device' '93:25 to xenstore.' + local level=debug + shift + logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/0/2048/physical-device' '93:25 to xenstore.' + xenstore-write backend/vbd/0/2048/physical-device 93:25 + success + xenstore_write backend/vbd/0/2048/hotplug-status connected + _xenstore_write backend/vbd/0/2048/hotplug-status connected + log debug 'Writing backend/vbd/0/2048/hotplug-status' 'connected to xenstore.' + local level=debug + shift + logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/0/2048/hotplug-status' 'connected to xenstore.' + xenstore-write backend/vbd/0/2048/hotplug-status connected + release_lock block + _setlockfd block + local i + (( i = 0 )) + (( i < 5 )) + '[' -z block -o block = block ']' + break + _lockdict[$i]=block + let _lockfd=200+i + _lockfile=/var/run/xen-hotplug/block + rm /var/run/xen-hotplug/block + exit 0
xl block-attach 0 'drbd:test.root,xvda1,w' ++ dirname /etc/xen/scripts/block-drbd + dir=/etc/xen/scripts + . /etc/xen/scripts/block-common.sh +++ dirname /etc/xen/scripts/block-drbd ++ dir=/etc/xen/scripts ++ . /etc/xen/scripts/xen-hotplug-common.sh +++ '[' -n '' ']' ++++ dirname /etc/xen/scripts/block-drbd +++ dir=/etc/xen/scripts +++ . /etc/xen/scripts/hotplugpath.sh ++++ SBINDIR=/usr/sbin ++++ BINDIR=/usr/bin ++++ LIBEXEC=/usr/lib/xen/bin ++++ LIBDIR=/usr/lib64 ++++ SHAREDIR=/usr/share ++++ PRIVATE_BINDIR=/usr/lib64/xen/bin ++++ XENFIRMWAREDIR=/usr/lib/xen/boot ++++ XEN_CONFIG_DIR=/etc/xen ++++ XEN_SCRIPT_DIR=/etc/xen/scripts ++++ XEN_LOCK_DIR=/var/lock ++++ XEN_RUN_DIR=/var/run/xen ++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging +++ . /etc/xen/scripts/logging.sh +++ . /etc/xen/scripts/xen-script-common.sh ++++ set -e +++ . /etc/xen/scripts/locking.sh ++++ LOCK_BASEDIR=/var/run/xen-hotplug +++ exec xl block-list 0 Vdev BE handle state evt-ch ring-ref BE-path 51713 0 0 4 109 8 /local/domain/0/backend/vbd/0/51713 cat /var/log/xen/xen-hotplug.log +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ set ++++ grep '^LC_' ++++ cut -d= -f1 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug add XENBUS_PATH=backend/vbd/0/51713 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: add XENBUS_PATH=backend/vbd/0/51713 ++ findCommand add ++ for arg in '"$@"' ++ expr index add = ++ command=add ++ return ++ '[' add '!=' add ']' ++ XENBUS_PATH=backend/vbd/0/51713 ++ xenstore_read_default backend/vbd/0/51713/type MISSING ++ xenstore-read backend/vbd/0/51713/type + t=phy + case "$command" in ++ xenstore_read_default backend/vbd/0/51713/physical-device MISSING ++ xenstore-read backend/vbd/0/51713/physical-device ++ echo MISSING + phys=MISSING + '[' MISSING '!=' MISSING ']' + '[' -n phy ']' ++ xenstore_read backend/vbd/0/51713/params +++ xenstore-read backend/vbd/0/51713/params ++ local v=test.root ++ '[' test.root '!=' '' ']' ++ echo test.root + p=test.root ++ xenstore_read backend/vbd/0/51713/mode +++ xenstore-read backend/vbd/0/51713/mode ++ local v=w ++ '[' w '!=' '' ']' ++ echo w + mode=w + case $t in + drbd_resource=test.root ++ /sbin/drbdadm role test.root + drbd_role=Secondary/Secondary + drbd_lrole=Secondary ++ /sbin/drbdadm sh-dev test.root + drbd_dev=/dev/drbd37 + '[' Secondary '!=' Primary ']' + /sbin/drbdadm primary test.root + dev=/dev/drbd37 ++ xenstore_read backend/vbd/0/51713/frontend-id +++ xenstore-read backend/vbd/0/51713/frontend-id ++ local v=0 ++ '[' 0 '!=' '' ']' ++ echo 0 + FRONTEND_ID=0 ++ xenstore_read_default /local/domain/0/vm unknown ++ xenstore-read /local/domain/0/vm ++ echo unknown + FRONTEND_UUID=unknown + claim_lock block + mkdir -p /var/run/xen-hotplug + _setlockfd block + local i + (( i = 0 )) + (( i < 0 )) + _lockdict[$i]=block + let _lockfd=200+i + _lockfile=/var/run/xen-hotplug/block + local rightfile + true + eval 'exec 200<>/var/run/xen-hotplug/block' ++ exec + flock -x 200 ++ perl -e ' open STDIN, "<&200" or die $!; my $fd_inum = (stat STDIN)[1]; die $! unless defined $fd_inum; my $file_inum = (stat $ARGV[0])[1]; print "y\n" if $fd_inum eq $file_inum; ' /var/run/xen-hotplug/block + rightfile=y + '[' xy = xy ']' + break + check_device_sharing /dev/drbd37 w + local dev=/dev/drbd37 ++ canonicalise_mode w ++ local mode=w ++ expr index w w ++ expr index w '!' ++ echo w + local mode=w + local result + '[' xw = 'x!' ']' ++ check_sharing /dev/drbd37 w ++ local dev=/dev/drbd37 ++ local mode=w +++ device_major_minor /dev/drbd37 +++ stat -L -c %t:%T /dev/drbd37 ++ local devmm=93:25 ++ local file ++ toskip='^$' +++ cat /proc/mounts +++ grep -v '^$' +++ cut -f 1 -d ' ' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e rootfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e devtmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e devpts ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e /dev/mapper/system-root ']' +++ device_major_minor /dev/mapper/system-root +++ stat -L -c %t:%T /dev/mapper/system-root ++ local d=fd:0 ++ '[' fd:0 = 93:25 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e proc ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e sysfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e securityfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e pstore ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e cgroup ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e systemd-1 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e mqueue ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e debugfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e tmpfs ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e /dev/sda1 ']' +++ device_major_minor /dev/sda1 +++ stat -L -c %t:%T /dev/sda1 ++ local d=8:1 ++ '[' 8:1 = 93:25 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e /dev/mapper/vgdomu-backup ']' +++ device_major_minor /dev/mapper/vgdomu-backup +++ stat -L -c %t:%T /dev/mapper/vgdomu-backup ++ local d=fd:2 ++ '[' fd:2 = 93:25 ']' ++ for file in '$(cat /proc/mounts | grep -v "$toskip" | cut -f 1 -d '\'' '\'')' ++ '[' -e none ']' ++ local base_path=backend/vbd +++ xenstore-list backend/vbd ++ for dom in '$(xenstore-list "$base_path")' +++ xenstore-list backend/vbd/0 ++ for dev in '$(xenstore-list "$base_path/$dom")' +++ xenstore_read_default backend/vbd/0/51713/physical-device '' +++ xenstore-read backend/vbd/0/51713/physical-device +++ echo '' ++ d= ++ '[' '' = 93:25 ']' ++ echo ok + result=ok + '[' ok '!=' ok ']' + write_dev /dev/drbd37 + local mm ++ device_major_minor /dev/drbd37 ++ stat -L -c %t:%T /dev/drbd37 + mm=93:25 + '[' -z 93:25 ']' + xenstore_write backend/vbd/0/51713/physical-device 93:25 + _xenstore_write backend/vbd/0/51713/physical-device 93:25 + log debug 'Writing backend/vbd/0/51713/physical-device' '93:25 to xenstore.' + local level=debug + shift + logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/0/51713/physical-device' '93:25 to xenstore.' + xenstore-write backend/vbd/0/51713/physical-device 93:25 + success + xenstore_write backend/vbd/0/51713/hotplug-status connected + _xenstore_write backend/vbd/0/51713/hotplug-status connected + log debug 'Writing backend/vbd/0/51713/hotplug-status' 'connected to xenstore.' + local level=debug + shift + logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/0/51713/hotplug-status' 'connected to xenstore.' + xenstore-write backend/vbd/0/51713/hotplug-status connected + release_lock block + _setlockfd block + local i + (( i = 0 )) + (( i < 5 )) + '[' -z block -o block = block ']' + break + _lockdict[$i]=block + let _lockfd=200+i + _lockfile=/var/run/xen-hotplug/block + rm /var/run/xen-hotplug/block + exit 0
Thanks, Bill. Those logs seem to indicate the block-attach completed successfully. That should mean that the drbd syntax is working correctly, and there is a problem in another layer. As this is the boot device, the problem could be in pygrub - which matches the log in comment #16. Can you provide some additional details on this disk? What OS and filesystem is on it? Also, what devices are these drbd devices made of? (Individual disks, iscsi, loop devices, etc?)
Ping, Bill. Can I get some additional details on the VM, so I can setup the rest of the test here? (If the VM is small, you could even upload it to ftp://ftp.novell.com/incoming.)
Hey Mike, The VM is a basic CentOS 5 install. It's too big to upload though. Looking at the errors, if we use the drbd: directive in the config file, the block-drbd script is never called. However, if we mount the drbd block device manually to dom0, the script is called and works correctly. Therefore I think the problem is in the xl executable or the xl libraries. Thoughts?
(In reply to comment #21) > Looking at the errors, if we use the drbd: directive in the config file, the > block-drbd script is never called. > > However, if we mount the drbd block device manually to dom0, the script is > called and works correctly. > > Therefore I think the problem is in the xl executable or the xl libraries. Could be. Let's back up a bit and see if strace shows anything interesting. Can you do the following and send me the strace.log file? strace -s 500 -f -o strace.log xl create <domU> In the meantime, I'll keep looking through the xl code.
Okay I've got the output, but it's 3500 lines long. How would you like me to submit this data?
(In reply to comment #23) > Okay I've got the output, but it's 3500 lines long. How would you like me to > submit this data? You can just gzip it and attach it to this bug. However, the strace runs I've done here haven't been helpfuly, so it's probably safe to not bother submitting it. I do have some good news though, I think I've found the root of the problem. It turns out that the disk is being treated as a local device and is being passed straight through. There is a patch to run the appropriate block setup script first in this thread: http://lists.xen.org/archives/html/xen-devel/2014-02/msg01092.html. I should have some test code before too long.
I've finished building the fix for this, but it might take another 1-2 hours to sync to the download area. When that completes (you'll get a 404 until then), can you please install the xen-tools RPM from my repo here: http://download.opensuse.org/repositories/home:/mlatimer:/branches:/Virtualization/openSUSE_13.1 You do still need to leave the fix in from comment #1, but with these two fixes, I think you should be working. As soon as I get the confirmation, I'll submit both of these fixes into the two appropriate packages (xen and drbd) to make sure they don't get lost.
Created attachment 580804 [details] Local attach support for PHY backends using scripts This is the required xen patch, originally found here: http://lists.xen.org/archives/html/xen-devel/2014-02/msg01092.html
Created attachment 580805 [details] Handle drbd devices referenced as phy devices by xenstore This is the required drbd patch, originally posted here: http://lists.xen.org/archives/html/xen-devel/2014-02/msg01895.html
Awesome! Please remember we also need to capture that the DRBD binaries on 13.1 are out of date well. The shipped kernel has 8.4.3, but the shipped tools in the drbd rpm are 8.3. The tools do NOT work because they are so far out of date. I had to upgrade to drbd-8.4.3-5.8.x86_64 to get the drbd admin tools to work.
(In reply to comment #25) > I've finished building the fix for this, but it might take another 1-2 hours to > sync to the download area. When that completes (you'll get a 404 until then), > can you please install the xen-tools RPM from my repo here: > > http://download.opensuse.org/repositories/home:/mlatimer:/branches:/Virtualization/openSUSE_13.1 I'm not sure why, but the publishing process still hasn't completed... Go ahead and grab the xen-tools rpm at the following link and let me know if this solves the problem: ftp://ftp.novell.com/outgoing/xen-tools-4.4.0_04-307.1.x86_64.rpm You might also need an updated xen-libs: ftp://ftp.novell.com/outgoing/xen-libs-4.4.0_04-307.1.x86_64.rpm
(In reply to comment #29) > (In reply to comment #25) > > I've finished building the fix for this, but it might take another 1-2 hours to > > sync to the download area. When that completes (you'll get a 404 until then), > > can you please install the xen-tools RPM from my repo here: > > > > http://download.opensuse.org/repositories/home:/mlatimer:/branches:/Virtualization/openSUSE_13.1 I fixed the publishing problem on my end, so the above repo should work now.
Yaaaaay! Hey Mike that got it! I installed the following and rebooted and was able to start the guest. I still have the patched block-drbd (to use phy) in place. xen-4.4.0_04-307.1.x86_64 xen-kmp-default-4.4.0_04_k3.11.10_7-307.1.x86_64 xen-libs-4.4.0_04-307.1.x86_64 xen-tools-4.4.0_04-307.1.x86_64
Thanks for the confirmation, Bill. I haven't heard what the drbd guys are going to do with the 13.1 problem. We may need a separate bug report specific to that issue. I'll commit the Xen side of this though, and continue to push the drbd guys to handle the rest.
Okay spoke too soon. Still don't have full functionality. I'm doing some troubleshooting to see where the problem is.
Okay the host starts up, but when I try to do a live migration, it fails. xl -vvvvv migrate test 10.0.51.202 Saving to migration stream new xl format (info 0x0/0x0/1202) libxl: debug: libxl.c:770:do_libxl_domain_suspend: ao 0x1a8a9e0: create: how=(nil) callback=(nil) poller=0x1a8aac0 libxl: debug: libxl_dom.c:1240:libxl__toolstack_save: domain=14 toolstack data size=8 libxl: debug: libxl.c:798:do_libxl_domain_suspend: ao 0x1a8a9e0: inprogress: poller=0x1a8aac0, flags=i migration target: Ready to receive domain. Loading new save file <incoming migration stream> (new xl fmt info 0x0/0x0/1202) Savefile contains xl domain config libxl-save-helper: debug: starting save: Success xc: detail: xc_domain_save_suse: starting save of domid 14 xc: detail: Had 0 unexplained entries in p2m table xc: progress: Reloading memory pages: 5120/98304 5% xc: progress: Reloading memory pages: 10240/98304 10% xc: progress: Reloading memory pages: 15360/98304 15% xc: progress: Reloading memory pages: 20480/98304 20% xc: progress: Reloading memory pages: 24576/98304 25% xc: progress: Reloading memory pages: 29696/98304 30% xc: progress: Reloading memory pages: 34816/98304 35% xc: progress: Reloading memory pages: 39936/98304 40% xc: progress: Reloading memory pages: 45056/98304 45% xc: progress: Reloading memory pages: 49152/98304 50% xc: progress: Reloading memory pages: 54272/98304 55% xc: progress: Reloading memory pages: 59392/98304 60% xc: progress: Reloading memory pages: 64512/98304 65% xc: progress: Reloading memory pages: 69632/98304 70% xc: progress: Reloading memory pages: 73728/98304 75% xc: progress: Reloading memory pages: 78848/98304 80% xc: progress: Reloading memory pages: 83968/98304 85% xc: progress: Reloading memory pages: 89088/98304 90% xc: progress: Reloading memory pages: 94208/98304 95% xc: detail: delta 6115ms, dom0 70%, target 0%, sent 524Mb/s, dirtied 9Mb/s 1800 pages xc: detail: delta 20ms, dom0 95%, target 0%, sent 948Mb/s, dirtied 67Mb/s 41 pages xc: detail: Start last iteration libxl: debug: libxl_dom.c:1070:libxl__domain_suspend_common_callback: issuing PV suspend request via XenBus control node libxl: debug: libxl_dom.c:1074:libxl__domain_suspend_common_callback: wait for the guest to acknowledge suspend request xc: progress: Reloading memory pages: 98381/98304 100% libxl: debug: libxl_dom.c:1121:libxl__domain_suspend_common_callback: guest acknowledged suspend request libxl: debug: libxl_dom.c:1125:libxl__domain_suspend_common_callback: wait for the guest to suspend libxl: debug: libxl_dom.c:1139:libxl__domain_suspend_common_callback: guest has suspended xc: detail: SUSPEND shinfo 000cf1c3 xc: detail: delta 204ms, dom0 7%, target 0%, sent 6Mb/s, dirtied 18Mb/s 118 pages xc: detail: delta 1ms, dom0 100%, target 0%, sent 3866Mb/s, dirtied 3866Mb/s 118 pages xc: detail: Total pages sent= 98540 (1.00x) xc: detail: (of which 0 were fixups) xc: detail: All memory is saved xc: detail: Save exit of domid 14 with rc=0, errno=0 libxl-save-helper: debug: complete r=0: Success libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x1a8a9e0: complete, rc=0 libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x1a8a9e0: destroy libxl: error: libxl_device.c:934:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd/7/2049 libxl: error: libxl_device.c:934:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd/7/2050 libxl: error: libxl_create.c:1054:domcreate_launch_dm: unable to add disk devices libxl: error: libxl_device.c:934:device_backend_callback: unable to remove device with path /local/domain/0/backend/vbd/7/2049 libxl: error: libxl_device.c:934:device_backend_callback: unable to remove device with path /local/domain/0/backend/vbd/7/2050 libxl: error: libxl.c:1478:devices_destroy_cb: libxl__devices_destroy failed for 7 migration target: Domain creation failed (code -3). libxl: error: libxl_utils.c:396:libxl_read_exactly: file/stream truncated reading ready message from migration receiver stream libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration target process [25449] exited with error status 3 Migration failed, resuming at sender. libxl: debug: libxl.c:433:libxl_domain_resume: ao 0x1a8a9e0: create: how=(nil) callback=(nil) poller=0x1a8aac0 libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x1a8a9e0: complete, rc=0 libxl: debug: libxl.c:436:libxl_domain_resume: ao 0x1a8a9e0: inprogress: poller=0x1a8aac0, flags=ic libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x1a8a9e0: destroy xc: debug: hypercall buffer: total allocations:23 total releases:23 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:18 misses:2 toobig:3
Do you know if live migrations are working correctly without drbd in the mix? I'll look into the code and see if there is some other location where the previous problem could exist.
When you tested live migration, is the second node running the following versions of xen components? > xen-4.4.0_04-307.1.x86_64 > xen-kmp-default-4.4.0_04_k3.11.10_7-307.1.x86_64 > xen-libs-4.4.0_04-307.1.x86_64 > xen-tools-4.4.0_04-307.1.x86_64
Yes everything is identical. However, I tested again again to confirm and I can't reproduce. I migrated back-and-forth several times without errors. In between the last test and this test, the xen dom0s were powered down. There may have been some deleted libraries or something that were held open until the reboot. So at this point the reported error looks like it was due to the upgrade.
(In reply to comment #37) > Yes everything is identical. > > However, I tested again again to confirm and I can't reproduce. I migrated > back-and-forth several times without errors. > > In between the last test and this test, the xen dom0s were powered down. > There may have been some deleted libraries or something that were held open > until the reboot. > > So at this point the reported error looks like it was due to the upgrade. Thanks for the good news. I'll leave this open for a bit just in case, but I do expect to be able to close this soon.
openSUSE-SU-2014:1279-1: An update that solves 10 vulnerabilities and has 8 fixes is now available. Category: security (important) Bug References: 798770,820873,833483,842006,858178,862608,864801,865682,867910,878841,880751,881900,891539,895798,895799,895802,896023,897657 CVE References: CVE-2013-4344,CVE-2013-4540,CVE-2014-2599,CVE-2014-3967,CVE-2014-3968,CVE-2014-4021,CVE-2014-7154,CVE-2014-7155,CVE-2014-7156,CVE-2014-7188 Sources used: openSUSE 12.3 (src): xen-4.2.4_04-1.32.1
openSUSE-SU-2014:1281-1: An update that solves 10 vulnerabilities and has four fixes is now available. Category: security (important) Bug References: 798770,820873,842006,864801,865682,875668,878841,880751,882127,895798,895799,895802,896023,897657 CVE References: CVE-2013-4344,CVE-2013-4540,CVE-2014-3124,CVE-2014-3967,CVE-2014-3968,CVE-2014-4021,CVE-2014-7154,CVE-2014-7155,CVE-2014-7156,CVE-2014-7188 Sources used: openSUSE 13.1 (src): xen-4.3.2_02-27.1
SUSE-SU-2014:1318-1: An update that solves 10 vulnerabilities and has 7 fixes is now available. Category: security (moderate) Bug References: 798770,833483,842006,858178,862608,864801,865682,867910,878841,880751,881900,882092,891539,895798,895799,895802,897657 CVE References: CVE-2013-4344,CVE-2013-4540,CVE-2014-2599,CVE-2014-3967,CVE-2014-3968,CVE-2014-4021,CVE-2014-7154,CVE-2014-7155,CVE-2014-7156,CVE-2014-7188 Sources used: SUSE Linux Enterprise Software Development Kit 11 SP3 (src): xen-4.2.4_04-0.9.1 SUSE Linux Enterprise Server 11 SP3 (src): xen-4.2.4_04-0.9.1 SUSE Linux Enterprise Desktop 11 SP3 (src): xen-4.2.4_04-0.9.1