Bugzilla – Bug 747931
systemd shut off network at boot time in a bridge over vlan configuration
Last modified: 2012-05-30 14:09:14 UTC
Created attachment 476952 [details] Yast logs User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0 My configuration looks like this on my XEN system (Linux 3.1.9-1.4-xen #1 SMP Fri Jan 27 08:55:10 UTC 2012 (efb5ff4) x86_64 x86_64 x86_64 GNU/Linux) -- vlan 100 -- bridge 3 | eth0|- vlan 200 -- bridge 2 (192.170.2.3/255.255.255.192) | -- vlan 5 -- bridge 1 When the system restart, it first starts the network and few moments after the network is completely down. I have also notice this 'bug' on a domu. Reproducible: Always Steps to Reproduce: 1. XEN with latest update 2. configure eth0 -> vlan -> bridge 1 |-> vlan2 -> bridge 2 (ip) |-> vlan3 -> bridge 3 3. reboot Actual Results: During the reboot a ping show that the network is activated on the xen server and then shut down. I am afraid to reboot my xen server... Expected Results: systemd should not stop the network
please install systemd package http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/ from and sysconfig package from http://download.opensuse.org/repositories/home:/mtomaschewski:/branches:/openSUSE:/12.1:/Update:/Test/openSUSE_12.1/ and retry again, they contains a lot of fixes, including some for network.
I have tested those packages on a domu that has the same issue, and still have the same problem after a reboot: Feb 21 12:06:38 napier systemd[1]: Startup finished in 4s 795ms 465us (kernel) + 38s 858ms 816us (userspace) = 43s 654ms 281us. Feb 21 12:06:39 napier network[1078]: Shutting down network interfaces: Feb 21 12:06:40 napier network[1078]: eth0 device: Intel Corporation 82540EM Gigabit Ethernet Co Feb 21 12:06:40 napier ifdown[1617]: eth0 device: Intel Corporation 82540EM Gigabit Ethernet Co Feb 21 12:06:45 napier network[1078]: ..doneShutting down service network . . . . . . . . ...done
Except of the systemd and sysconfig packages, please update all samba related packages (especially samba-client) from: http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/ Note: You don't need to _use_ samba client/server -- it is enough, that the samba-client package is installed (it is by default). When this does not help, please set DEBUG=EXTRA in /etc/sysconfig/network/config, reboot and attach a tar.bz2 of /dev/.sysconfig/network when it occurs again.
I have check on a domu that has the same issue and there is no samba related packages installed. I have done the DEBUG=EXTRA thing and put the result in a file. Please that the the issue seems only appear on systems that has been upgraded from 11.4 to 12.1
Created attachment 477398 [details] DEBUG of network operations (sysconfig)
Hi, Please note that there are no samba related packages in http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/
samba packages have been removed from home:fcrozat:systemd repository because they are now available as 12.1 official updates since this morning, please make sure to install them.
# for i in exdeb.* ; do echo "`head -1 \"$i\"` $i" ; done | cut -b 12- | sort -> ifup eth0 -o hotplug executed by udev rule 14:35:14.786939917 exdeb.ifup_408.292.1106_.eth0.-o.hotplug -> network.service started by systemd 14:35:15.318825942 exdeb.network_418.1._.start -> children of network.service 14:35:20.075431866 exdeb.ifstatus_616.418._.lo 14:35:21.402162843 exdeb.ifup_685.418._.lo.-o.rc.onboot 14:35:24.797735478 exdeb.ifup_791.418._.eth0.-o.rc.onboot 14:35:28.526129276 exdeb.ifstatus_918.418._.eth0.-o.rc.onboot 14:35:31.865805686 exdeb.ifstatus_1014.418._.eth0.-o.rc.onboot -> network.service were started here and everything were fine. But then, systemd (PID 1) executed the "network-remotefs start" (as PID 1109) and "network stop" (as PID 1110): 14:35:36.142116474 exdeb.network-remotefs_1109.1._.start 14:35:36.164957394 exdeb.network_1110.1._.stop -> The "network-remotefs start" made an "exec network start -o remotefs": 14:35:38.196808810 exdeb.network_1109.1._.start.-o.remotefs that does not do anything (no remote-fs depending ifcfg's there). Looks like a dependency problem... I don't see any samba involved. Romain: When there are no samba related packages (basically samba-client), that provide /etc/sysconfig/network/if-up.d/ hooks, you don't need to update anything. Please reboot with DEBUG=EXTRA again and set also the boot parameter: noresume systemd.log_level=debug systemd.log_target=kmsg splash=0 showopts that is splash=silent -> splash=0, resume=... -> noresume, remove the "quiet" option and add the systemd.* options. After the system has booted, call: tail -500 /var/log/messages > /dev/.sysconfig/network/messages.txt dmesg > /dev/.sysconfig/network/dmesg.txt zypper se -i --detail > /dev/.sysconfig/network/pkgs.txt and attach an archive of /dev/.sysconfig/network again. BTW: I'm currently using: # zypper se -i --sort-by-repo --detail | grep -E "systemd|sysconfig" i | systemd | Paket | 37-3.153.1 | i586 | repo-fcrozat i | systemd-presets-branding-openSUSE | Paket | 0.1.0-32.1 | noarch | repo-fcrozat i | systemd-sysvinit | Paket | 37-3.153.1 | i586 | repo-fcrozat i | sysconfig | Paket | 0.75.4-2.10.test.1 | i586 | repo-mt-12.1 Hmm... why I've got an i586 kvm machine when installing from url? -> Bug :-)
Hi, Oups, I have reboot yesterday before reading your post. I have set the debug=extra in sysconfig/network and here are the results. I will reboot the weekend with those parameters. But, please not that this time after the reboot (systemd and sysconfig package updated with the rpm from your repo and samba packages updates with latest available packages from the official repo), the network works. On the domy where I have the same issue and no samba rpms installed, I will test also those extra parameters
Created attachment 477903 [details] dom0 debug extra files On the xen dom0 server where I have the issue, thos are the files resulting after the debug=extra in sysconfig/network
(In reply to comment #9) > Hi, [...] > with latest available packages from the official repo), the network works. This sounds good. The logs of the test are in comment 10, right? At least I don't see any log files that the network would be stopped after start... When it works, I don't need the systemd logs, that is you don't need to boot with enabled systemd debug again. Only when it does not work ... > On the domy where I have the same issue and no samba rpms installed, I will > test also those extra parameters OK, thanks for your testing!!
Created attachment 479774 [details] sysconfig debug with boot parameters Here are the systemd, messages, dmesg and network logs. Please note that the issue reappear even after updating to latest systemd and samba packages...
(In reply to comment #12) > Created an attachment (id=479774) [details] > sysconfig debug with boot parameters > > Here are the systemd, messages, dmesg and network logs. > Please note that the issue reappear even after updating to latest systemd > and samba packages... OK, ... then let's reassign this one to systemd maintainer. Bug 727771 is also open again :-/
which version of systemd package is installed on your system right now ? could you try to downgrade from one version and see if it works better.
please test systemd package from http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/ (wait until the date there has been updated to May 23 at minimal, package is in rebuilding state atm), it should fix this issue.
This is an autogenerated message for OBS integration: This bug (747931) was mentioned in https://build.opensuse.org/request/show/122059 Factory / systemd
(In reply to comment #15) > please test systemd package from > http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/ > (wait until the date there has been updated to May 23 at minimal, package is in > rebuilding state atm), it should fix this issue. I have two Tumbleweed DomU's on a relatively slow Xen-Host (Intel(R) Xeon(TM) CPU 3.20GHz). Dom0 is a Tumbleweed install too. I too had the problem of a shut off network after boot. I was able to reproduce this quite reliably when I restarted two DomU's at the same time or when for whatever reason there was a high load in the system. I installed the above packages. They make the problem occur much less often, but it is still there. Before this patch it was quite sure that at least one of the both rebooted DomU's shut down its network after boot. Now in about one of four reboots, one of the systems shuts down its network. /dev/.sysconfig/network tgz attached
Created attachment 492332 [details] /dev/.sysconfig/network
please make sure you have systemd-37-3.164.1 or later installed. 37-3.163.1 didn't had the patch
(In reply to comment #19) > please make sure you have systemd-37-3.164.1 or later installed. Ok - updated. 10 simultanious reboots of both DomU's done, no problems so far. For me the problem is solved. Thanks a lot. Can you explain what you have done for the fix? And when can we expect this update in mainline 12.1/Tumbleweed?
In fact, we had fixed the bug sometime ago (making sure network.service is considered a "RemainAfterExit" service, ie when no process are remaining in its cgroup, it won't become dead) but there was a bug in the parsing code I added in systemd to process the "X-Systemd-RemainAfterExit" LSB header (in /etc/init.d/network ) which was ignored when added after Description: header. Let's push this fix as maintenance update !
pushed to maintenance team : mr#122366
Thanks for the submission. I started an update for 12.1.
This is an autogenerated message for OBS integration: This bug (747931) was mentioned in https://build.opensuse.org/request/show/122369 Factory / systemd
Update released. Resolved fixed.
openSUSE-RU-2012:0664-1: An update that has four recommended fixes can now be installed. Category: recommended (low) Bug References: 727771,746594,746595,747931 CVE References: Sources used: openSUSE 12.1 (src): systemd-37-3.14.1, systemd-gtk-37-3.14.1