Bug 1182227

Created attachment 846102 [details]
startup

Created attachment 846103 [details]
rdsosreport

Most likely due to the wicked dracut module rewrite.

(In reply to Fabian Vogt from comment #3)
> Most likely due to the wicked dracut module rewrite.

Yep, adding support for that in combustion was not hard.

Unfortunately the wicked dracut integration seems to be broken:
- With just "rd.neednet=1", it doesn't output a config at all (/tmp/dracut.xml  is empty). "ip=dhcp" appears to be necessary in addition, while that was the default previously.
- Even with "rd.neednet=1 ip=dhcp", it doesn't even try DHCP. /tmp/dracut.xml contains two <interface/> elements with empty <name/>. wicked ifup considers that an invalid config and so nothing gets configured.

There also does not appear to be any error handling whatsoever, so the system boots up just fine even though there wasn't even an attempt to get networking up.

*** Bug 1180364 has been marked as a duplicate of this bug. ***

(In reply to Fabian Vogt from comment #4)
> Unfortunately the wicked dracut integration seems to be broken:
> - With just "rd.neednet=1", it doesn't output a config at all
> (/tmp/dracut.xml  is empty). "ip=dhcp" appears to be necessary in addition,
> while that was the default previously.
> - Even with "rd.neednet=1 ip=dhcp", it doesn't even try DHCP.
> /tmp/dracut.xml contains two <interface/> elements with empty <name/>.
> wicked ifup considers that an invalid config and so nothing gets configured.
> 
> There also does not appear to be any error handling whatsoever, so the
> system boots up just fine even though there wasn't even an attempt to get
> networking up.

The dracut.xml creation is done in  modules.d/35network-wicked/wicked-config.sh
It creates the following xml file:

-->
sh-5.0# 
sh-5.0# wicked show-config --ifconfig dracut:cmdline:/tmp/cmdline.conf
<interface origin="dracut:cmdline">
  <name namespace="ethernet">
    <permanent-address>52:54:00:dc:43:b6</permanent-address>
  </name>
  <ethernet/>
  <link/>
  <ipv4>
    <enabled>true</enabled>
    <arp-verify>true</arp-verify>
  </ipv4>
  <ipv4:dhcp>
    <enabled>true</enabled>
    <update>default-route,dns,nis,ntp,nds,mtu,tz,boot</update>
    <defer-timeout>15</defer-timeout>
    <recover-lease>true</recover-lease>
    <release-lease>false</release-lease>
  </ipv4:dhcp>
</interface>
<interface origin="dracut:cmdline">
  <name namespace="ethernet">
    <permanent-address>52:54:00:dc:43:b6</permanent-address>
  </name>
  <ethernet/>
  <link/>
  <ipv4>
    <enabled>true</enabled>
    <arp-verify>true</arp-verify>
  </ipv4>
  <ipv4:dhcp>
    <enabled>true</enabled>
    <update>default-route,dns,nis,ntp,nds,mtu,tz,boot</update>
    <defer-timeout>15</defer-timeout>
    <recover-lease>true</recover-lease>
    <release-lease>false</release-lease>
  </ipv4:dhcp>
</interface>
sh-5.0# 
--<

The network device name is not present resulting in an wicked error:

-->
sh-5.0# wicked --log-level debug bootstrap --ifconfig /tmp/dracut.xml all
wicked: __ni_config_parse_ifconfig_source: Adding ifconfig firmware:
wicked: __ni_config_parse_ifconfig_source: Adding ifconfig compat:
wicked: __ni_config_parse_ifconfig_source: Adding ifconfig wicked:
wicked: cannot get interface name - config has no valid <name> node
wicked: cannot get interface name - config has no valid <name> node
wicked: skipping lo interface: no configuration provided
wicked: skipping ens3 interface: no configuration provided
sh-5.0#
--<

I've changed the dracut.xml file manually like this:

-->
sh-5.0# cat /tmp/dracut.xml
<interface origin="dracut:cmdline">
  <name>ens3</name>
  <ethernet/>
  <link/>
  <ipv4>
    <enabled>true</enabled>
    <arp-verify>true</arp-verify>
  </ipv4>
  <ipv4:dhcp>
    <enabled>true</enabled>
    <update>default-route,dns,nis,ntp,nds,mtu,tz,boot</update>
    <defer-timeout>15</defer-timeout>
    <recover-lease>true</recover-lease>
    <release-lease>false</release-lease>
  </ipv4:dhcp>
</interface>
<interface origin="dracut:cmdline">
  <name>ens3</name>
  <ethernet/>
  <link/>
  <ipv4>
    <enabled>true</enabled>
    <arp-verify>true</arp-verify>
  </ipv4>
  <ipv4:dhcp>
    <enabled>true</enabled>
    <update>default-route,dns,nis,ntp,nds,mtu,tz,boot</update>
    <defer-timeout>15</defer-timeout>
    <recover-lease>true</recover-lease>
    <release-lease>false</release-lease>
  </ipv4:dhcp>
</interface>
--<

and then the interface setup worked:

-->
sh-5.0# wicked --log-level debug bootstrap --ifconfig /tmp/dracut.xml all
wicked: __ni_config_parse_ifconfig_source: Adding ifconfig firmware:
wicked: __ni_config_parse_ifconfig_source: Adding ifconfig compat:
wicked: __ni_config_parse_ifconfig_source: Adding ifconfig wicked:
wicked: ni_ifconfig_read_wicked_xml_file: wicked:xml:/tmp/dracut.xml:1
wicked: ni_ifconfig_read_wicked_xml_file: wicked:xml:/tmp/dracut.xml:17
wicked: skipping lo interface: no configuration provided
wicked: skipping ens9 interface: no configuration provided
wicked: skipping ens10 interface: no configuration provided
wicked: ens3: configuration applied to nanny
[ 4504.035566] NET: Registered protocol family 17
ens3            up
sh-5.0# 
--<

I'm also experiencing this bug exactly as described here. I would love to add useful information if anyone has any request or way for me to be helpful.

Happy to share any relevant information as well. Facing this issue using ignition from a network location instead of local to the device.

Converting the `ip=dhcp` line to wicked´s configuration format is still not supported by wicked. This is troublesome as it means that wicked needs to probe all interfaces for a valid DHCP offer and -for the sake of NFS root scenarios- chose the first valid offer which includes a root-path option. As said, we currently have no way of representing this in our configuration format but this should be supported in the future.

A workaround would be to use the alternative syntax of `ip=<interface>:dhcp`. But even with the workaround I haven´t been able to make the nfs root use case to work as it just configures the network and stops without attempting to mount the nfs root (seems that the nfsroot.sh is unaware that the network is up?)

I can reproduce this bug, network as a dependency in Combustion fails to be satisfied.

There is no problem in activating a network interface from the emergency mode instead.

@Enrico have you found it possible to omit the [# combustion: network] and 'manually' up the interface via combustion script? That would be an interesting workaround.

(In reply to Anthony Rabbito from comment #11)
> omit the [# combustion: network] and
> 'manually' up the interface via combustion script? That would be an
> interesting workaround.

Tried. The interface does get `up` during the Combustion process, but commands after `ip link set […] up` that make use on an internet connection still fail. I'm not knowledgeable in the boot sequence (at all), so doing this might have been pointless from the start.

This looks like the bad commit? https://git.kernel.org/pub/scm/boot/dracut/dracut.git/commit/?id=4026cd3b012be7f06c7572bdf50dd54a18ecb684

(In reply to Rubén Torrero Marijnissen from comment #9)
> Converting the `ip=dhcp` line to wicked´s configuration format is still not
> supported by wicked. This is troublesome as it means that wicked needs to
> probe all interfaces for a valid DHCP offer and -for the sake of NFS root
> scenarios- chose the first valid offer which includes a root-path option. As
> said, we currently have no way of representing this in our configuration
> format but this should be supported in the future.

Ok, so I was unfortunately correct with my analysis. This makes the current network module pretty much useless when there's no previous configuration copied from the host. Can dracut revert to the old implementation?

> A workaround would be to use the alternative syntax of
> `ip=<interface>:dhcp`.

The images use net.ifnames=0 already anyway, so just using "ip=eth0:dhcp" explicitly as workaround might work around this in many cases, but not completely.

> But even with the workaround I haven´t been able to
> make the nfs root use case to work as it just configures the network and
> stops without attempting to mount the nfs root (seems that the nfsroot.sh is
> unaware that the network is up?)

Was hidden in some layers of indirection, but it's probably a missing call to /sbin/netroot when an interface is up: https://github.com/dracutdevs/dracut/blob/2d83bce21bfc874b29c1fb99e8fabb843f038725/modules.d/35network-legacy/ifup.sh#L501

(In reply to Fabian Vogt from comment #14)
> (In reply to Rubén Torrero Marijnissen from comment #9)
> > Converting the `ip=dhcp` line to wicked´s configuration format is still not
> > supported by wicked. This is troublesome as it means that wicked needs to
> > probe all interfaces for a valid DHCP offer and -for the sake of NFS root
> > scenarios- chose the first valid offer which includes a root-path option. As
> > said, we currently have no way of representing this in our configuration
> > format but this should be supported in the future.
> 
> Ok, so I was unfortunately correct with my analysis. This makes the current
> network module pretty much useless when there's no previous configuration
> copied from the host. Can dracut revert to the old implementation?

A dumb revert would be like this:

index 298b1a798..010bc49dd 100755
--- a/modules.d/40network/module-setup.sh
+++ b/modules.d/40network/module-setup.sh
@@ -17,8 +17,8 @@ depends() {
     done
 
     if [ -z "$network_handler" ]; then
-        if find_binary wicked &> /dev/null; then
-            network_handler="network-wicked"
+#        if find_binary wicked &> /dev/null; then
+#            network_handler="network-wicked"
         elif [[ -x $dracutsysrootdir/usr/libexec/nm-initrd-generator ]]; then
             network_handler="network-manager"
         else

but in order to allow the implementation to mature, this would be drastic. Maybe add a cmdline parameter like rd.usedracut? or check if someone has added the dracut module manually?

> but in order to allow the implementation to mature, this would be drastic.

This meant to day "but in order to allow the implementation to mature, we should not be so drastic, because it could no longer be selected."

$ cat /etc/dracut.conf.d/wicked.conf 
add_dracutmodules+=" network-legacy "
omit_dracutmodules+=" network-wicked "

(In reply to Andreas Schwab from comment #17)
> $ cat /etc/dracut.conf.d/wicked.conf 
> add_dracutmodules+=" network-legacy "
> omit_dracutmodules+=" network-wicked "

A valid workaround, but I'd hate to ship that.

(In reply to Daniel Molkentin from comment #15)
> (In reply to Fabian Vogt from comment #14)
> > (In reply to Rubén Torrero Marijnissen from comment #9)
> > > Converting the `ip=dhcp` line to wicked´s configuration format is still not
> > > supported by wicked. This is troublesome as it means that wicked needs to
> > > probe all interfaces for a valid DHCP offer and -for the sake of NFS root
> > > scenarios- chose the first valid offer which includes a root-path option. As
> > > said, we currently have no way of representing this in our configuration
> > > format but this should be supported in the future.
> > 
> > Ok, so I was unfortunately correct with my analysis. This makes the current
> > network module pretty much useless when there's no previous configuration
> > copied from the host. Can dracut revert to the old implementation?
> 
> A dumb revert would be like this:
> 
> index 298b1a798..010bc49dd 100755
> --- a/modules.d/40network/module-setup.sh
> +++ b/modules.d/40network/module-setup.sh
> @@ -17,8 +17,8 @@ depends() {
>      done
>  
>      if [ -z "$network_handler" ]; then
> -        if find_binary wicked &> /dev/null; then
> -            network_handler="network-wicked"
> +#        if find_binary wicked &> /dev/null; then
> +#            network_handler="network-wicked"
>          elif [[ -x $dracutsysrootdir/usr/libexec/nm-initrd-generator ]];
> then
>              network_handler="network-manager"
>          else
> 
> but in order to allow the implementation to mature, this would be drastic.

In its current state it's pretty much unusable, so it should not be enabled
by default.

> Maybe add a cmdline parameter like rd.usedracut? or check if someone has
> added the dracut module manually?

Possible. For now I sent a sr which patches module-setup.sh temporarily
to have working images again: https://build.opensuse.org/request/show/887331
Once it arrives in Tumbleweed, I will extend the openQA test case to verify
that networking in the initrd also works, so that this doesn't break again
in the future.

(In reply to Daniel Molkentin from comment #18)
> (In reply to Andreas Schwab from comment #17)
> > $ cat /etc/dracut.conf.d/wicked.conf 
> > add_dracutmodules+=" network-legacy "
> > omit_dracutmodules+=" network-wicked "
> 
> A valid workaround, but I'd hate to ship that.

That's actually what I tried at first as well, but it didn't work, even
after fixing https://github.com/dracutdevs/dracut/issues/1341.

(In reply to Rubén Torrero Marijnissen from comment #9)
> Converting the `ip=dhcp` line to wicked´s configuration format is still not
> supported by wicked. This is troublesome as it means that wicked needs to
> probe all interfaces for a valid DHCP offer and -for the sake of NFS root
> scenarios- chose the first valid offer which includes a root-path option. As
> said, we currently have no way of representing this in our configuration
> format but this should be supported in the future.
> 
> A workaround would be to use the alternative syntax of
> `ip=<interface>:dhcp`. But even with the workaround I haven´t been able to
> make the nfs root use case to work as it just configures the network and
> stops without attempting to mount the nfs root (seems that the nfsroot.sh is
> unaware that the network is up?)

Sorry that took a while.
Further tests turned out that wicked seems to be confused if both parameters:

ifname= and ip= 

contain the same interface name.
So the following command line fails:

-->
root=nfs4:192.168.57.254:/nfsroot:rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.57.141,local_lock=none,addr=192.168.57.254 ifname=ens3:52:54:00:dc:43:b6 ip=ens3:dhcp rootfstype=nfs4 rootflags=rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.57.141,local_lock=none,addr=192.168.57.254 initrd=initrd console=ttyS0
--<

removing the 'ifname=ens3:52:54:00:dc:43:b6' parameter results in the following output from 'wicked show-config --ifconfig dracut:cmdline:/home/tblume/Bugs/1180364/cmdline.17357.conf'

-->
<interface origin="dracut:cmdline">
  <name>ens3</name>
  <ethernet/>
  <link/>
  <ipv4>
    <enabled>true</enabled>
    <arp-verify>true</arp-verify>
  </ipv4>
  <ipv4:dhcp>
    <enabled>true</enabled>
    <update>default-route,dns,nis,ntp,nds,mtu,tz,boot</update>
    <defer-timeout>15</defer-timeout>
    <recover-lease>true</recover-lease>
    <release-lease>false</release-lease>
  </ipv4:dhcp>
</interface>
--<

That looks valid.
Renaming ip=ens3:dhcp to ip=eth0:dhcp outputs this xml file:

-->
<interface origin="dracut:cmdline">
  <name namespace="ethernet">
    <permanent-address>52:54:00:dc:43:b6</permanent-address>
  </name>
  <ethernet/>
  <link/>
</interface>
<interface origin="dracut:cmdline">
  <name>eth0</name>
  <ethernet/>
  <link/>
  <ipv4>
    <enabled>true</enabled>
    <arp-verify>true</arp-verify>
  </ipv4>
  <ipv4:dhcp>
    <enabled>true</enabled>
    <update>default-route,dns,nis,ntp,nds,mtu,tz,boot</update>
    <defer-timeout>15</defer-timeout>
    <recover-lease>true</recover-lease>
    <release-lease>false</release-lease>
  </ipv4:dhcp>
</interface>
--<

which also looks ok for eth0.
Removing ip=ens3 instead results in this most probably invalid config:

-->
<interface origin="dracut:cmdline">
  <name namespace="ethernet">
    <permanent-address>52:54:00:dc:43:b6</permanent-address>
  </name>
  <ethernet/>
  <link/>
</interface>
--<

So, maybe wicked just needs to ignore the ifname parameter (at least the name from there)?
It doesn't seem to be necessary.

Sorry, I should probably have mentioned that my example config above is from the duplicate bug#1180364 where the ip= parameter did contain an interface name.
With ip=dhcp it still fails, I'm trying to find a workaround from within dracut.

(In reply to Thomas Blume from comment #21)
> With ip=dhcp it still fails, I'm trying to find a workaround from within
> dracut.

wicked client (equivalent of ifup) requires per-interface configuration. That is how wicked is designed. Wicked dracut parser is simply wrong.

The dracut legacy module basically implements hotplug - it runs ifup script from within udev rule; ifup parses command line and picks up either interface-specific config or generic one. It also waits for at least one interface to come up. In addition its ifup can be called directly, is using the same parser and works whether called from within udev rule or manually.

To add feature parity to wicked module there are two possibilities

1. Command line parser should generate policy instead of configuration; this policy will be directly loaded into backend. This will move hotplug handling into wicked. It won't allow running "wicked ifup" manually if needed.

2. Implement same wrapper for wicked ifup as legacy module does - generate wicked configuration for specific interface and call wicked with it. It need not really be xml, could be good ole ifcfg file. And add udev rules to call this wrapper. And of cause name it ifup so it could be called explicitly. And add same waiting for network being up.

Considering that a) there is exactly zero documentation for wicked policies and b) running ifup/down manually still requires explicit configuration - I'd say 1 is pretty much unrealistic. For 2 one could actually simply copy legacy scripts and only add ifcfg generation. But then it begs the question - why wicked at all?

What exactly wicked module does that legacy module does not that justifies all those efforts?

Any progress on this? MicroOS carries a nasty workaround atm that should be buried rather sooner than later.

(In reply to Ludwig Nussel from comment #23)
> Any progress on this? MicroOS carries a nasty workaround atm that should be
> buried rather sooner than later.

The wicked guys are working on this.
Marius, could you comment on the progress?

What I can do (will do tomorrow) is to create a pull with cleanup of the parser
so we get rid of shuffling the `ifname=` rename-matches into interface names
used by ip=, vlan=, ...:
```
  <name namespace="ethernet">
    <permanent-address>52:54:00:dc:43:b6</permanent-address>
  </name>
```
with diverse other fixes (config generated twice, broken vlan=, ...).
The `ip=dhcp` thingy aka implement dhcp probing in bootstrap instead to just
alias it to "ifup" is not yet covered / implemented.

(In reply to Marius Tomaschewski from comment #25)
> What I can do (will do tomorrow) is to create a pull with cleanup of the
> parser
> so we get rid of shuffling the `ifname=` rename-matches into interface names
> used by ip=, vlan=, ...:
> ```
>   <name namespace="ethernet">
>     <permanent-address>52:54:00:dc:43:b6</permanent-address>
>   </name>
> ```
> with diverse other fixes (config generated twice, broken vlan=, ...).
> The `ip=dhcp` thingy aka implement dhcp probing in bootstrap instead to just
> alias it to "ifup" is not yet covered / implemented.

Unfortunately that's the most important part...

IMO all of the options the network-legacy module supports have to be supported by network-wicked as well. It's what's documented in dracut.cmdline and relied on by all users.

(In reply to Fabian Vogt from comment #26)
> > The `ip=dhcp` thingy aka implement dhcp probing in bootstrap instead to just
> > alias it to "ifup" is not yet covered / implemented.
> 
> Unfortunately that's the most important part...

I understand and will implement it as soon as I can. But please also understand,
that I have to work on approved features and this all is completely out of order.

(In reply to Marius Tomaschewski from comment #27)
> (In reply to Fabian Vogt from comment #26)
> > > The `ip=dhcp` thingy aka implement dhcp probing in bootstrap instead to just
> > > alias it to "ifup" is not yet covered / implemented.
> > 
> > Unfortunately that's the most important part...
> 
> I understand and will implement it as soon as I can. But please also
> understand,
> that I have to work on approved features and this all is completely out of
> order.

I completely understand that, but it really makes me question why network-wicked became the default when it's known incomplete and the lack of resources means that it'll stay that way for some time.

As long as network-legacy is available, MicroOS will have to continue using that instead, but I suggest switching the default back to network-legacy for all of Tumbleweed.

(In reply to Fabian Vogt from comment #26)
> IMO all of the options the network-legacy module supports have to be
> supported by network-wicked as well. It's what's documented in
> dracut.cmdline and relied on by all users.

OK, please write down what all you need except of the `ip=dhcp` and
put the features you need into some order -- preferably in jira, so
this does need to be continued out of order again as until now.

AFAIS in comment 4 above, you need also `rd.neednet=1`, which (unlike
the `ip=dhcp`) also triggers to probe, but does not require a valid
root-path from dhcp, but just an IP setup, right?

I see, "rd.route=" and several other options aren't parsed yet,
but only "ip=", "vlan=", "bond=", "team=", "bridge=" and "ifname=".

BTW: In `ifup` (where "bootstrap" is still currently aliased to),
it's needed to use also the `--ifconfig fimware:` option for ibft;
in a true "bootstrap" we can add it to a default config source set.

(In reply to Fabian Vogt from comment #28)
> I completely understand that, but it really makes me question why
> network-wicked became the default when it's known incomplete and the lack of
> resources means that it'll stay that way for some time.
> 
> As long as network-legacy is available, MicroOS will have to continue using
> that instead, but I suggest switching the default back to network-legacy for
> all of Tumbleweed.

No idea about the decision at dracut side. We've started to implement this
in wicked, because we know that it's important and required sooner or later
(AFAIR, upstream started to drop/replace network-legacy with backends),
that's all I know about.

OK, we've just discussed the importance of this issue with my Team-Lead just
now and I'll spend next week to implement the bootstrap part (`ip=dhcp` and `rd.neednet=1`, the other network opts if possible too).
Going to create pull request with the cleanup mentioned above now.

Ok, the initial cleanup is in https://github.com/openSUSE/wicked/pull/873

Thomas, Fabian,
could you please review/test if the vlan=,bond=,team=,bridge=,ip= generate
some errors/warnings on stderr in a:
 `wicked show-config --ifconfig dracut:cmdline:/tmp/dracut.cmdline`
or the configs in the output are missing something or you see coding bugs
I've added?

Great to see some progress here!

(In reply to Marius Tomaschewski from comment #32)
> Ok, the initial cleanup is in https://github.com/openSUSE/wicked/pull/873
> 
> Thomas, Fabian,
> could you please review/test if the vlan=,bond=,team=,bridge=,ip= generate
> some errors/warnings on stderr in a:
>  `wicked show-config --ifconfig dracut:cmdline:/tmp/dracut.cmdline`
> or the configs in the output are missing something or you see coding bugs
> I've added?

I built an image with the new wicked and the hack to switch back to network-legacy removed, but to my surprise it still used network-legacy. While that is IMO a good thing, I don't think that was intentional and got broken here: https://github.com/dracutdevs/dracut/pull/1405/files#r663786125

I worked around that locally for some of the tests.

Booting with "ip=eth0:dhcp rd.break" works as expected still.

For the others I don't have the necessary setup to really test them in practice.
Just looking at the config and assuming that it does what it looks like (which "ip=dhcp" for instance doesn't), I did some tests:
- "ip=192.168.123.45::192.168.123.1:255.255.255.0:foobar:eth0:dhcp": Looks good
- "ip=br0:dhcp bridge=br0:eth0": Looks good
- "ip=br0:dhcp bridge=br0:eth0,eth1": Looks good
- "ip=br0:dhcp bridge": Broken, should be treated like "bridge=br0:eth0"
- "ip=eth0:dhcp vlan=vlan5:eth0" (and other combos): Looks good
- "ip=bond0:dhcp bond": Looks good
- "ip=bond0:dhcp bond=bond0:eth0,eth1:mode=balance-rr": Broken, "wicked: dracut:cmdline bond 'bond0': rejecting invalid options: mode=balance-rr"
- "ip=team0:dhcp team=team0:eth0,eth1": Looks good

Hi colleagues, sorry for the delay.
Unfortunately I had to reconstruct all my test setups.
That's been almost done now and I started testing.

Using a wicked with Marius patches and taking Fabians hint about the wicked module into consideration, I've tested on firmware (iBFT) provided iscsi root.

The boot failed with:

-->
[    3.838523] localhost dracut-pre-udev[334]: /lib/dracut-lib.sh@412(source_all): '[' -e //lib/dracut/hooks/pre-udev/99-wicked-run.sh ']'
[    3.838523] localhost dracut-pre-udev[334]: /lib/dracut-lib.sh@415(source_all): . //lib/dracut/hooks/pre-udev/99-wicked-run.sh
[    3.838523] localhost dracut-pre-udev[334]: ///lib/dracut/hooks/pre-udev/99-wicked-run.sh@4(source): systemctl start wickedd
[    3.866282] localhost systemd[1]: Finished dracut cmdline hook.
[    3.866314] localhost dracut-pre-udev[334]: ///lib/dracut/hooks/pre-udev/99-wicked-run.sh@6(source): wicked bootstrap --ifconfig /tmp/dracut.xml all
[...]
[    3.868507] localhost wicked[351]: ifup: no matching interfaces
[    3.868680] localhost dracut-pre-udev[351]: wicked: ifup: no matching interfaces
--<

I didn't see a file dracut.cmdline created. Consequently wicked would fail reading the file:

-->
sh-5.1# wicked show-config --ifconfig dracut:cmdline:/tmp/dracut.cmdline
wicked: Unable to read config source 'dracut:cmdline:/tmp/dracut.cmdline'
sh-5.1# 
--<

The config files in /tmp in the initrd show:

-->
sh-5.1# ls -l /tmp/
total 16
-rw-r--r-- 1 root root 498 Jul  9 09:11 cmdline.212.conf
-rw-r--r-- 1 root root   0 Jul  9 09:11 dracut.xml
-rw-r--r-- 1 root root   0 Jul  9 09:11 iscsi_set_initiator
-rw-r--r-- 1 root root   8 Jul  9 09:13 iscsistarted-firmware
-rw-r--r-- 1 root root   8 Jul  9 09:13 iscsistarted-iscsi:
sh-5.1#
sh-5.1#
sh-5.1# cat cmdline.212.conf 
 rd.driver.pre=btrfs rd.driver.pre=scsi_dh_alua rd.driver.pre=scsi_dh_emc rd.driver.pre=scsi_dh_rdac rd.driver.pre=dm_multipath rd.iscsi.ibft=1 rd.iscsi.firmware=1 root=/dev/mapper/3600c0ff000daac375842d46001000000-part2 rootfstype=btrfs rootflags=rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot,subvol=@/.snapshots/1/snapshot   BOOT_IMAGE=/boot/vmlinuz-5.12.13-1-default root=UUID=4f9b8d6b-7469-419f-a5f6-585d16766561 linux console=tty0 console=ttyS0 mitigations=auto rd.debug
sh-5.1# 
sh-5.1# cat dracut.xml 
sh-5.1# 
--<

So, wicked fails to create a valid config therewith.
However, that doesn't seem to be necessary because wicked can read the information directly from firmware.

Running:

wicked bootstrap eth0

does the network setup from firmware.
I can establish the iscsi connection, mount sysroot and boot successfully then.

So, Marius, maybe wicked could try per default setting up the nic from firmware if rd.iscsi.ibft=1 is specified?

I will continue testing with my other iscsiroot setups (keeping needinfo).
Unfortunately, I'm still blocked with nfsroot, because of bug#1187463.

(In reply to Thomas Blume from comment #34)
> Running:
> 
> wicked bootstrap eth0
> 
> does the network setup from firmware.

No, IMO because it's just aliased to `ifup` and you passed an --ifconfig
option, that overrides the default source set / behavior to setup ibft or
the image is missing parts of wicked.

> So, Marius, maybe wicked could try per default setting up the nic from
> firmware if rd.iscsi.ibft=1 is specified?

It does not need rd.iscsi.ibft=1. iBFT settings are defined in BIOS and
`wicked ifup all` (later in the system) reads them directly from there.

I'm almost done with initial the `wicked bootstrap all` implementation,
incl. `ip=dhcp` (needs some tweaks+cleanup tomorrow), which is using both,
the (ibft) `firmware:` and `dracut:cmdline` config sources by default.

Let's sync on Wednesday in about the details - I should have RPMs until
then and you can fix the contra-productive (show-config) calls dracut is
doing until now...

This is an autogenerated message for OBS integration:
This bug (1182227) was mentioned in
https://build.opensuse.org/request/show/922580 Factory / wicked

This is an autogenerated message for OBS integration:
This bug (1182227) was mentioned in
https://build.opensuse.org/request/show/929130 Factory / wicked

SUSE-RU-2022:0317-1: An update that has 15 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1057592,1156920,1160654,1178357,1181163,1181812,1182227,1183407,1183495,1188019,1189560,1192164,1192311,1192353,1194392
CVE References: 
JIRA References: SLE-9750
Sources used:
SUSE Linux Enterprise Module for Basesystem 15-SP3 (src):    wicked-0.6.68-150300.4.5.1
SUSE Linux Enterprise Micro 5.1 (src):    wicked-0.6.68-150300.4.5.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2022:0318-1: An update that has 19 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1029961,1057592,1156920,1160654,1177215,1178357,1181163,1181186,1181812,1182227,1183407,1183495,1188019,1189560,1192164,1192311,1192353,1194392,954329
CVE References: 
JIRA References: SLE-9750
Sources used:
SUSE OpenStack Cloud Crowbar 8 (src):    wicked-0.6.68-38.45.1
SUSE OpenStack Cloud 8 (src):    wicked-0.6.68-38.45.1
SUSE Linux Enterprise Server for SAP 12-SP3 (src):    wicked-0.6.68-38.45.1
SUSE Linux Enterprise Server 12-SP3-LTSS (src):    wicked-0.6.68-38.45.1
SUSE Linux Enterprise Server 12-SP3-BCL (src):    wicked-0.6.68-38.45.1
SUSE Linux Enterprise Server 12-SP2-BCL (src):    wicked-0.6.68-38.45.1
HPE Helion Openstack 8 (src):    wicked-0.6.68-38.45.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

openSUSE-RU-2022:0317-1: An update that has 15 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1057592,1156920,1160654,1178357,1181163,1181812,1182227,1183407,1183495,1188019,1189560,1192164,1192311,1192353,1194392
CVE References: 
JIRA References: SLE-9750
Sources used:
openSUSE Leap 15.3 (src):    wicked-0.6.68-150300.4.5.1

SUSE-RU-2022:0315-1: An update that has 19 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1029961,1057592,1156920,1160654,1177215,1178357,1181163,1181186,1181812,1182227,1183407,1183495,1188019,1189560,1192164,1192311,1192353,1194392,954329
CVE References: 
JIRA References: SLE-9750
Sources used:
SUSE Linux Enterprise Server 12-SP5 (src):    wicked-0.6.68-3.16.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2022:0345-1: An update that has 19 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1029961,1057592,1156920,1160654,1177215,1178357,1181163,1181186,1181812,1182227,1183407,1183495,1188019,1189560,1192164,1192311,1192353,1194392,954329
CVE References: 
JIRA References: SLE-9750
Sources used:
SUSE Linux Enterprise Server for SAP 15-SP1 (src):    wicked-0.6.68-3.24.1
SUSE Linux Enterprise Server 15-SP1-LTSS (src):    wicked-0.6.68-3.24.1
SUSE Linux Enterprise Server 15-SP1-BCL (src):    wicked-0.6.68-3.24.1
SUSE Linux Enterprise High Performance Computing 15-SP1-LTSS (src):    wicked-0.6.68-3.24.1
SUSE Linux Enterprise High Performance Computing 15-SP1-ESPOS (src):    wicked-0.6.68-3.24.1
SUSE Enterprise Storage 6 (src):    wicked-0.6.68-3.24.1
SUSE CaaS Platform 4.0 (src):    wicked-0.6.68-3.24.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2022:0346-1: An update that has 19 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1029961,1057592,1156920,1160654,1177215,1178357,1181163,1181186,1181812,1182227,1183407,1183495,1188019,1189560,1192164,1192311,1192353,1194392,954329
CVE References: 
JIRA References: SLE-9750
Sources used:
SUSE Manager Server 4.1 (src):    wicked-0.6.68-3.8.1
SUSE Manager Retail Branch Server 4.1 (src):    wicked-0.6.68-3.8.1
SUSE Manager Proxy 4.1 (src):    wicked-0.6.68-3.8.1
SUSE Linux Enterprise Server for SAP 15-SP2 (src):    wicked-0.6.68-3.8.1
SUSE Linux Enterprise Server 15-SP2-LTSS (src):    wicked-0.6.68-3.8.1
SUSE Linux Enterprise Server 15-SP2-BCL (src):    wicked-0.6.68-3.8.1
SUSE Linux Enterprise Micro 5.0 (src):    wicked-0.6.68-3.8.1
SUSE Linux Enterprise High Performance Computing 15-SP2-LTSS (src):    wicked-0.6.68-3.8.1
SUSE Linux Enterprise High Performance Computing 15-SP2-ESPOS (src):    wicked-0.6.68-3.8.1
SUSE Enterprise Storage 7 (src):    wicked-0.6.68-3.8.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2022:0346-2: An update that has 19 recommended fixes and contains one feature can now be installed.

Category: recommended (moderate)
Bug References: 1029961,1057592,1156920,1160654,1177215,1178357,1181163,1181186,1181812,1182227,1183407,1183495,1188019,1189560,1192164,1192311,1192353,1194392,954329
CVE References: 
JIRA References: SLE-9750
Sources used:
SUSE Linux Enterprise Realtime Extension 15-SP2 (src):    wicked-0.6.68-3.8.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

I am trying to get an auto-setup working with MicroOS over Combustion. I think this is still a problem with now NetworkManager.

I am on MicroOS 20220712. I gave up on Ingition and am now trying Combustion. Which does not set a static IP. The script is run but the end result is again an IP coming from the DHCPC instead of the static one here. This is the configuration file:

```
#!/bin/bash
# combustion: network
# Redirect output to the console
exec > >(exec tee -a /dev/tty0) 2>&1

# Set a password for root, generate the hash with "openssl passwd -6"
echo 'root:...' | chpasswd -e

# Add a public ssh key and enable sshd
mkdir -pm700 /root/.ssh/
echo '...' >> /root/.ssh/authorized_keys
systemctl enable sshd.service

# Set up hostname and network
nmcli general hostname example.com
nmcli connection add con-name "enp1s0" ifname enp1s0 type ethernet ip4 192.168.122.10/24 gw4 192.168.122.1
nmcli connection up "enp1s0" ifname enp1s0
nmcli connection delete "Wired Connection"
nmcli connection reload

# Leave a marker
echo "Configured with combustion" >> /etc/issue.d/combustion
echo "" >> /etc/issue.d/combustion
```

Doing the "nmcli" commands by hand after the host is IP, works. Even after reboot. So i am pulling my heads out why this is not working with Combustion, which was meant for such cases, right?!

(In reply to Markus Zimmermann from comment #48)
> I am trying to get an auto-setup working with MicroOS over Combustion. I
> think this is still a problem with now NetworkManager.

Fabian, that rather looks like a combustion issue, which is quite out of scope for the assignee (wicked-maintainers).
Do you now whom to assign it?

One additional observation: shouldn't NetworkManager change the connections with the command? Because there is only the standard "Wired Connection" afterwards.

In comparison when i do that:

```
cat <<EOT >> /etc/NetworkManager/system-connections/enp1s0.nmconnection
[connection]
id=enp1s0
type=ethernet
interface-name=enp1s0

[ethernet]

[ipv4]
address1=192.168.122.10/24,192.168.122.1
method=manual
EOT
chmod 0700 /etc/NetworkManager/system-connections
chmod 0600 /etc/NetworkManager/system-connections/enp1s0.nmconnection
```

I still do not have a static IP, but now i have two connections at least showing up:

```
# nmcli connection show
NAME              UUID                                  TYPE      DEVICE 
Wired Connection  2e1e20dd-217c-4891-8a97-ce6a88659773  ethernet  enp1s0 
enp1s0            360d1330-11cf-3a91-ae05-590058fc3e82  ethernet  -- 
```

Whatever it is, this bug says to me that MicroOS cannot do a static IP address automatic setup via Combustion nor Ignation. I tried both. Same problem. I do not know what else to do since there is no cloud-init installed (AFAIK) for the KVM version and i do not know how to forward kernel arguments when i am using no location argument with "virt-install", only the .qcow2 as base.

(In reply to Markus Zimmermann from comment #50)
> One additional observation: shouldn't NetworkManager change the connections
> with the command? Because there is only the standard "Wired Connection"
> afterwards.
> 
> In comparison when i do that:
> 
> ```
> cat <<EOT >> /etc/NetworkManager/system-connections/enp1s0.nmconnection
> [connection]
> id=enp1s0
> type=ethernet
> interface-name=enp1s0
> 
> [ethernet]
> 
> [ipv4]
> address1=192.168.122.10/24,192.168.122.1
> method=manual
> EOT
> chmod 0700 /etc/NetworkManager/system-connections
> chmod 0600 /etc/NetworkManager/system-connections/enp1s0.nmconnection
> ```
> 
> I still do not have a static IP, but now i have two connections at least
> showing up:
> 
> ```
> # nmcli connection show
> NAME              UUID                                  TYPE      DEVICE 
> Wired Connection  2e1e20dd-217c-4891-8a97-ce6a88659773  ethernet  enp1s0 
> enp1s0            360d1330-11cf-3a91-ae05-590058fc3e82  ethernet  -- 
> ```
> 
> Whatever it is, this bug says to me that MicroOS cannot do a static IP
> address automatic setup via Combustion nor Ignation. I tried both. Same
> problem. I do not know what else to do since there is no cloud-init
> installed (AFAIK) for the KVM version and i do not know how to forward
> kernel arguments when i am using no location argument with "virt-install",
> only the .qcow2 as base.

Is this about configuring network for use during the combustion run or the
configured system? If it's about the former, the network configuration is
indeed completely handled by dracut and can only be set by adding ip=
options to the kernel cmdline. If you have a use-case for setting that up
with a combustion script, please file a new report.

Thanks for your work.
Sorry for the late response.
I will close this issue because of these reasons.

- [# combustion: network] works fine.
  BTW, I cannot confirm which way this is a workaround or an essential fix.
- Now MicroOS has NetworkManager instead of Wicked.
- To me, it looks like the rollout of the fix is over.

I have another issue with NetworkManager. I will file a new report it.
Markus, Could you file a new report if you still have a problem?

Now [# combustion: network] works.