Bug 1142988

Summary: networking problems using cloud-init in leap 15.1 images
Product: [openSUSE] openSUSE.org Reporter: Carsten Hoeger <choeger>
Component: 3rd party softwareAssignee: Robert Schweikert <rjschwei>
Status: RESOLVED FIXED QA Contact: E-mail List <bnc-team-screening>
Severity: Major    
Priority: P5 - None CC: alynx.zhou, choeger, prometheoid, rjschwei
Version: unspecifiedFlags: rjschwei: needinfo? (choeger)
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: /var/log/cloud-init.log
systemd-analyze plot on leap 15.1
systemd-analyze plot on leap 42.3

Description Carsten Hoeger 2019-07-26 09:23:04 UTC
Created attachment 811728 [details]
/var/log/cloud-init.log

Using the image from 2019-07-24: http://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.1/images/openSUSE-Leap-15.1-OpenStack.aarch64-0.0.4-Build6.7.qcow2

opensuse@buildservice:~> rpm -qa | grep cloud-init
cloud-init-config-suse-19.1-lp151.6.1.x86_64
cloud-init-19.1-lp151.6.1.x86_64

in bootcmd of cloud-init there's a ping -c 1 build.opensuse.org, see below
it fails resolving the hostname.

after cloud-init is done, resolving that host is working

opensuse@buildservice:~> ping -c 1 build.opensuse.org
PING login.opensuse.org (195.135.221.133) 56(84) bytes of data.
64 bytes from login.opensuse.org (195.135.221.133): icmp_seq=1 ttl=56 time=11.5 ms

--- login.opensuse.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.556/11.556/11.556/0.000 ms

This issue might be the same as with bug 1116767

opensuse@buildservice:~> cat /var/log/cloud-init-output.log
Cloud-init v. 19.1 running 'init-local' at Fri, 26 Jul 2019 09:15:56 +0000. Up 8.52 seconds.
2019-07-26 09:16:02,657 - stages.py[ERROR]: Unable to render networking. Network config is likely broken: No available network renderers found. Searched through list: ['eni', 'sysconfig', 'netplan']
Forcing raw metadata refresh
Retrieving repository 'openSUSE-Leap-15.1-Cloud' metadata [.error]
Repository 'openSUSE-Leap-15.1-Cloud' is invalid.
[openSUSE-Leap-15.1-Cloud|http://download.opensuse.org/repositories/Cloud:/Tools/openSUSE_Leap_15.1/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Leap-15.1-Cloud' because of the above error.
Forcing raw metadata refresh
Retrieving repository 'openSUSE-Leap-15.1-Non-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Non-Oss' is invalid.
[openSUSE-Leap-15.1-Non-Oss|http://download.opensuse.org/distribution/leap/15.1/repo/non-oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Leap-15.1-Non-Oss' because of the above error.
Forcing raw metadata refresh
Retrieving repository 'openSUSE-Leap-15.1-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Oss' is invalid.
[openSUSE-Leap-15.1-Oss|http://download.opensuse.org/distribution/leap/15.1/repo/oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Leap-15.1-Oss' because of the above error.
Forcing raw metadata refresh
Retrieving repository 'openSUSE-Leap-15.1-Update' metadata [.error]
Repository 'openSUSE-Leap-15.1-Update' is invalid.
[openSUSE-Leap-15.1-Update|http://download.opensuse.org/update/leap/15.1/oss] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Leap-15.1-Update' because of the above error.
Forcing raw metadata refresh
Retrieving repository 'openSUSE-Leap-15.1-Update-Non-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Update-Non-Oss' is invalid.
[repo-update-non-oss|http://download.opensuse.org/update/leap/15.1/non-oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE-Leap-15.1-Update-Non-Oss' because of the above error.
Could not refresh the repositories because of errors.
Retrieving repository 'openSUSE-Leap-15.1-Cloud' metadata [.error]
Repository 'openSUSE-Leap-15.1-Cloud' is invalid.
[openSUSE-Leap-15.1-Cloud|http://download.opensuse.org/repositories/Cloud:/Tools/openSUSE_Leap_15.1/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Cloud' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Non-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Non-Oss' is invalid.
[openSUSE-Leap-15.1-Non-Oss|http://download.opensuse.org/distribution/leap/15.1/repo/non-oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Non-Oss' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Oss' is invalid.
[openSUSE-Leap-15.1-Oss|http://download.opensuse.org/distribution/leap/15.1/repo/oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Oss' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Update' metadata [.error]
Repository 'openSUSE-Leap-15.1-Update' is invalid.
[openSUSE-Leap-15.1-Update|http://download.opensuse.org/update/leap/15.1/oss] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Update' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Update-Non-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Update-Non-Oss' is invalid.
[repo-update-non-oss|http://download.opensuse.org/update/leap/15.1/non-oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Update-Non-Oss' because of the above error.
Some of the repositories have not been refreshed because of an error.
Loading repository data...
Reading installed packages...
Package 'openssl-devel' not found.
'gcc' not found in package names. Trying capabilities.
No provider of 'gcc' found.
'libffi48-devel' not found in package names. Trying capabilities.
No provider of 'libffi48-devel' found.
'python-devel' not found in package names. Trying capabilities.
No provider of 'python-devel' found.
'python-pip' not found in package names. Trying capabilities.
No provider of 'python-pip' found.
Loading repository data...
Warning: No repositories defined. Operating only with the installed resolvables. Nothing can be installed.
Reading installed packages...
Resolving package dependencies...

The following 3 packages are going to be REMOVED:
  gettext-runtime-mini grub2 grub2-i386-pc

3 packages to remove.
After the operation, 37.6 MiB will be freed.
Continue? [y/n/v/...? shows all options] (y): y
(1/3) Removing grub2-i386-pc-2.02-lp151.21.3.1.noarch [.....done]
(2/3) Removing grub2-2.02-lp151.21.3.1.x86_64 [.....done]
Additional rpm output:
warning: /etc/default/grub saved as /etc/default/grub.rpmsave


(3/3) Removing gettext-runtime-mini-0.19.8.1-lp151.4.50.x86_64 [.....done]

Retrieving repository 'openSUSE-Leap-15.1-Cloud' metadata [.error]
Repository 'openSUSE-Leap-15.1-Cloud' is invalid.
[openSUSE-Leap-15.1-Cloud|http://download.opensuse.org/repositories/Cloud:/Tools/openSUSE_Leap_15.1/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Cloud' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Non-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Non-Oss' is invalid.
[openSUSE-Leap-15.1-Non-Oss|http://download.opensuse.org/distribution/leap/15.1/repo/non-oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Non-Oss' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Oss' is invalid.
[openSUSE-Leap-15.1-Oss|http://download.opensuse.org/distribution/leap/15.1/repo/oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Oss' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Update' metadata [.error]
Repository 'openSUSE-Leap-15.1-Update' is invalid.
[openSUSE-Leap-15.1-Update|http://download.opensuse.org/update/leap/15.1/oss] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Update' because of the above error.
Retrieving repository 'openSUSE-Leap-15.1-Update-Non-Oss' metadata [.error]
Repository 'openSUSE-Leap-15.1-Update-Non-Oss' is invalid.
[repo-update-non-oss|http://download.opensuse.org/update/leap/15.1/non-oss/] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'openSUSE-Leap-15.1-Update-Non-Oss' because of the above error.
Some of the repositories have not been refreshed because of an error.
Loading repository data...
Reading installed packages...
'grub2' not found in package names. Trying capabilities.
No provider of 'grub2' found.
/run/cloud-init/tmp/tmpuhs9p2fn.sh: line 7: pip: command not found
ping: build.opensuse.org: Name or service not known
Cloud-init v. 19.1 running 'init' at Fri, 26 Jul 2019 09:16:03 +0000. Up 15.79 seconds.
ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: |  eth0  | False |     .     |     .     |   .   | fa:16:3e:30:20:e0 |
ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |  host |         .         |
ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: +-------+-------------+---------+-----------+-------+
2019-07-26 09:16:11,219 - util.py[WARNING]: Failed to run bootcmd module bootcmd
2019-07-26 09:16:11,223 - util.py[WARNING]: Running module bootcmd (<module 'cloudinit.config.cc_bootcmd' from '/usr/lib/python3.6/site-packages/cloudinit/config/cc_bootcmd.py'>) failed
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:x3DgJiXezxe85+9HmzGMQkWAOxrimqRRVAYHEYGT5Uc root@buildservice
The key's randomart image is:
+---[RSA 2048]----+
|  +BBE. o ..o.   |
| +..+. = o . .   |
|  o. .o = o +    |
|   ... + O . o   |
|  . . . S B o +  |
| . . . . . o + +.|
|  + o       . ..=|
| . o           +.|
|               .+|
+----[SHA256]-----+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
SHA256:Wk0jXSc4y+KRq3szQfIGqMl276CLU487RQ+ectKylxM root@buildservice
The key's randomart image is:
+---[DSA 1024]----+
|           .o .  |
|         .o. o   |
|     .  .o+o     |
|    + o =+o.     |
| . * + *S+.      |
|  X E .o*        |
| o @.+.o .       |
|..+.=.o +        |
|..+= ooo o       |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
The key fingerprint is:
SHA256:5IMGxalDlioO5HV1+VQg+oG8YDKGrpq1YvJ8Cmg6aIo root@buildservice
The key's randomart image is:
+---[ECDSA 256]---+
|     ooo o..o.   |
| ...+o+ +...     |
|o..Bo+ +..o      |
|+.o *..+o ..     |
|oo   .o.S.       |
|o.   .   .       |
|=..              |
|XB ..            |
|E+=o             |
+----[SHA256]-----+
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
The key fingerprint is:
SHA256:erRbAWG5XsTBjmFINwDtcdIXdf0S+rYarjXF6jfFELA root@buildservice
The key's randomart image is:
+--[ED25519 256]--+
|    .+o+=+o+o... |
|      =o*o=  oo .|
|     . =.B  E. o.|
|      . o.o ..o .|
|       .S..  .o+ |
|       o.. . oo o|
|      . o . =. o |
|       . o + o+  |
|        . ..+o . |
+----[SHA256]-----+
Cloud-init v. 19.1 running 'modules:config' at Fri, 26 Jul 2019 09:16:28 +0000. Up 40.71 seconds.
Comment 1 Carsten Hoeger 2019-07-26 13:57:41 UTC
Maybe related (I am not a cloud-init expert):

systemd-analyze plot shows that wicked.service is started after cloud-config.target in 15.1 while it is started before cloud-config.target in 42.3.
Comment 2 Carsten Hoeger 2019-07-26 13:58:21 UTC
Created attachment 811768 [details]
systemd-analyze plot on leap 15.1
Comment 3 Carsten Hoeger 2019-07-26 13:58:38 UTC
Created attachment 811769 [details]
systemd-analyze plot on leap 42.3
Comment 4 Carsten Hoeger 2019-10-18 08:55:32 UTC
Any chance to get this fixed?
Recent images still show the same issue.
Comment 5 Raymond Finzel 2019-10-21 01:02:28 UTC
Dealing with the same issue, using Tumbleweed image from https://download.opensuse.org/tumbleweed/appliances/openSUSE-Tumbleweed-JeOS.x86_64-OpenStack-Cloud.qcow2 to spin up a DigitalOcean VM.

I've pushed a few wicked/cloud-init units around using guestfish and haven't been able to resolve the problem yet.
Comment 6 Robert Schweikert 2019-10-22 18:17:30 UTC
2019-07-26 09:16:02,657 - stages.py[ERROR]: Unable to render networking. Network config is likely broken: No available network renderers found. Searched through list: ['eni', 'sysconfig', 'netplan']

Indicates that ifup/ifdown was not found and cloud-init could not determine in which format the network configuration should be written.

Since cloud-init cannot write the network configuration networking cannot work if there is no default device configuration, such as ifcfg-eth0 built in.

This is independent of the ordering. In 19.2 currently in Cloud:Tools the order of unit files is modified. However this does not fix the issue w.r.t. cloud-init not being able to write the network configuration to begin with.
Comment 7 Raymond Finzel 2019-10-22 18:53:38 UTC
https://github.com/cloud-init/cloud-init/blob/02c8214eac857e29b40ecc65992c1da6983083e1/cloudinit/net/sysconfig.py#L745-L758

These lines contain directories that don't exist in the cloud image - could be the availability check that's failing. Trying to identify whether there are suitable symlinks that could fill the void.
Comment 8 Robert Schweikert 2019-10-22 20:06:58 UTC
Should be fixed with the latest version in CLoud:Tools cloud-init 19.2, can someone please test and confirm?
Comment 9 Raymond Finzel 2019-10-22 20:45:31 UTC
Hello, Robert thank you so much for this -

It did not fix the problem, unfortunately.

I think I've tracked the issue down. There was a change here -> https://github.com/cloud-init/cloud-init/commit/0948cdfbef2052cdf839f24d6a17d457aa9fd4d3#diff-b3309b48873bd17b3fc8f29bd84c50fa that does not appear to be in the opensuse version. "opensuse-tumbleweed" and "opensuse-leap" (which are returned from the get_linux_distro method) are not in the KNOWN_DISTROS dictionary. Instead the system_info method from that package needs to be used.

Thanks so much for all your help, and the excellent work you do maintaining this package for opensuse.
Comment 10 Raymond Finzel 2019-10-22 20:53:55 UTC
(to be clear I think the change you made shortcircuiting with ifup/ifdown was necessary, especially since those hardcoded paths are still in the most recent commit of cloud-init https://github.com/cloud-init/cloud-init/blob/02c8214eac857e29b40ecc65992c1da6983083e1/cloudinit/net/sysconfig.py , it just also needs the changes from that diff)
Comment 11 Robert Schweikert 2019-10-22 20:56:25 UTC
Thanks for spotting this, yes we need another change in the package. The condition is an "and" thus since currently "available_sysconfig() always returns False it wouldn't matter even if the first part of the check is correct, grmbl. This will take me a little bit.
Comment 12 Robert Schweikert 2019-10-22 21:14:07 UTC
OK, the next build in Cloud:Tools should fix the issue.

Another quick test?

Thanks
Comment 13 Raymond Finzel 2019-10-22 22:39:57 UTC
(In reply to Robert Schweikert from comment #12)
> OK, the next build in Cloud:Tools should fix the issue.
> 
> Another quick test?
> 
> Thanks

By jove it works! Thanks so much for this, was going to give up but I'm so glad I get to use opensuse for this project now.

If anyone else needs to get this working fast, grab the qcow2 cloud image, install libguestfs (command for opensuse host)

$ sudo zypper in libguestfs

run

$ virt-customize -a  <path_to_qcow2_file> --run-command 'zypper --non-interactive addrepo -C https://download.opensuse.org/repositories/Cloud:Tools/openSUSE_Tumbleweed/Cloud:Tools.repo' --run-command 'zypper --gpg-auto-import-keys refresh' --install cloud-init-19.2-6.1.x86_64

Thanks again Robert
Comment 14 Swamp Workflow Management 2019-10-22 23:20:05 UTC
This is an autogenerated message for OBS integration:
This bug (1142988) was mentioned in
https://build.opensuse.org/request/show/741941 Factory / cloud-init
Comment 17 Carsten Hoeger 2019-10-28 15:45:22 UTC
Yes, cloud-init is now working, thanks.
Comment 18 Carsten Hoeger 2019-10-28 15:47:06 UTC
Yes, cloud-init is now working, thanks.
Comment 19 Swamp Workflow Management 2019-11-28 20:12:30 UTC
SUSE-SU-2019:3097-1: An update that solves one vulnerability and has 6 fixes is now available.

Category: security (moderate)
Bug References: 1099358,1129124,1136440,1142988,1144363,1151488,1154092
CVE References: CVE-2019-0816
Sources used:
SUSE Linux Enterprise Module for Public Cloud 15 (src):    cloud-init-19.2-5.18.1
SUSE Linux Enterprise Module for Open Buildservice Development Tools 15 (src):    cloud-init-19.2-5.18.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 20 Swamp Workflow Management 2019-11-28 20:19:09 UTC
SUSE-SU-2019:3096-1: An update that solves one vulnerability and has 6 fixes is now available.

Category: security (moderate)
Bug References: 1099358,1129124,1136440,1142988,1144363,1151488,1154092
CVE References: CVE-2019-0816
Sources used:
SUSE Linux Enterprise Module for Public Cloud 15-SP1 (src):    cloud-init-19.2-8.11.1
SUSE Linux Enterprise Module for Open Buildservice Development Tools 15-SP1 (src):    cloud-init-19.2-8.11.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 21 Swamp Workflow Management 2019-12-04 05:11:31 UTC
openSUSE-SU-2019:2633-1: An update that solves one vulnerability and has 6 fixes is now available.

Category: security (moderate)
Bug References: 1099358,1129124,1136440,1142988,1144363,1151488,1154092
CVE References: CVE-2019-0816
Sources used:
openSUSE Leap 15.1 (src):    cloud-init-19.2-lp151.2.9.1
Comment 22 Swamp Workflow Management 2019-12-05 20:12:23 UTC
SUSE-SU-2019:3191-1: An update that solves one vulnerability and has 6 fixes is now available.

Category: security (moderate)
Bug References: 1099358,1129124,1136440,1142988,1144363,1151488,1154092
CVE References: CVE-2019-0816
Sources used:
SUSE Linux Enterprise Module for Public Cloud 12 (src):    cloud-init-19.2-37.33.1
SUSE CaaS Platform 3.0 (src):    cloud-init-19.2-37.33.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.