Bug 1226105

Summary: Snapshot 20240606 breaks libvirt NAT networks
Product: [openSUSE] openSUSE Tumbleweed Reporter: Eyad Issa <eyadlorenzo>
Component: Virtualization:OtherAssignee: James Fehlig <jfehlig>
Status: RESOLVED INVALID QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: alexander.graul, aplanas, eyadlorenzo, santiago.zarate
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: Output of virsh net-dumpxml default
Output of virsh test-slem5
Output of nft list ruleset ip

Description Eyad Issa 2024-06-07 22:33:49 UTC
Found after updating and trying to use a VM. The VM is connected to the NAT network but it can't connect to the external net. It does not happen on 20240605.

I suspect it's apparmor fault. On the new snapshot (which, by the way, had apparmor profiles updates) I get these error on the journal:

---
mag 28 22:12:57 xxx virtqemud[15293]: Failed to open file '/sys/kernel/security/apparmor/profiles': Permission denied
mag 28 22:12:57 xxx virtqemud[15293]: Failed to read AppArmor profiles list '/sys/kernel/security/apparmor/profiles': Permission denied
mag 28 22:12:57 xxx virtqemud[15293]: Failed to open file '/sys/kernel/security/apparmor/profiles': Permission denied
mag 28 22:12:57 xxx virtqemud[15293]: Failed to read AppArmor profiles list '/sys/kernel/security/apparmor/profiles': Permission denied
---

While I don't get them on the 20240605 snapshot. I rolled back and it works again.

On the old snapshot, running zypper dup I get:

---
The following 111 packages are going to be upgraded:
  apparmor-abstractions apparmor-docs apparmor-parser apparmor-parser-lang apparmor-profiles apparmor-utils
  apparmor-utils-lang chrony chrony-pool-openSUSE corepack21 crun dialog dialog-lang fakeroot gnome-control-center
  gnome-control-center-color gnome-control-center-goa gnome-control-center-lang gnome-control-center-user-faces
  gnome-control-center-users gnome-shell-search-provider-seahorse go1.21-doc go1.21-libstd hugo hugo-bash-completion
  just just-bash-completion just-fish-completion libapparmor-devel libapparmor1 libbpf1 libcjson1 libdialog15
  libfreerdp2-2 libkrun1 libndctl6 libsamba-policy0-python3 libstorage-ng-lang libstorage-ng-ruby libstorage-ng1
  libsvn_auth_gnome_keyring-1-0 libtelepathy-glib0 libtommath1 libtommath1-x86-64-v3 libvirt libvirt-client
  libvirt-client-qemu libvirt-daemon libvirt-daemon-common libvirt-daemon-config-network
  libvirt-daemon-config-nwfilter libvirt-daemon-driver-interface libvirt-daemon-driver-libxl
  libvirt-daemon-driver-network libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter
  libvirt-daemon-driver-qemu libvirt-daemon-driver-secret libvirt-daemon-driver-storage
  libvirt-daemon-driver-storage-core libvirt-daemon-driver-storage-disk libvirt-daemon-driver-storage-iscsi
  libvirt-daemon-driver-storage-iscsi-direct libvirt-daemon-driver-storage-logical
  libvirt-daemon-driver-storage-mpath libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-scsi
  libvirt-daemon-lock libvirt-daemon-log libvirt-daemon-plugin-lockd libvirt-daemon-proxy libvirt-daemon-qemu
  libvirt-libs libwinpr2-2 man nodejs21 nodejs21-devel npm21 openSUSE-release openSUSE-release-appliance-custom
  patterns-base-apparmor patterns-base-base patterns-base-basesystem patterns-base-basic_desktop
  patterns-base-documentation patterns-base-enhanced_base patterns-base-minimal_base patterns-base-sw_management
  patterns-base-x11 patterns-base-x11_enhanced podman python3-apparmor python311-jaraco.functools
  python311-libvirt-python python311-platformdirs ruby3.3-rubygem-ruby-augeas samba samba-ad-dc-libs samba-client
  samba-client-libs samba-libs samba-libs-python3 samba-python3 seahorse seahorse-lang subversion
  subversion-bash-completion subversion-perl typelib-1_0-TelepathyGlib-0_12 xen-libs xwayland

The following 10 patterns are going to be upgraded:
  apparmor base basesystem basic_desktop documentation enhanced_base minimal_base sw_management x11 x11_enhanced

The following product is going to be upgraded:
  openSUSE Tumbleweed  20240605-0 -> 20240606-0

The following NEW package is going to be installed:
  libkrunfw4

The following package is going to be REMOVED:
  libkrunfw3
---

Let me know if you need any more details.
Comment 1 James Fehlig 2024-06-10 16:22:32 UTC
I'm unable to reproduce using 20240607. VM is connected to the 'default' NAT network, obtains an IP address, and can reach external hosts. Nor do I see any of the journal messages you mention.

To rule out apparmor, can you stop it, unload all profiles, and see if your VM can then reach the external net? E.g.

systemctl stop apparmor.service
aa-teardown

Can you also provide the configuration of your libvirt NATed network? E.g.

virsh net-dumpxml <network-name>
Comment 2 Eyad Issa 2024-06-10 21:40:29 UTC
I tried to upgrade to 20240609 and now it apparently works again. Sorry for the trouble!
Comment 3 Alexander Graul 2024-06-11 11:57:53 UTC
I have the same issue. Existing or new VMs alike can't communicate to external hosts, e.g.:

localhost:~ # tracepath suse.com
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.122.1                                         0.159ms 
 1:  192.168.122.1                                         0.077ms 
 2:  no reply
 3:  no reply

The VM has the following network configuration:
localhost:~ # ip a
[ skipping lo device ]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:2d:e6:f1 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 192.168.122.40/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0
       valid_lft 2946sec preferred_lft 2946sec
    inet6 fe80::5054:ff:fe2d:e6f1/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
localhost:~ # ip r
default via 192.168.122.1 dev eth0 proto dhcp src 192.168.122.40 metric 20100 
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.40 metric 100 

`virsh domifaddr test-slem5` reports the same from the host (running 20240609). 

I'm running the modular deamons (libvirtd.service is not running) and will attach the output of `virsh dumpxml test-slem5` and `virsh net-dumpxml default`. I also tried other NAT networks I have defined, with the same result (all packets are dropped, tracepath only shows one hop to the hypervisor).

Please let me if there is anything else I can share to track this down.
Comment 4 Alexander Graul 2024-06-11 11:59:26 UTC
Created attachment 875420 [details]
Output of virsh net-dumpxml default
Comment 5 Alexander Graul 2024-06-11 12:00:09 UTC
Created attachment 875421 [details]
Output of virsh test-slem5
Comment 6 James Fehlig 2024-06-11 17:09:13 UTC
I'm using similar network and VM '<interface>' config, but don't see the problem. Do you have nftables installed? If so, what's the output of 'nft list ruleset ip'? Also, any iptables* packages installed?
Comment 7 Alexander Graul 2024-06-12 08:09:52 UTC
I have these nftables / iptables packages installed:

➤ zypper se -i '/(nf|ip)tables.*/'

S | Name                 | Summary                                                                     | Type
--+----------------------+-----------------------------------------------------------------------------+--------
i | iptables             | IP packet filter administration utilities                                   | package
i | iptables-backend-nft | Metapackage to make nft the default backend for iptables/arptables/ebtables | package
i | libnftables1         | nftables firewalling command interface                                      | package
i | nftables             | Userspace utility to access the nf_tables packet filter                     | package
i | python311-nftables   | Python bindings for nftables                                                | package

The 'nft list ruleset ip' output is a bit long for a comment, I'll add it as an attachment.
Comment 8 Alexander Graul 2024-06-12 08:11:07 UTC
Created attachment 875435 [details]
Output of nft list ruleset ip
Comment 9 James Fehlig 2024-06-12 18:03:59 UTC
(In reply to Alexander Graul from comment #7)
> i | iptables-backend-nft | Metapackage to make nft the default backend for
> iptables/arptables/ebtables | package

iptables-backend-nft creates two top level nftable tables that are shared by all applications using iptables. You can see them in your 'nft list ruleset ip' output, with a warning not to touch :-).

With libvirt's switch to using nftables directly, it now creates its own top level table. And recall in nftables, for a packet to get through it must be allowed by all top level tables. Notice in your output the 'ip filter' table has

type filter hook forward priority filter; policy drop;

This github issue comment has more details

https://gitlab.com/libvirt/libvirt/-/issues/644#note_1940628451

The issue also describes why it works prior to the switch to using nftables directly, and a workaround. E.g. in /etc/libvirt/network.conf, set

firewall_backend = "iptables"

and restart virtnetworkd (or libvirtd if you're using monolithic daemon). Does this work for you?

FYI, this post has a nice description of relationship between iptables, iptables-nft, and nftables

https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables#
Comment 10 Alexander Graul 2024-06-13 11:27:19 UTC
Thank you, setting firewall_backend = "iptables" works! I'll read up on the linked issue and documentation, it's an area I know too little about.

I'm closing the bug as invalid again.