Bugzilla – Bug 1226105
Snapshot 20240606 breaks libvirt NAT networks
Last modified: 2024-06-13 11:27:19 UTC
Found after updating and trying to use a VM. The VM is connected to the NAT network but it can't connect to the external net. It does not happen on 20240605. I suspect it's apparmor fault. On the new snapshot (which, by the way, had apparmor profiles updates) I get these error on the journal: --- mag 28 22:12:57 xxx virtqemud[15293]: Failed to open file '/sys/kernel/security/apparmor/profiles': Permission denied mag 28 22:12:57 xxx virtqemud[15293]: Failed to read AppArmor profiles list '/sys/kernel/security/apparmor/profiles': Permission denied mag 28 22:12:57 xxx virtqemud[15293]: Failed to open file '/sys/kernel/security/apparmor/profiles': Permission denied mag 28 22:12:57 xxx virtqemud[15293]: Failed to read AppArmor profiles list '/sys/kernel/security/apparmor/profiles': Permission denied --- While I don't get them on the 20240605 snapshot. I rolled back and it works again. On the old snapshot, running zypper dup I get: --- The following 111 packages are going to be upgraded: apparmor-abstractions apparmor-docs apparmor-parser apparmor-parser-lang apparmor-profiles apparmor-utils apparmor-utils-lang chrony chrony-pool-openSUSE corepack21 crun dialog dialog-lang fakeroot gnome-control-center gnome-control-center-color gnome-control-center-goa gnome-control-center-lang gnome-control-center-user-faces gnome-control-center-users gnome-shell-search-provider-seahorse go1.21-doc go1.21-libstd hugo hugo-bash-completion just just-bash-completion just-fish-completion libapparmor-devel libapparmor1 libbpf1 libcjson1 libdialog15 libfreerdp2-2 libkrun1 libndctl6 libsamba-policy0-python3 libstorage-ng-lang libstorage-ng-ruby libstorage-ng1 libsvn_auth_gnome_keyring-1-0 libtelepathy-glib0 libtommath1 libtommath1-x86-64-v3 libvirt libvirt-client libvirt-client-qemu libvirt-daemon libvirt-daemon-common libvirt-daemon-config-network libvirt-daemon-config-nwfilter libvirt-daemon-driver-interface libvirt-daemon-driver-libxl libvirt-daemon-driver-network libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter libvirt-daemon-driver-qemu libvirt-daemon-driver-secret libvirt-daemon-driver-storage libvirt-daemon-driver-storage-core libvirt-daemon-driver-storage-disk libvirt-daemon-driver-storage-iscsi libvirt-daemon-driver-storage-iscsi-direct libvirt-daemon-driver-storage-logical libvirt-daemon-driver-storage-mpath libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-scsi libvirt-daemon-lock libvirt-daemon-log libvirt-daemon-plugin-lockd libvirt-daemon-proxy libvirt-daemon-qemu libvirt-libs libwinpr2-2 man nodejs21 nodejs21-devel npm21 openSUSE-release openSUSE-release-appliance-custom patterns-base-apparmor patterns-base-base patterns-base-basesystem patterns-base-basic_desktop patterns-base-documentation patterns-base-enhanced_base patterns-base-minimal_base patterns-base-sw_management patterns-base-x11 patterns-base-x11_enhanced podman python3-apparmor python311-jaraco.functools python311-libvirt-python python311-platformdirs ruby3.3-rubygem-ruby-augeas samba samba-ad-dc-libs samba-client samba-client-libs samba-libs samba-libs-python3 samba-python3 seahorse seahorse-lang subversion subversion-bash-completion subversion-perl typelib-1_0-TelepathyGlib-0_12 xen-libs xwayland The following 10 patterns are going to be upgraded: apparmor base basesystem basic_desktop documentation enhanced_base minimal_base sw_management x11 x11_enhanced The following product is going to be upgraded: openSUSE Tumbleweed 20240605-0 -> 20240606-0 The following NEW package is going to be installed: libkrunfw4 The following package is going to be REMOVED: libkrunfw3 --- Let me know if you need any more details.
I'm unable to reproduce using 20240607. VM is connected to the 'default' NAT network, obtains an IP address, and can reach external hosts. Nor do I see any of the journal messages you mention. To rule out apparmor, can you stop it, unload all profiles, and see if your VM can then reach the external net? E.g. systemctl stop apparmor.service aa-teardown Can you also provide the configuration of your libvirt NATed network? E.g. virsh net-dumpxml <network-name>
I tried to upgrade to 20240609 and now it apparently works again. Sorry for the trouble!
I have the same issue. Existing or new VMs alike can't communicate to external hosts, e.g.: localhost:~ # tracepath suse.com 1?: [LOCALHOST] pmtu 1500 1: 192.168.122.1 0.159ms 1: 192.168.122.1 0.077ms 2: no reply 3: no reply The VM has the following network configuration: localhost:~ # ip a [ skipping lo device ] 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:2d:e6:f1 brd ff:ff:ff:ff:ff:ff altname enp1s0 inet 192.168.122.40/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0 valid_lft 2946sec preferred_lft 2946sec inet6 fe80::5054:ff:fe2d:e6f1/64 scope link noprefixroute valid_lft forever preferred_lft forever localhost:~ # ip r default via 192.168.122.1 dev eth0 proto dhcp src 192.168.122.40 metric 20100 192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.40 metric 100 `virsh domifaddr test-slem5` reports the same from the host (running 20240609). I'm running the modular deamons (libvirtd.service is not running) and will attach the output of `virsh dumpxml test-slem5` and `virsh net-dumpxml default`. I also tried other NAT networks I have defined, with the same result (all packets are dropped, tracepath only shows one hop to the hypervisor). Please let me if there is anything else I can share to track this down.
Created attachment 875420 [details] Output of virsh net-dumpxml default
Created attachment 875421 [details] Output of virsh test-slem5
I'm using similar network and VM '<interface>' config, but don't see the problem. Do you have nftables installed? If so, what's the output of 'nft list ruleset ip'? Also, any iptables* packages installed?
I have these nftables / iptables packages installed: ➤ zypper se -i '/(nf|ip)tables.*/' S | Name | Summary | Type --+----------------------+-----------------------------------------------------------------------------+-------- i | iptables | IP packet filter administration utilities | package i | iptables-backend-nft | Metapackage to make nft the default backend for iptables/arptables/ebtables | package i | libnftables1 | nftables firewalling command interface | package i | nftables | Userspace utility to access the nf_tables packet filter | package i | python311-nftables | Python bindings for nftables | package The 'nft list ruleset ip' output is a bit long for a comment, I'll add it as an attachment.
Created attachment 875435 [details] Output of nft list ruleset ip
(In reply to Alexander Graul from comment #7) > i | iptables-backend-nft | Metapackage to make nft the default backend for > iptables/arptables/ebtables | package iptables-backend-nft creates two top level nftable tables that are shared by all applications using iptables. You can see them in your 'nft list ruleset ip' output, with a warning not to touch :-). With libvirt's switch to using nftables directly, it now creates its own top level table. And recall in nftables, for a packet to get through it must be allowed by all top level tables. Notice in your output the 'ip filter' table has type filter hook forward priority filter; policy drop; This github issue comment has more details https://gitlab.com/libvirt/libvirt/-/issues/644#note_1940628451 The issue also describes why it works prior to the switch to using nftables directly, and a workaround. E.g. in /etc/libvirt/network.conf, set firewall_backend = "iptables" and restart virtnetworkd (or libvirtd if you're using monolithic daemon). Does this work for you? FYI, this post has a nice description of relationship between iptables, iptables-nft, and nftables https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables#
Thank you, setting firewall_backend = "iptables" works! I'll read up on the linked issue and documentation, it's an area I know too little about. I'm closing the bug as invalid again.