Bug 1184779 - apparmor startup fails during boot with "Too many open files" if an include file includes itsself
apparmor startup fails during boot with "Too many open files" if an include f...
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: AppArmor
Current
x86-64 openSUSE Tumbleweed
: P5 - None : Normal (vote)
: ---
Assigned To: Christian Boltz
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2021-04-15 08:27 UTC by Martin Wilck
Modified: 2021-12-01 22:36 UTC (History)
1 user (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
full boot log (journalctl ) (466.11 KB, text/plain)
2021-04-15 08:28 UTC, Martin Wilck
Details
backup of etc/apparmor.d (48.68 KB, application/x-xz)
2021-04-15 12:58 UTC, Martin Wilck
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Wilck 2021-04-15 08:27:46 UTC
Since a couple of weeks, I've noticed that the startup of apparmor.service on my workstation takes considerable time during boot. Today, the system entered emergency mode with these error messages:

> Apr 15 09:51:47 apollon apparmor.systemd[2044]: Restarting AppArmor
> Apr 15 09:51:47 apollon apparmor.systemd[2044]: Reloading AppArmor profiles
> [  110.957956] apollon apparmor.systemd[2224]: AppArmor parser error for /etc/apparmor.d in profile /etc/apparmor.d/abstractions/bash at line 11: Could not open 'abi/3.0': Too many open files
> [  111.150984] apollon apparmor.systemd[2471]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.libvirtd
> [  111.346699] apollon systemd[1]: systemd-rfkill.service: Succeeded.
> [  115.639827] apollon apparmor.systemd[2505]: AppArmor parser error for /etc/apparmor.d/usr.share.openqa.script.openqa in profile /etc/apparmor.d/abstractions/bash at line 11: Could not open 'abi/3.0': Too many open files
> [  115.726333] apollon apparmor.systemd[2044]: Error: At least one profile failed to load
> [  115.727028] apollon systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
> [  115.727243] apollon systemd[1]: apparmor.service: Failed with result 'exit-code'.
> [  115.727295] apollon systemd[1]: Failed to start Load AppArmor profiles.

In emergency mode, I tried to run "systemctl start apparmor" manually, with the same result.

Because I needed the system up, I deinstalled openQA. After that, apparmor started in a wink. (note that I don't use openQA on this system. It was installed from past experiments and never removed).

> [  265.104588] apollon [RPM][2860]: erase openQA 4.6.1617960903.4aa567c96-2.1.noarch: success
> ...
> [  270.584039] apollon systemd[1]: Starting Load AppArmor profiles...
> [  270.591930] apollon apparmor.systemd[2922]: Restarting AppArmor
> [  270.591930] apollon apparmor.systemd[2922]: Reloading AppArmor profiles
> [  270.643749] apollon apparmor.systemd[2969]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.libvirtd
> [  270.648461] apollon systemd[1]: Finished Load AppArmor profiles.
> [  284.296476] apollon systemd[1]: Reloading.

apparmor 3.0.1-5
systemd-246.13-1.1.x86_64
kernel-default-5.11.11-1.2.x86_64

Note that I had a nother problem during this which I'll report separately. It may or may not be related to this one. Anyway, the other problem was gone by simply re-mounting /var/tmp in emergency mode, whereas this problem persisted until openQA was uninstalled.
Comment 1 Martin Wilck 2021-04-15 08:28:20 UTC
Created attachment 848399 [details]
full boot log (journalctl )
Comment 2 Martin Wilck 2021-04-15 08:30:33 UTC
Disclaimer: I'm reporting this here because I think it might indicate some problem. I don't know if this is reproducible.
Comment 3 Martin Wilck 2021-04-15 08:49:49 UTC
The other issue I mentioned is 1184782. It caused failure to mount /var/tmp. This happened before the apparmor problem. Note though that after I'd mounted /var/tmp in emergency mode, apparmor still wouldn't start.
Comment 4 Christian Boltz 2021-04-15 12:04:39 UTC
One of your log entries says that the one of the failures happened in an OpenQA profile, therefore I guess having the openqa-profile(s) might help to reproduce this bug.

Do you have a backup of /etc/apparmor.d/ before uninstalling OpenQA? If so, please attach it (or send it by mail if you have profiles you don't want to have in public).

If not, can you tell me which OpenQA package(s) you uninstalled?
(/var/log/zypp/history should help to answer that.)
(Bonus points if you also know which profiles they contain, but that's something I can dig out myself.)
Comment 5 Martin Wilck 2021-04-15 12:58:46 UTC
Created attachment 848413 [details]
backup of etc/apparmor.d

I can hardly believe it, I had a backup indeed.
The only package I deinstalled was openQA itself (see comment 0).
Comment 6 Christian Boltz 2021-04-15 21:49:14 UTC
Thanks! I can reproduce the error locally.

The reason for the "too many open files" is that your 
/etc/apparmor.d/local/usr.share.openqa.script.openqa has

    #include <local/usr.share.openqa.script.openqa>

so it includes itsself, and causes a nice endless loop - which makes reaching any limit for open files quite easy ;-)

Handling this will need an upstream fix for both apparmor_parser and the tools (aa-logprof etc.).

Until we have proper handling of self-includes (and even when we have it), I'd recommend not to do self includes ;-)
Comment 7 Martin Wilck 2021-04-15 21:55:45 UTC
Ugh. I hope I didn't do that... but I'm pretty sure I never touched that file.

Strange that I didn't run into this all the time then. I vaguely recall that I had a similar issue months ago but it didn't occur again and I didn't pursue it.
Comment 8 Christian Boltz 2021-04-15 22:46:41 UTC
This is one of the cases where my apparmor-profile-collector package is useful to quickly look at the profiles we ship.

local/usr.share.openqa.script.openqa looks like the file you had (including the self-include), which means you are innocent ;-)

I submitted bug 1184838 for openQA.

BTW: https://gitlab.com/apparmor/apparmor/-/merge_requests/742 is the proposed fix for the aa-* tools. I'm afraid the fix for apparmor_parser won't be that simple ;-)
Comment 9 OBSbugzilla Bot 2021-08-07 12:00:12 UTC
This is an autogenerated message for OBS integration:
This bug (1184779) was mentioned in
https://build.opensuse.org/request/show/910591 Factory / apparmor
Comment 10 Christian Boltz 2021-08-07 12:23:44 UTC
This is fixed in AppArmor 3.0.3 which I just submitted.