Bugzilla – Bug 1191532
[Build 20211008] Samba 4.15 - failed to start
Last modified: 2022-03-17 21:37:37 UTC
## Observation openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-extra_tests_filesystem@64bit fails in [cifs](https://openqa.opensuse.org/tests/1963069/modules/cifs/steps/42) ## Test suite description Maintainer: QE Core Filesystem related tests, for example snapper and btrfs features. ## Reproducible Fails since (at least) Build [20210920](https://openqa.opensuse.org/tests/1928938) ## Expected result Last good: [20210918](https://openqa.opensuse.org/tests/1927312) (or more recent) ## Further details Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=extra_tests_filesystem&version=Tumbleweed)
Oct 11 03:14:33.977220 susetest systemd[1]: Starting Samba SMB Daemon... Oct 11 03:14:34.106459 susetest update-apparmor-samba-profile[10127]: Reloading updated AppArmor profile for Samba... Oct 11 03:14:34.311268 susetest smbd[10138]: [2021/10/11 03:14:34.311197, 0] ../../source3/smbd/server.c:1738(main) Oct 11 03:14:34.311757 susetest smbd[10138]: smbd version 4.15.0-git.185.378416e547cSUSE-oS15.5-x86_64 started. Oct 11 03:14:34.311854 susetest smbd[10138]: Copyright Andrew Tridgell and the Samba Team 1992-2021 Oct 11 03:14:34.372923 susetest systemd[1]: Started Samba SMB Daemon. Oct 11 03:14:34.378281 susetest smbd[10138]: [2021/10/11 03:14:34.378238, 0] ../../lib/util/become_daemon.c:119(exit_daemon) Oct 11 03:14:34.378315 susetest smbd[10138]: exit_daemon: daemon failed to start: Samba failed to init printing subsystem, error code 13 Oct 11 03:14:34.384147 susetest systemd[1]: smb.service: Main process exited, code=exited, status=1/FAILURE Oct 11 03:14:34.384329 susetest systemd[1]: smb.service: Failed with result 'exit-code'.
Similar issues were seen in Arch after the upgrade to 4.15: https://bbs.archlinux.org/viewtopic.php?id=269906
seems there is a new helper binary 'exec-ed' by smbd type=AVC msg=audit(1633946330.327:242): apparmor="DENIED" operation="exec" profile="smbd" name="/usr/lib64/samba/samba-bgqd" pid=2431 comm="smbd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 this probably needs a new profile :( so either running in complain mode or disabling spools will workaround this man samba-bgqd NAME samba-bgqd - This is an internal helper program performing asynchronous printing-related jobs. SYNOPSIS samba-bgqd DESCRIPTION This tool is part of the samba(7) suite. samba-bgqd is an helper program to be spawned by smbd or spoolssd to perform jobs like updating the printer list or other management tasks asynchronously on demand. It is not intended to be called by users or administrators.
(In reply to Noel Power from comment #3) > type=AVC msg=audit(1633946330.327:242): apparmor="DENIED" operation="exec" > profile="smbd" name="/usr/lib64/samba/samba-bgqd" pid=2431 comm="smbd" > requested_mask="x" denied_mask="x" fsuid=0 ouid=0 > > this probably needs a new profile :( I'd recommend to add the following rule to the smbd profile: /usr/lib*/samba/samba-bgqd Px -> samba-bgqd, Also create a new empty profile /etc/apparmor.d/samba-bgqd with the following content: abi <abi/3.0>, include <tunables/global> profile samba-bgqd /usr/lib*/samba/samba-bgqd (complain) { include <abstractions/base> } rcapparmor reload rcsmbd restart and then use samba (and especially samba-bgqd) for a while. If you attach the resulting audit.log to this bugreport, I'll convert it to a profile ;-)
(In reply to Christian Boltz from comment #4) > (In reply to Noel Power from comment #3) > > type=AVC msg=audit(1633946330.327:242): apparmor="DENIED" operation="exec" > > profile="smbd" name="/usr/lib64/samba/samba-bgqd" pid=2431 comm="smbd" > > requested_mask="x" denied_mask="x" fsuid=0 ouid=0 > > > > this probably needs a new profile :( > > I'd recommend to add the following rule to the smbd profile: > > /usr/lib*/samba/samba-bgqd Px -> samba-bgqd, > > Also create a new empty profile /etc/apparmor.d/samba-bgqd with the > following content: > > abi <abi/3.0>, > include <tunables/global> > > profile samba-bgqd /usr/lib*/samba/samba-bgqd (complain) { > include <abstractions/base> > } > > rcapparmor reload > rcsmbd restart > > and then use samba (and especially samba-bgqd) for a while. > > If you attach the resulting audit.log to this bugreport, I'll convert it to > a profile ;-) thanks, I've already sortof done this (including some basic rules to just get samba running, trying to setup the printserver stuff in order to get it to 'complain' but this is a little unfamilar for me. Really appreciate the offer to help with the profile, I am sure we will take up the offer :-) when we have some data
Created attachment 853138 [details] audit log with 'dummy' template in complain mode
Created attachment 853140 [details] audit log from the 'minimal' profile (not in complain mode) we enabled that allowed smbd to start (optimistically created since I am not very apparmor enabled) see below for profile... abi <abi/3.0>, include <tunables/global> profile samba-bgqd /usr/lib*/samba/samba-bgqd { include <abstractions/base> include <abstractions/samba> include <abstractions/cups-client> include <abstractions/nameservice> @{run}/samba/samba-bgqd.pid rwk, # Site-specific additions and overrides. See local/README for details. include if exists <local/usr.sbin.samba.bgqd> }
@Christian if there is any more info we can provide please let use know
(In reply to Noel Power from comment #7) > see below for profile... Your profile looks quite good, even if there are a few small differences to what the log says ;-) My result/profile (based on the two logs) is: - additions to the usr.sbin.smbd profile: signal send set=term peer=samba-bgqd, /usr/lib*/samba/samba-bgqd Px -> samba-bgqd, - samba-bgqd profile: abi <abi/3.0>, include <tunables/global> profile samba-bgqd /usr/lib*/samba/samba-bgqd { include <abstractions/base> include <abstractions/cups-client> include <abstractions/nameservice> include <abstractions/samba> # capability net_admin, # configure network interface etc. - really? signal receive set=term peer=smbd, @{PROC}/sys/kernel/core_pattern r, @{run}/samba/samba-bgqd.pid wk, # also r? /var/lib/samba/lock/msg.lock/[0-9]* rw, # why not @{run}/samba/msg.lock/[0-9]* ? # Site-specific additions and overrides. See local/README for details. include if exists <local/samba-bgqd> } As you can see, my profile includes some comments and questions: - capability net_admin is quite powerful (configure network interface etc.). Does samba really need that? (Note: I've seen denials for this capability for all samba binaries in your logs, and it somehow reminds me to boo#991901#c2 - even if the systemd fix was merged years ago. Does samba have code to (wild guess from another AppArmor developer) "set huge tcp buffers for performance"?) - the logs only contain wk permissions for the pid file. Is r (read) also needed? (Maybe when restarting samba or samba-bgqd?) - /var/lib/samba/lock/msg.lock/[0-9]* - AFAIK the other samba binaries all use @{run}/samba/msg.lock/[0-9]* ? (see abstractions/samba). Is this intentional or a bug in samba-bgqd? Please test with my profile (optionally with the complain flag added), and tell me if you get any ALLOWED or DENIED lines in audit.log with it.
(In reply to Christian Boltz from comment #9) > (In reply to Noel Power from comment #7) > > see below for profile... > > Your profile looks quite good, hehe only because it is a copy of the smbd profile with most things except the includes removed > even if there are a few small differences to > what the log says ;-) if you mean it maybe tries to fix things not triggered then that is entirely possible, I optimistically took content from the smbd profile and then ran hoping things 'worked' :-) > > My result/profile (based on the two logs) is: > > - additions to the usr.sbin.smbd profile: > > signal send set=term peer=samba-bgqd, > /usr/lib*/samba/samba-bgqd Px -> samba-bgqd, > > - samba-bgqd profile: > > abi <abi/3.0>, > > include <tunables/global> > > profile samba-bgqd /usr/lib*/samba/samba-bgqd { > include <abstractions/base> > include <abstractions/cups-client> > include <abstractions/nameservice> > include <abstractions/samba> > > # capability net_admin, # configure network interface etc. - really? > > signal receive set=term peer=smbd, > > @{PROC}/sys/kernel/core_pattern r, > @{run}/samba/samba-bgqd.pid wk, # also r? hmm, I stole this line from smbd profile and I don't remember intentionally dropping the r, I'd guess this was a stray and unintentional delete > /var/lib/samba/lock/msg.lock/[0-9]* rw, # why not > @{run}/samba/msg.lock/[0-9]* ? That is interesting, why not indeed, I'll need to chase this in the code, normally samba uses the 'run' directory (which is set from configure) since this location is side-by-side to the helper binary path I'd guess it is intentional but I will check. For the moment we will have to accept the path I guess (as if this is a bug it will take some time to filter from upstream) > > # Site-specific additions and overrides. See local/README for details. > include if exists <local/samba-bgqd> > } > > As you can see, my profile includes some comments and questions: > - capability net_admin is quite powerful (configure network interface etc.). > Does samba really need that? (Note: I've seen denials for this capability > for all samba binaries in your logs, and it somehow reminds me to > boo#991901#c2 - even if the systemd fix was merged years ago. Does samba > have code to (wild guess from another AppArmor developer) "set huge tcp > buffers for performance"?) urgh, I had completely wiped that from my memory (partly age, partly just avoiding bad memories :-)) no idea about large buffer setting (with samba anything weird is possible) I'll try and look further (trying to debug this stuff is pita) > - the logs only contain wk permissions for the pid file. Is r (read) also > needed? (Maybe when restarting samba or samba-bgqd?) > - /var/lib/samba/lock/msg.lock/[0-9]* - AFAIK the other samba binaries all > use @{run}/samba/msg.lock/[0-9]* ? (see abstractions/samba). Is this > intentional or a bug in samba-bgqd? see above, if you are happy lets set the path as is in the profile for now and I'll chase it further > > Please test with my profile (optionally with the complain flag added), and > tell me if you get any ALLOWED or DENIED lines in audit.log with it. will do, hopefully I'll run the same steps as before later today Thanks again Christian for the help
(In reply to Noel Power from comment #10) > > /var/lib/samba/lock/msg.lock/[0-9]* rw, # why not > > @{run}/samba/msg.lock/[0-9]* ? > > That is interesting, why not indeed, I'll need to chase this in the code, > normally samba uses the 'run' directory (which is set from configure) since > this location is side-by-side to the helper binary path I'd guess it is > intentional but I will check. For the moment we will have to accept the path > I guess (as if this is a bug it will take some time to filter from upstream) > /var/lib/samba/lock/msg.lock/[0-9]* rw, # why not > @{run}/samba/msg.lock/[0-9]* ? ok, this '/var/lib/samba/lock' path is ok, this is for internal messaging and is already covered for example in the samba abstraction by the /var/lib/samba/** rwk, rule (as it would be used by smbd, winbind etc.)
(In reply to Noel Power from comment #10) > (In reply to Christian Boltz from comment #9) > > (In reply to Noel Power from comment #7) > > > see below for profile... > > > > As you can see, my profile includes some comments and questions: > > - capability net_admin is quite powerful (configure network interface etc.). > > Does samba really need that? (Note: I've seen denials for this capability > > for all samba binaries in your logs, and it somehow reminds me to > > boo#991901#c2 - even if the systemd fix was merged years ago. Does samba > > have code to (wild guess from another AppArmor developer) "set huge tcp > > buffers for performance"?) > > urgh, I had completely wiped that from my memory (partly age, partly just > avoiding bad memories :-)) no idea about large buffer setting (with samba > anything weird is possible) I'll try and look further (trying to debug this > stuff is pita) > hmm seems we are still getting hit with that bug :/ 3853 execve("/usr/lib64/samba/samba-bgqd", ["/usr/lib64/samba/samba-bgqd", "--ready-signal-fd=47", "--parent-watch-fd=13", "--debuglevel=10", "-F"], 0x55bd0525e2a0 /* 13 vars */ <unfinished ...> 3849 <... clone3 resumed>) = 3853 [...] 3853 write(3, "[2021/10/15 11:30:56, 2, pid=38"..., 114) = 114 3853 geteuid() = 0 3853 write(3, " added interface enp1s0 ip=192."..., 88) = 88 3853 socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4 3853 getsockopt(4, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0 3853 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0 3853 getsockopt(4, SOL_SOCKET, SO_SNDBUF, [425984], [4]) = 0 3853 setsockopt(4, SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted) The socket operations above are from systemd afaics
(In reply to Noel Power from comment #12) > (In reply to Noel Power from comment #10) > > > /var/lib/samba/lock/msg.lock/[0-9]* rw, # why not > > > @{run}/samba/msg.lock/[0-9]* ? > > > > That is interesting, why not indeed, I'll need to chase this in the code, > > normally samba uses the 'run' directory (which is set from configure) since > > this location is side-by-side to the helper binary path I'd guess it is > > intentional but I will check. For the moment we will have to accept the path > > I guess (as if this is a bug it will take some time to filter from upstream) > > /var/lib/samba/lock/msg.lock/[0-9]* rw, # why not > > @{run}/samba/msg.lock/[0-9]* ? > > ok, this '/var/lib/samba/lock' path is ok, this is for internal messaging > and is already covered for example in the samba abstraction by the > > /var/lib/samba/** rwk, > > rule (as it would be used by smbd, winbind etc.) so, I am not entirely convinced that @{run}/samba/msg.lock/ is a valid lock path today, possibly this was a path used in the past, at least on tw samba is configured with '--with-lockdir=/var/lib/samba/lock' anyway I think maybe a cleanup of the samba apparmor profiles is a task for another day and/or bug
Created attachment 853166 [details] audit log from Christian's latest profile
Your new audit.log looks like we got the profile for samba-bgqd right, even if there are some (unrelated) denials left: * capability net_admin for all processes - which probably needs further investigation * requests for _read_ access to /var/log/samba/log.* operation="rename_src" profile="smbd" name="/var/log/samba/log.smbd" comm="cleanupd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 operation="rename_src" profile="smbd" name="/var/log/samba/log.smbd" comm="smbd-notifyd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 operation="rename_src" profile="smbd" name="/var/log/samba/log.smbd" comm="smbd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 operation="rename_src" profile="winbindd" name="/var/log/samba/log.wb-TESTDOMAIN1" comm="winbindd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 That's something for a separate commit ;-) (also, do you have an idea why smbd and winbindd would need read access here? "rename_src" might be a hint that they want to rename the file, but those operation names are not always 100% reliable. Also note the comm=... part which might give better hints.) * sending the term signal to samba-bgqd (already covered in comment #9, needs to be added to the smbd profile) * sending a signal to unconfined - I'd prefer not to allow this ;-) operation="signal" profile="smbd" comm="smbd" requested_mask="send" denied_mask="send" signal=term peer="unconfined" I submitted the samba-bgqd profile and the needed additions in the smbd profile to the AppArmor package and upstream to get printing with samba working again: upstream: https://gitlab.com/apparmor/apparmor/-/merge_requests/807 Tumbleweed: SR 925551 The remaining issues are worth separate (and less urgent) commits, ideally after some investigation. Oh, and thanks for checking the path used for msg.lock! (Feel free to open separate bugs for those issues to keep this bugreport readable.)
This is an autogenerated message for OBS integration: This bug (1191532) was mentioned in https://build.opensuse.org/request/show/925557 Factory / apparmor
*** Bug 1191787 has been marked as a duplicate of this bug. ***
The updated AppArmor package is included in the latest Tumbleweed shapshot.
*** Bug 1192336 has been marked as a duplicate of this bug. ***
openSUSE-RU-2021:4014-1: An update that has two recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1191532,1191690 CVE References: JIRA References: Sources used: openSUSE Leap 15.3 (src): apparmor-2.13.6-3.8.1, libapparmor-2.13.6-3.8.1
SUSE-RU-2021:4014-1: An update that has two recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1191532,1191690 CVE References: JIRA References: Sources used: SUSE MicroOS 5.1 (src): apparmor-2.13.6-3.8.1, libapparmor-2.13.6-3.8.1 SUSE Linux Enterprise Module for Server Applications 15-SP3 (src): apparmor-2.13.6-3.8.1 SUSE Linux Enterprise Module for Basesystem 15-SP3 (src): apparmor-2.13.6-3.8.1, libapparmor-2.13.6-3.8.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
openSUSE-SU-2022:0283-1: An update that solves 8 vulnerabilities, contains one feature and has two fixes is now available. Category: security (important) Bug References: 1139519,1183572,1183574,1188571,1191227,1191532,1192684,1193690,1194859,1195048 CVE References: CVE-2020-27840,CVE-2021-20277,CVE-2021-20316,CVE-2021-36222,CVE-2021-43566,CVE-2021-44141,CVE-2021-44142,CVE-2022-0336 JIRA References: SLE-23329 Sources used: openSUSE Leap 15.3 (src): apparmor-2.13.6-150300.3.11.2, krb5-1.19.2-150300.8.3.2, krb5-mini-1.19.2-150300.8.3.2, ldb-2.4.1-150300.3.10.1, libapparmor-2.13.6-150300.3.11.1, samba-4.15.4+git.324.8332acf1a63-150300.3.25.3, sssd-1.16.1-150300.23.17.3, talloc-2.3.3-150300.3.3.2, talloc-man-2.3.3-150300.3.3.1, tdb-1.4.4-150300.3.3.2, tevent-0.11.0-150300.3.3.2, tevent-man-0.11.0-150300.3.3.1
SUSE-SU-2022:0283-1: An update that solves 8 vulnerabilities, contains one feature and has two fixes is now available. Category: security (important) Bug References: 1139519,1183572,1183574,1188571,1191227,1191532,1192684,1193690,1194859,1195048 CVE References: CVE-2020-27840,CVE-2021-20277,CVE-2021-20316,CVE-2021-36222,CVE-2021-43566,CVE-2021-44141,CVE-2021-44142,CVE-2022-0336 JIRA References: SLE-23329 Sources used: SUSE Linux Enterprise Module for Server Applications 15-SP3 (src): apparmor-2.13.6-150300.3.11.2, krb5-1.19.2-150300.8.3.2 SUSE Linux Enterprise Module for Python2 15-SP3 (src): samba-4.15.4+git.324.8332acf1a63-150300.3.25.3 SUSE Linux Enterprise Module for Basesystem 15-SP3 (src): apparmor-2.13.6-150300.3.11.2, krb5-1.19.2-150300.8.3.2, ldb-2.4.1-150300.3.10.1, libapparmor-2.13.6-150300.3.11.1, samba-4.15.4+git.324.8332acf1a63-150300.3.25.3, sssd-1.16.1-150300.23.17.3, talloc-2.3.3-150300.3.3.2, talloc-man-2.3.3-150300.3.3.1, tdb-1.4.4-150300.3.3.2, tevent-0.11.0-150300.3.3.2, tevent-man-0.11.0-150300.3.3.1 SUSE Linux Enterprise Micro 5.1 (src): apparmor-2.13.6-150300.3.11.2, krb5-1.19.2-150300.8.3.2, ldb-2.4.1-150300.3.10.1, libapparmor-2.13.6-150300.3.11.1, sssd-1.16.1-150300.23.17.3, talloc-2.3.3-150300.3.3.2, tdb-1.4.4-150300.3.3.2, tevent-0.11.0-150300.3.3.2 SUSE Linux Enterprise High Availability 15-SP3 (src): samba-4.15.4+git.324.8332acf1a63-150300.3.25.3 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Just for the record: I run into this issue today on SLES15-SP3-HA on a active-passive clustered samba and SUSE-SU-2022:0283-1 is applied+reboot. Does not help but disabling Samba printing stuff does :) https://forums.opensuse.org/showthread.php/560864-Samba-failed-to-init-printing-subsystem?p=3073148#post3073148
Had the same issue. Checked the profile and the above change is already applied. So checking the profile, I found that I needed to add the following (to local/sambe-bgqd: owner /proc/*/fd/ r, That does the trick and samba does start again (with print service still enabled). BTW: samba still complains that it is missing the net_config capability, but this does not seem to harm the start (true for all smb-related processes like nmbd as well): type=AVC msg=audit(1647552511.743:157): apparmor="DENIED" operation="capable" profile="smbd" pid=1982 comm="smbd" capability=12 capname="net_admin"