Bug 1215891

Summary: OpenVPN/PAM "failed to map segment from shared object" after glibc update
Product: [openSUSE] PUBLIC SUSE Linux Enterprise Server 15 SP4 Reporter: Georg Pfuetzenreuter <georg.pfuetzenreuter>
Component: BasesystemAssignee: Andreas Schwab <schwab>
Status: RESOLVED FIXED QA Contact:
Severity: Critical    
Priority: P2 - High CC: bernet, felix.niederwanger, frede, georg.pfuetzenreuter, jkosina, jslaby, markus, meissner, Michael.Dobrovnik, mjambor, opensuse, Robert, schwab, vbabka
Version: unspecified   
Target Milestone: ---   
Hardware: x86-64   
OS: SLES 15   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Georg Pfuetzenreuter 2023-10-03 09:36:03 UTC 
Hi,

after updating packages on two machines acting as VPN gateways, we observe OpenVPN (from the Basesystem module) to fail PAM authentication after some hours of operation, with many messages such as the following being printed:

```
Oct 02 02:50:16 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_unix.so): /usr/lib64/libgssapi_krb5.so.2: cannot apply additional memory protection after relocation: Cannot allocate memory
Oct 02 02:50:26 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_unix.so): libcom_err.so.2: failed to map segment from shared object
Oct 02 02:50:50 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_unix.so): /lib64/security/pam_unix.so: cannot map zero-fill pages
...
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_unix.so): /lib64/security/pam_unix.so: failed to map segment from shared object
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM adding faulty module: /lib64/security/pam_unix.so
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_umask.so): /lib64/security/pam_umask.so: failed to map segment from shared object
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM adding faulty module: /lib64/security/pam_umask.so
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_mail.so): /lib64/security/pam_mail.so: failed to map segment from shared object
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM adding faulty module: /lib64/security/pam_mail.so
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_warn.so): /lib64/security/pam_warn.so: failed to map segment from shared object
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM adding faulty module: /lib64/security/pam_warn.so
Oct 02 02:57:34 carmen5 openvpn[2558]: PAM unable to dlopen(/lib64/security/pam_deny.so): /lib64/security/pam_deny.so: failed to map segment from shared object
```

We found downgrading glibc from 2.31-150300.58.1 to glibc-2.31-150300.52.2 mitigates the issue.

Besides glibc, the following versions are in use:

```
# zypper se -is glibc openvpn pam

S  | Name                    | Type    | Version             | Arch   | Repository
---+-------------------------+---------+---------------------+--------+------------------------------
i+ | glibc                   | package | 2.31-150300.52.2    | x86_64 | SLE-Module-Basesystem-UPDATES
i  | glibc-devel             | package | 2.31-150300.58.1    | x86_64 | SLE-Module-Basesystem-UPDATES
i  | glibc-extra             | package | 2.31-150300.58.1    | x86_64 | SLE-Module-Basesystem-UPDATES
i  | glibc-locale            | package | 2.31-150300.58.1    | x86_64 | SLE-Module-Basesystem-UPDATES
i  | glibc-locale-base       | package | 2.31-150300.58.1    | x86_64 | SLE-Module-Basesystem-UPDATES
i+ | ha-openvpn-script       | package | 0.1-19.90           | noarch | (System Packages)
i  | linux-glibc-devel       | package | 5.14-150400.6.6.1   | x86_64 | SLE-Module-Basesystem-UPDATES
i+ | openvpn                 | package | 2.5.6-150400.3.6.1  | x86_64 | SLE-Module-Basesystem-UPDATES
i+ | openvpn-auth-pam-plugin | package | 2.5.6-150400.3.6.1  | x86_64 | SLE-Module-Basesystem-UPDATES
i  | pam                     | package | 1.3.0-150000.6.61.1 | x86_64 | SLE-Module-Basesystem-UPDATES
i  | pam-config              | package | 1.1-3.3.1           | x86_64 | SLE-Module-Basesystem-POOL

# uname -r
5.14.21-150400.24.81-default
# grep PRETTY /etc/os-release
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"
```

Would appreciate any advice.
Original internal discussion in https://suse.slack.com/archives/C02CGEGEZ7E/p1696257421377779.
Comment 1 Andreas Schwab 2023-10-04 11:34:19 UTC 
/lib64/security/pam_unix.so contains segments with an alignment bigger than the pages size, so it surely triggers the new behaviour.

That looks as if the number of vm segments approaches vm.max_map_count, when mprotect needs to split a vm segment.  Can you please attach a dump of /proc/$PID/maps when the error happens?
Comment 2 Georg Pfuetzenreuter 2023-10-04 11:39:40 UTC 
Hi Andreas,

thanks for getting back. I will re-install the new version to re-introduce the behavior on Thursday and report back once it happens.
Comment 3 Andreas Schwab 2023-10-04 11:44:41 UTC 
Never mind, I was able to find a simple reproducer.
Comment 4 Andreas Schwab 2023-10-04 12:52:03 UTC 
*** Bug 1215805 has been marked as a duplicate of this bug. ***
Comment 5 Marcus Meissner 2023-10-04 13:42:39 UTC 
*** Bug 1215923 has been marked as a duplicate of this bug. ***
Comment 7 Andreas Schwab 2023-10-09 14:38:09 UTC 
*** Bug 1215992 has been marked as a duplicate of this bug. ***
Comment 8 Joerg Frede 2023-10-14 12:49:00 UTC 
So we have now 5 people that have this problem 4 tickets are merged together and nobody cares about it? 
Can we at least set this to confirmed?
Comment 9 Dirk Stoecker 2023-10-16 08:38:01 UTC 
I'd also like to know what happens. I stopped security updates an about 70 servers, because risking the login capability is really nothing I want to have beside the already know issue that IMAP wont work after the updates.
Comment 10 Maintenance Automation 2023-10-18 12:30:08 UTC 
SUSE-SU-2023:4110-1: An update that solves one vulnerability and has one security fix can now be installed.

Category: security (important)
Bug References: 1215286, 1215891
CVE References: CVE-2023-4813
Sources used:
SUSE Linux Enterprise Micro for Rancher 5.3 (src): glibc-2.31-150300.63.1
SUSE Linux Enterprise Micro 5.3 (src): glibc-2.31-150300.63.1
SUSE Linux Enterprise Micro for Rancher 5.4 (src): glibc-2.31-150300.63.1
SUSE Linux Enterprise Micro 5.4 (src): glibc-2.31-150300.63.1
SUSE Linux Enterprise Micro 5.5 (src): glibc-2.31-150300.63.1
Basesystem Module 15-SP4 (src): glibc-2.31-150300.63.1
Basesystem Module 15-SP5 (src): glibc-2.31-150300.63.1
Development Tools Module 15-SP4 (src): glibc-2.31-150300.63.1, glibc-utils-src-2.31-150300.63.1
Development Tools Module 15-SP5 (src): glibc-2.31-150300.63.1, glibc-utils-src-2.31-150300.63.1
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP3 (src): glibc-2.31-150300.63.1, glibc-utils-src-2.31-150300.63.1
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src): glibc-2.31-150300.63.1, glibc-utils-src-2.31-150300.63.1
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src): glibc-2.31-150300.63.1, glibc-utils-src-2.31-150300.63.1
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src): glibc-2.31-150300.63.1, glibc-utils-src-2.31-150300.63.1
SUSE Manager Proxy 4.2 (src): glibc-2.31-150300.63.1
SUSE Manager Retail Branch Server 4.2 (src): glibc-2.31-150300.63.1
SUSE Manager Server 4.2 (src): glibc-2.31-150300.63.1
SUSE Enterprise Storage 7.1 (src): glibc-2.31-150300.63.1, glibc-utils-src-2.31-150300.63.1
SUSE Linux Enterprise Micro 5.1 (src): glibc-2.31-150300.63.1
SUSE Linux Enterprise Micro 5.2 (src): glibc-2.31-150300.63.1
SUSE Linux Enterprise Micro for Rancher 5.2 (src): glibc-2.31-150300.63.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 11 Robert Divko 2023-10-18 13:37:33 UTC 
Works for me --> no pam errors anymore (related to
Bug 1215805 - keine Authentifizierung per dovecot nach update; PAM adding faulty module), what was the problem?
Comment 12 Marcus Meissner 2023-10-19 07:25:51 UTC 
the fix references:

- dl-map-segment-align-munmap.patch: elf: Align argument of __munmap to
page size (bsc#1215891, BZ #28676)

so basically a previous fix was incorrect in regards to memory alignment which led to the failures yoiu saw
Comment 13 Joerg Frede 2023-10-23 06:42:11 UTC 
Seems to work also for Leap 15.5 tnx.