|
Bugzilla – Full Text Bug Listing |
| Summary: | MMTests/gitsource: About 8 percent performance decrease between native and container run | ||
|---|---|---|---|
| Product: | [openSUSE] openSUSE Distribution | Reporter: | Andreas Herrmann <aherrmann> |
| Component: | Containers | Assignee: | Andreas Herrmann <aherrmann> |
| Status: | RESOLVED WONTFIX | QA Contact: | E-mail List <qa-bugs> |
| Severity: | Normal | ||
| Priority: | P5 - None | CC: | aherrmann, jack, kernel-performance-bugs |
| Version: | Leap 15.4 | ||
| Target Milestone: | --- | ||
| Hardware: | Other | ||
| OS: | Other | ||
| Whiteboard: | |||
| Found By: | --- | Services Priority: | |
| Business Priority: | Blocker: | --- | |
| Marketing QA Status: | --- | IT Deployment: | --- |
Can you please share a guide how to reproduce these numbers? Also, what platform & podman versions are you on? That's on x86. Kernel 5.14.21-150400.24.63-default. podman: podman version 4.4.4 runc: runc version 1.1.5 commit: v1.1.5-0-gf19387a6bec4 spec: 1.0.2-dev go: go1.19.9 libseccomp: 2.5.3 To reproduce something like the following should work (mainline misses some patches that are WIP to support the container run of gitsource, also the version bump to v2.16.6 of gitsource is missing) # git clone https://github.com/gormanm/mmtests.git # cd mmtests # ./run-mmtests.sh -n -c configs/config-workload-shellscripts native-run # ./run-container.sh -n -c configs/config-workload-shellscripts podman-run # cd work/log # compare-kernels --basline native-run --compare podman-run Note: mmtests expects that benchmarks are run as root. I plan to look into this myself. E.g. to do some profiling. (In reply to Andreas Herrmann from comment #2) > That's on x86. > Kernel 5.14.21-150400.24.63-default. > > podman: > podman version 4.4.4 > > runc: > runc version 1.1.5 > commit: v1.1.5-0-gf19387a6bec4 > spec: 1.0.2-dev > go: go1.19.9 > libseccomp: 2.5.3 > > To reproduce something like the following should work > (mainline misses some patches that are WIP to support the container run of > gitsource, also the version bump to v2.16.6 of gitsource is missing) I'll add a comment when those pending patches are in the upstream repo. > # git clone https://github.com/gormanm/mmtests.git > # cd mmtests > # ./run-mmtests.sh -n -c configs/config-workload-shellscripts native-run > # ./run-container.sh -n -c configs/config-workload-shellscripts podman-run > # cd work/log > # compare-kernels --basline native-run --compare podman-run That should be: # ../../compare-kernels.sh --baseline ... > Note: mmtests expects that benchmarks are run as root. > > I plan to look into this myself. E.g. to do some profiling. The nature of gitsource benchmark is to create a lot of short living processes. It was suggested to use a simpler workload with similar characteristics to try to reproduce and profile. I've used a shell_bensh.sh script as suggested at https://lore.kernel.org/all/20230608111408.s2minsenlcjow7q3@quack3/ but with 50000 iterations. For this workload the comparison is (with spec_store_bypass mitigation for seccomp threads): native podman Amean User 17.26 ( 0.00%) 18.58 * -7.67%* Amean System 4.42 ( 0.00%) 5.23 * -18.40%* Amean Elapsed 21.79 ( 0.00%) 24.00 * -10.13%* Amean CPU 99.00 ( 0.00%) 99.00 ( 0.00%) For this workload the comparison is (without spec_store_bypass mitigation for seccomp threads): native podman Amean User 17.29 ( 0.00%) 17.55 * -1.50%* Amean System 4.44 ( 0.00%) 4.84 * -9.08%* Amean Elapsed 21.85 ( 0.00%) 22.58 * -3.34%* Amean CPU 99.00 ( 0.00%) 99.00 ( 0.00%) Thus this workload does not show the same amount of overhead as the gitsource benchmark for the container environment. Interesting, thanks for the tests! So this suggests there's something else than the pure fork+exec() being more costly in podman. BTW, did you run shell_bench.sh bound to one (or two) CPUs as Mel has suggested? (In reply to Jan Kara from comment #5) > Interesting, thanks for the tests! So this suggests there's something else > than the pure fork+exec() being more costly in podman. Yes, most likely something else. > BTW, did you run > shell_bench.sh bound to one (or two) CPUs as Mel has suggested? The provided results are for tests where I had bound execution to one CPU. (I had also done tests with binding it to two CPUs. It didn't make a big difference.) I've started to narrow down which of the many scripts in git-source/t (make test) are contributing disproportionately to the performance drop under container environment. So far I've figured that tests using GPG show a significant drop. I'll dig further into this and check what else shows significant overhead. FYI.
Performance numbers split into categories for each 't[0-9]' prefix look as follows:
native: 22.15 podman: 23.13 104.00% .98 (gitsource-t8)
native: 54.09 podman: 55.47 102.00% 1.38 (gitsource-t9)
native: 37.37 podman: 39.66 106.00% 2.29 (gitsource-t2)
native: 60.36 podman: 63.00 104.00% 2.64 (gitsource-t0)
native: 55.75 podman: 58.69 105.00% 2.94 (gitsource-t1)
native: 57.64 podman: 63.08 109.00% 5.44 (gitsource-t4)
native: 55.79 podman: 61.45 110.00% 5.66 (gitsource-t6)
native: 118.50 podman: 128.73 108.00% 10.23 (gitsource-t5)
native: 162.32 podman: 173.35 106.00% 11.03 (gitsource-t3)
native: 152.55 podman: 172.05 112.00% 19.50 (gitsource-t7)
So tests t7*.sh added about 19 seconds of additional runtime.
Among those following scripts account for the most overhead:
native: .33 podman: 1.20 363.00% .87 (t7612-merge-verify-signatures.sh)
native: 8.75 podman: 9.66 110.00% .91 (t7613-merge-submodule.sh)
native: 11.56 podman: 12.78 110.00% 1.22 (t7112-reset-submodule.sh)
native: .43 podman: 2.72 632.00% 2.29 (t7030-verify-tag.sh)
native: .63 podman: 3.53 560.00% 2.90 (t7510-signed-commit.sh)
native: 3.34 podman: 6.42 192.00% 3.08 (t7004-tag.sh)
(In reply to Andreas Herrmann from comment #8) > FYI. > > Performance numbers split into categories for each 't[0-9]' prefix look as > follows: > > native: 22.15 podman: 23.13 104.00% .98 (gitsource-t8) > native: 54.09 podman: 55.47 102.00% 1.38 (gitsource-t9) > native: 37.37 podman: 39.66 106.00% 2.29 (gitsource-t2) > native: 60.36 podman: 63.00 104.00% 2.64 (gitsource-t0) > native: 55.75 podman: 58.69 105.00% 2.94 (gitsource-t1) > native: 57.64 podman: 63.08 109.00% 5.44 (gitsource-t4) > native: 55.79 podman: 61.45 110.00% 5.66 (gitsource-t6) > native: 118.50 podman: 128.73 108.00% 10.23 (gitsource-t5) > native: 162.32 podman: 173.35 106.00% 11.03 (gitsource-t3) > native: 152.55 podman: 172.05 112.00% 19.50 (gitsource-t7) I've tried to disable tests using GPG and the split performance numbers now look as follows: native: 22.50 podman: 23.50 104.00% 1.00 (gitsource-t8) native: 37.69 podman: 39.84 105.00% 2.15 (gitsource-t2) native: 54.17 podman: 56.35 104.00% 2.18 (gitsource-t9) native: 60.48 podman: 62.97 104.00% 2.49 (gitsource-t0) native: 57.18 podman: 60.50 105.00% 3.32 (gitsource-t4) native: 55.22 podman: 58.91 106.00% 3.69 (gitsource-t1) native: 52.77 podman: 56.66 107.00% 3.89 (gitsource-t6) native: 141.67 podman: 149.71 105.00% 8.04 (gitsource-t7) native: 162.98 podman: 172.62 105.00% 9.64 (gitsource-t3) native: 116.06 podman: 126.22 108.00% 10.16 (gitsource-t5) > So tests t7*.sh added about 19 seconds of additional runtime. > Among those following scripts account for the most overhead: > > native: .33 podman: 1.20 363.00% .87 > (t7612-merge-verify-signatures.sh) > native: 8.75 podman: 9.66 110.00% .91 > (t7613-merge-submodule.sh) > native: 11.56 podman: 12.78 110.00% 1.22 > (t7112-reset-submodule.sh) > native: .43 podman: 2.72 632.00% 2.29 > (t7030-verify-tag.sh) > native: .63 podman: 3.53 560.00% 2.90 > (t7510-signed-commit.sh) > native: 3.34 podman: 6.42 192.00% 3.08 (t7004-tag.sh) One component adding overhead is AppAmor. For containers there is a 'containers-default-0.51.1' profile. Starting the container with '--security-opt apparmor=unconfined' results in following split performance numbers: native: 54.38 podman: 55.31 101.00% .93 (noaa-gitsource-t9) native: 22.29 podman: 23.39 104.00% 1.10 (noaa-gitsource-t8) native: 37.70 podman: 38.96 103.00% 1.26 (noaa-gitsource-t2) native: 60.33 podman: 62.11 102.00% 1.78 (noaa-gitsource-t0) native: 53.14 podman: 55.17 103.00% 2.03 (noaa-gitsource-t6) native: 55.87 podman: 58.58 104.00% 2.71 (noaa-gitsource-t1) native: 57.06 podman: 60.55 106.00% 3.49 (noaa-gitsource-t4) native: 141.51 podman: 146.45 103.00% 4.94 (noaa-gitsource-t7) native: 116.58 podman: 123.50 105.00% 6.92 (noaa-gitsource-t5) native: 163.39 podman: 173.55 106.00% 10.16 (noaa-gitsource-t3) (In reply to Andreas Herrmann from comment #7) > I've started to narrow down which of the many scripts in git-source/t (make > test) are contributing disproportionately to the performance drop under > container environment. > > So far I've figured that tests using GPG show a significant drop. It seems that the base container image is FIPS compliant. In contrast to the host the container has installed: i+ | patterns-base-fips | FIPS 140-2 specific packages | package i | fips | FIPS 140-2 specific packages | pattern patterns-base-fips pulls in libgcrypt20-hmac. This library is not available on non-fips-complient host per default. gpg behaves differently depending on whether libgcrypt20-hmac is installed or not. Removing libgcrypt20-hmac from the container results in lower elapsed time for several git commands calling gpg commands. So, for an apples to apples comparison of git commands using gpg either libgcrypt20-hmac needs to be installed in the host or it needs to be removed from the container. I've adapted run-container.sh to add some options to selectively switch of confinements for apparmor and seccomp and to remove FIPS support from Leap container image.
I've retested with kernel 5.14.21-150400.24.66-default.
Results for gitsource benchmark are as follows:
native podman podman
sccmp-aa-fips aa-fips
Amean User 426.43 473.99 * -11.15%* 448.41 * -5.15%*
Amean System 159.76 203.06 * -27.10%* 191.22 * -19.69%*
Amean Elapsed 607.70 695.16 * -14.39%* 659.73 * -8.56%*
Amean CPU 96.00 97.00 * -1.04%* 96.00 ( 0.00%)
sccmp - seccomp confinement on
aa - apparmor confinement on
fips - FIPS packages installed in container image
Default kernel parameters were used. Thus 'Spec store bypass' mitigation was enabled for seccomp threads.
Removing one of seccomp, or apparmor confinement, or FIPS support gives following results:
native -------------------podman---------------------
aa-fips sccmp-fips sccmp-aa
Amean User 426.43 448.41 -5.15% 475.55 -11.52% 462.39 -8.43%
Amean Syst 159.76 191.22 -19.69% 196.29 -22.86% 205.96 -28.92%
Amean Elap 607.70 659.73 -8.56% 690.92 -13.69% 685.98 -12.88%
Amean CPU 96.00 96.00 0.00% 97.00 -1.04% 97.00 -1.04%
sccmp - seccomp confinement on
aa - apparmor confinement on
fips - FIPS packages installed in container image
Compare this to the 'podman sccmp-aa-fips' result above.
Large improvement when seccomp confinement is switch off (because then the penalty of the 'Spec store bypass' mitigation is avoided).
Minor improvement when either apparmor confinement or FIPS support is removed.
Following a comparison what each of seccomp confinement, apparmor confinement and FIPS support add to the runtime.
native -------------------podman---------------------
aa fips sccmp
Amean User 426.43 433.60 -1.68% 446.64 -4.74% 461.13 -8.14%
Amean Syst 159.76 188.84 -18.20% 178.48 -11.71% 194.38 -21.67%
Amean Elap 607.70 642.21 -5.68% 645.78 -6.27% 675.62 -11.18%
Amean CPU 96.00 96.00 0.00% 96.00 0.00% 96.67 -0.69%
sccmp - seccomp confinement on
aa - apparmor confinement on
fips - FIPS packages installed in container image
Most expensive is seccomp confinement (due to 'Spec store bypass' mitigation for seccomp threads).
Both apparmor confinement and FIPS support have a significant performance impact.
Disabling all three -- seccomp and apparmor confinement, and FIPS support -- gives following comparison:
native podman podman
aa-fips no_scaafps
Amean User 426.43 448.41 -5.15% 433.58 -1.68%
Amean System 159.76 191.22 -19.69% 179.52 -12.36%
Amean Elapsed 607.70 659.73 -8.56% 632.58 -4.10%
Amean CPU 96.00 96.00 0.00% 96.00 0.00%
no_scaafps - no seccomp/apparmor confinement, no FIPS
aa - apparmor confinement on
fips - FIPS packages installed in container image
Performance decrease is reduced to about 4 percent for elapsed time.
(In reply to Andreas Herrmann from comment #14) > Following a comparison what each of seccomp confinement, apparmor > confinement and FIPS support add to the runtime. Wording is a bit unfortunate. It shows the overhead of running the workload in a container with either of the three options enabled in comparison to a native run. Running the workload containerized alone (w/o any of the 3 options) has already a performance impact -- as shown in comment #15. > native -------------------podman--------------------- > aa fips sccmp > Amean User 426.43 433.60 -1.68% 446.64 -4.74% 461.13 -8.14% > Amean Syst 159.76 188.84 -18.20% 178.48 -11.71% 194.38 -21.67% > Amean Elap 607.70 642.21 -5.68% 645.78 -6.27% 675.62 -11.18% > Amean CPU 96.00 96.00 0.00% 96.00 0.00% 96.67 -0.69% > > sccmp - seccomp confinement on > aa - apparmor confinement on > fips - FIPS packages installed in container image > > Most expensive is seccomp confinement (due to 'Spec store bypass' mitigation > for seccomp threads). > > Both apparmor confinement and FIPS support have a significant performance > impact. Again wording is not optimal. A better illustration of the impact on performance of each of the 3 components is following comparison (using 'no_scaafps as baseline): ----------------------podman------------------------- no_scaafps aa fips sccmp Amean User 433.58 433.60 -0.00% 446.64 -3.01% 461.13 -6.35% Amean Syst 179.52 188.84 -5.19% 178.48 0.58% 194.38 -8.28% Amean Elap 632.58 642.21 -1.52% 645.78 -2.09% 675.62 -6.80% Amean CPU 96.00 96.00 0.00% 96.00 0.00% 96.67 -0.69% no_scaafps - no seccomp/apparmor confinement, no FIPS sccmp - seccomp confinement on aa - apparmor confinement on fips - FIPS packages installed in container image I've done additional measurements with kernel 5.14.21-150400.24.66-default after booting the system with additional kernel parameters 'spectre_v2_user=prctl spec_store_bypass_disable=prctl' which disables these security mitigations for seccomp threads. I'll add further comments with comparison results for this setup. (In reply to Andreas Herrmann from comment #12) <snip> > Results for gitsource benchmark are as follows: > > native podman podman > sccmp-aa-fips aa-fips > > Amean User 426.43 473.99 * -11.15%* 448.41 * -5.15%* > Amean System 159.76 203.06 * -27.10%* 191.22 * -19.69%* > Amean Elapsed 607.70 695.16 * -14.39%* 659.73 * -8.56%* > Amean CPU 96.00 97.00 * -1.04%* 96.00 ( 0.00%) ('spectre_v2_user=prctl spec_store_bypass_disable=prctl') native podman podman sccmp-aa-fips aa-fips Amean User 427.92 447.98 -4.69% 447.29 -4.53% Amean System 159.57 192.01 -20.33% 191.59 -20.07% Amean Elapsed 607.70 659.41 -8.51% 658.70 -8.39% Amean CPU 96.00 96.67 -0.69% 96.67 -0.69% > sccmp - seccomp confinement on > aa - apparmor confinement on > fips - FIPS packages installed in container image > > Default kernel parameters were used. Thus 'Spec store bypass' mitigation was > enabled for seccomp threads. (In reply to Andreas Herrmann from comment #13) > Removing one of seccomp, or apparmor confinement, or FIPS support gives > following results: > > native -------------------podman--------------------- > aa-fips sccmp-fips sccmp-aa > Amean User 426.43 448.41 -5.15% 475.55 -11.52% 462.39 -8.43% > Amean Syst 159.76 191.22 -19.69% 196.29 -22.86% 205.96 -28.92% > Amean Elap 607.70 659.73 -8.56% 690.92 -13.69% 685.98 -12.88% > Amean CPU 96.00 96.00 0.00% 97.00 -1.04% 97.00 -1.04% ('spectre_v2_user=prctl spec_store_bypass_disable=prctl') native -------------------podman--------------------- aa-fips sccmp-fips sccmp-aa Amean User 427.92 447.29 -4.53% 448.00 -4.69% 436.21 -1.94% Amean Syst 159.57 191.59 -20.07% 181.83 -13.95% 193.31 -21.14% Amean Elap 607.70 658.70 -8.39% 649.54 -6.88% 648.42 -6.70% Amean CPU 96.00 96.67 -0.69% 96.00 0.00% 97.00 -1.04% > sccmp - seccomp confinement on > aa - apparmor confinement on > fips - FIPS packages installed in container image <snip> (In reply to Andreas Herrmann from comment #14) > Following a comparison what each of seccomp confinement, apparmor > confinement and FIPS support add to the runtime. > > native -------------------podman--------------------- > aa fips sccmp > Amean User 426.43 433.60 -1.68% 446.64 -4.74% 461.13 -8.14% > Amean Syst 159.76 188.84 -18.20% 178.48 -11.71% 194.38 -21.67% > Amean Elap 607.70 642.21 -5.68% 645.78 -6.27% 675.62 -11.18% > Amean CPU 96.00 96.00 0.00% 96.00 0.00% 96.67 -0.69% ('spectre_v2_user=prctl spec_store_bypass_disable=prctl') native -------------------podman--------------------- aa fips sccmp Amean User 427.92 434.92 -1.64% 445.61 -4.13% 431.72 -0.89% Amean Syst 159.57 190.77 -19.55% 179.86 -12.71% 179.49 -12.48% Amean Elap 607.70 644.76 -6.10% 643.59 -5.91% 631.02 -3.84% Amean CPU 96.00 97.00 -1.04% 97.00 -1.04% 96.00 0.00% > sccmp - seccomp confinement on > aa - apparmor confinement on > fips - FIPS packages installed in container image <snip> (In reply to Andreas Herrmann from comment #15) > Disabling all three -- seccomp and apparmor confinement, and FIPS support -- > gives following comparison: > > native podman podman > aa-fips no_scaafps > Amean User 426.43 448.41 -5.15% 433.58 -1.68% > Amean System 159.76 191.22 -19.69% 179.52 -12.36% > Amean Elapsed 607.70 659.73 -8.56% 632.58 -4.10% > Amean CPU 96.00 96.00 0.00% 96.00 0.00% ('spectre_v2_user=prctl spec_store_bypass_disable=prctl') native podman podman aa-fips no_scaafps Amean User 427.92 447.29 -4.53% 431.90 -0.93% Amean System 159.57 191.59 -20.07% 179.08 -12.23% Amean Elapsed 607.70 658.70 -8.39% 629.67 -3.61% Amean CPU 96.00 96.67 -0.69% 96.67 -0.69% > no_scaafps - no seccomp/apparmor confinement, no FIPS > aa - apparmor confinement on > fips - FIPS packages installed in container image > > Performance decrease is reduced to about 4 percent for elapsed time. (In reply to Andreas Herrmann from comment #16) <snip> > A better illustration of the impact on performance of each of the 3 > components is following comparison (using 'no_scaafps as baseline): > > ----------------------podman------------------------- > no_scaafps aa fips sccmp > Amean User 433.58 433.60 -0.00% 446.64 -3.01% 461.13 -6.35% > Amean Syst 179.52 188.84 -5.19% 178.48 0.58% 194.38 -8.28% > Amean Elap 632.58 642.21 -1.52% 645.78 -2.09% 675.62 -6.80% > Amean CPU 96.00 96.00 0.00% 96.00 0.00% 96.67 -0.69% ('spectre_v2_user=prctl spec_store_bypass_disable=prctl') ----------------------podman------------------------- no_scaafps aa fips sccmp Amean User 431.90 434.92 -0.70% 445.61 -3.18% 431.72 0.04% Amean Syst 179.08 190.77 -6.53% 179.86 -0.43% 179.49 -0.23% Amean Elap 629.67 644.76 -2.40% 643.59 -2.21% 631.02 -0.21% Amean CPU 96.67 97.00 -0.34% 97.00 -0.34% 96.00 0.69% > no_scaafps - no seccomp/apparmor confinement, no FIPS > sccmp - seccomp confinement on > aa - apparmor confinement on > fips - FIPS packages installed in container image (In reply to Andreas Herrmann from comment #4) > The nature of gitsource benchmark is to create a lot of short living > processes. > It was suggested to use a simpler workload with similar characteristics to > try to reproduce and profile. > > I've used a shell_bensh.sh script as suggested at > https://lore.kernel.org/all/20230608111408.s2minsenlcjow7q3@quack3/ > but with 50000 iterations. > > For this workload the comparison is (with spec_store_bypass mitigation for > seccomp threads): > native podman > Amean User 17.26 ( 0.00%) 18.58 * -7.67%* > Amean System 4.42 ( 0.00%) 5.23 * -18.40%* > Amean Elapsed 21.79 ( 0.00%) 24.00 * -10.13%* > Amean CPU 99.00 ( 0.00%) 99.00 ( 0.00%) > > For this workload the comparison is (without spec_store_bypass mitigation > for seccomp threads): > native podman > Amean User 17.29 ( 0.00%) 17.55 * -1.50%* > Amean System 4.44 ( 0.00%) 4.84 * -9.08%* > Amean Elapsed 21.85 ( 0.00%) 22.58 * -3.34%* > Amean CPU 99.00 ( 0.00%) 99.00 ( 0.00%) > > Thus this workload does not show the same amount of overhead as the > gitsource benchmark for the container environment. When shell_bench.sh is run in a container that is unconfined wrt APPARMOR and SECCOMP the picture doesn't change. There is still a 3-4% longer runtime of the script. (I've revalidated this on the same test system (but with kernel 5.14.21-150400.24.69-default). (In reply to Andreas Herrmann from comment #23) > (In reply to Andreas Herrmann from comment #4) > > The nature of gitsource benchmark is to create a lot of short living > > processes. > > It was suggested to use a simpler workload with similar characteristics to > > try to reproduce and profile. > > > > I've used a shell_bensh.sh script as suggested at > > https://lore.kernel.org/all/20230608111408.s2minsenlcjow7q3@quack3/ > > but with 50000 iterations. <snip> Marginal note: This test is sensitive to the used environment (ie. number of env variables). I've learned this when comparing env of native tests versus env that is used in the container (which had fewer env variables in my case). Here is an example to demonstrate this (native run): marvin4:~/:[0]# env -i bash marvin4:/root/:[0]# sync ; echo 3 > /proc/sys/vm/drop_caches marvin4:/root/:[0]# /usr/bin/time bash shell_bensh.sh 16.83user 4.78system 0:20.90elapsed 103%CPU (0avgtext+0avgdata 3260maxresident)k 1176inputs+0outputs (6major+5033569minor)pagefaults 0swaps marvin4:/root/:[0]# for i in $(seq 1 1000); do export MYTEST_ENV_$i=foo; done marvin4:/root/:[0]# /usr/bin/time bash shell_bensh.sh 22.31user 6.01system 0:27.37elapsed 103%CPU (0avgtext+0avgdata 3744maxresident)k 0inputs+0outputs (0major+5481414minor)pagefaults 0swaps marvin4:/root/:[0]# for i in $(seq 1 1000); do unset MYTEST_ENV_$i; done marvin4:/root/:[0]# /usr/bin/time bash shell_bensh.sh 17.23user 4.47system 0:20.92elapsed 103%CPU (0avgtext+0avgdata 3228maxresident)k 0inputs+0outputs (0major+5083382minor)pagefaults 0swaps I am closing this bug report. Half of the performance regression comes from AppArmor confinement and FIPS support of the container. When those factors are removed there is a performance regression of 3-4 percent which is comparable to the overhead measured with the simple shell_bensh.sh script as mentioned in comment #4. I seems that this is the normal overhead of running such a workload in container environment. |
With MMTests/gitsource benchmark (bumped to version v2.16.6) there is a performance drop between native run and container run (with podman): native podman Amean User 429.09 ( 0.00%) 446.15 * -3.98%* Amean System 161.25 ( 0.00%) 188.93 * -17.17%* Amean Elapsed 610.56 ( 0.00%) 658.94 * -7.92%* Amean CPU 96.00 ( 0.00%) 96.00 ( 0.00%) The podman numbers are for a kernel boot with kernel parameters 'spectre_v2_user=prctl spec_store_bypass_disable=prctl' to not enable these security mitigations for seccomp threads. (If enabled, performance decreases even more.) Would be nice to understand where the additional time is spent in the container run.