Bug 1216158

Summary:	[Build 20231011-1] oscap xccdf eval command stuck in test oscap_xccdf_eval_remote
Product:	[openSUSE] PUBLIC SUSE Linux Enterprise Server 15 SP5	Reporter:	Joaquín Rivera <jeriveramoya>
Component:	Security Certifications	Assignee:	Certification Bugs <certification-bugs>
Status:	RESOLVED FIXED	QA Contact:
Severity:	Normal
Priority:	P5 - None	CC:	daniel.mach, felice.maccaro, marcela.maslanova, rbranco, rumen.chikov
Version:	unspecified
Target Milestone:	---
Hardware:	S/390
OS:	Other
URL:	https://openqa.suse.de/tests/12463240/modules/oscap_xccdf_eval_remote/steps/42
Whiteboard:	SCAP
Found By:	openQA	Services Priority:
Business Priority:		Blocker:	Yes
Marketing QA Status:	---	IT Deployment:	---

Description Joaquín Rivera 2023-10-12 06:32:06 UTC

## Observation

openQA test in scenario sle-15-SP5-Server-DVD-Updates-s390x-stig@s390x-kvm fails in
[oscap_xccdf_eval_remote](https://openqa.suse.de/tests/12463240/modules/oscap_xccdf_eval_remote/steps/42)

The command got stuck for 50 minutes (which is the script timeout) in this scenario running in maintenance updates.

See logs attached to openQA job in Logs and Assets tab.
Script code: https://github.com/search?q=repo%3Aos-autoinst%2Fos-autoinst-distri-opensuse%20oscap_evaluate_remote&type=code


## Expected result
Last good: [20231010-1](https://openqa.suse.de/tests/12450777) (or more recent)

Comment 1 Marcus Meissner 2023-10-12 07:13:24 UTC

It has probably run out of memory again, looking at the follow screen shots it seems in an OOM situation.

--fetch-remote-resources will really use multiple gigabytes of memory

Comment 2 Ricardo Branco 2023-10-12 07:36:26 UTC

Fwiw, in public cloud tests we download the compressed file, uncompress it and then run it with --local-files like this:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/publiccloud/hardened.pm

Comment 3 Marcus Meissner 2023-10-12 08:04:38 UTC

Did you see if it reduced memory usage of openscap?

Comment 4 Ricardo Branco 2023-10-12 08:16:00 UTC

(In reply to Marcus Meissner from comment #3)
> Did you see if it reduced memory usage of openscap?

I didn't analyze it but at least it worked.

It seems to me that oscap was trying to download the whole file which is 350MB because it was taking too much time, or perhaps downloaded the compressed file and tried to uncompress in memory itself.

Comment 5 Joaquín Rivera 2023-10-12 09:27:43 UTC

It succeed with less ram, it was quite overdimensioned, and no idea how guest shared memory in that situation, I set 8GiB to be conservative and the timeout needs to be bumped because in rare ocassion we made in 16' but normally was in the boundary of the timeout of 50'. https://openqa.suse.de/tests/12464688#step/oscap_xccdf_eval_remote/42
So no bug :) thanks for the feedback. we will keep in mind this local option for future development.

Comment 6 Ricardo Branco 2023-10-17 11:00:18 UTC

I could make it run in cloud with instances with 8G of RAM adding 4G of swap.

Comment 7 Joaquín Rivera 2023-10-17 12:46:29 UTC

I noticed that the installer when does it for you to apply the policies also download it first to apply stig, but then we wouldn't test that option if we don't use the fetch remote option.

Comment 8 Rumen Chikov 2023-11-13 07:53:48 UTC

Hello Joaquín 

Because in your message we have:
++
So no bug :) thanks for the feedback. we will keep in mind this local option for future development.
++

may we consider that this bug is closed and to change it status, otherwise we continue to have this bug in our list.

Thank you in advance for your feedback.
Have a nice day
Rumen

Comment 9 Marcela Maslanova 2024-05-06 12:30:56 UTC

I'm closing the bug based on the last comment.

Comment 10 Ricardo Branco 2024-05-06 12:35:40 UTC

Fwiw, the workaround with swap didn't work sometimes even with 16G RAM.  We had to increase our VM's to 32G.