Bug 1212824

Summary: [Build 20230627] apptainer: Write failed because no space left on device
Product: [openSUSE] openSUSE Tumbleweed Reporter: Dominique Leuenberger <dimstar>
Component: OtherAssignee: Yiannis Bonatakis <ioannis.bonatakis>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: dleuenberger
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
URL: https://openqa.opensuse.org/tests/3390518/modules/apptainer/steps/42
Whiteboard:
Found By: openQA Services Priority:
Business Priority: Blocker: Yes
Marketing QA Status: --- IT Deployment: ---

Description Dominique Leuenberger 2023-06-28 19:08:39 UTC 
## Observation

Since today, the hpc_apptainer test is failing with 'no space on device'

df shows sufficient space left:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           728M     0  728M   0% /dev/shm
tmpfs           292M  4.9M  287M   2% /run
/dev/vda2        28G  2.3G   26G   9% /
tmpfs           728M   24K  728M   1% /tmp
/dev/vda2        28G  2.3G   26G   9% /.snapshots
/dev/vda2        28G  2.3G   26G   9% /home
/dev/vda2        28G  2.3G   26G   9% /var
/dev/vda2        28G  2.3G   26G   9% /boot/grub2/i386-pc
/dev/vda2        28G  2.3G   26G   9% /boot/grub2/x86_64-efi
/dev/vda2        28G  2.3G   26G   9% /usr/local
/dev/vda2        28G  2.3G   26G   9% /opt
/dev/vda2        28G  2.3G   26G   9% /root
/dev/vda2        28G  2.3G   26G   9% /srv
tmpfs           146M     0  146M   0% /run/user/0

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-containers_hpc_apptainer@64bit fails in
[apptainer](https://openqa.opensuse.org/tests/3390518/modules/apptainer/steps/42)

## Test suite description
Maintainer: dheidler. Extra tests about CLI software in container module


## Reproducible

Fails since (at least) Build [20230627](https://openqa.opensuse.org/tests/3389026)


## Expected result

Last good: [20230626](https://openqa.opensuse.org/tests/3383456) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=containers_hpc_apptainer&version=Tumbleweed)
Comment 1 Egbert Eich 2023-06-30 19:35:30 UTC 
Apparently, the command `apptainer build container.sif apptainer_container.def` died while it tried to create the squashfs. It is conceivable that apptainer temporarily needs some scratch space while creating the sif file. If space on the test system is too scares this needs to be remedied by QA. It is unrelated to the software.
Assigning to Ioannis.
Comment 2 Dominique Leuenberger 2023-06-30 20:30:19 UTC 
(In reply to Egbert Eich from comment #1)
> Apparently, the command `apptainer build container.sif
> apptainer_container.def` died while it tried to create the squashfs. It is
> conceivable that apptainer temporarily needs some scratch space while
> creating the sif file. If space on the test system is too scares this needs
> to be remedied by QA. It is unrelated to the software.
> Assigning to Ioannis.

AS can be seen in the output of df in comment #0 we are on a 30GB disk with 2.3GB occupied. If we need more free space than that, everybody will be doomed (only chance would be for apptainer to use /tmp which is tmpfs and < 1GB free... but that should not have changed recently)
Comment 3 Dominique Leuenberger 2023-08-10 09:00:55 UTC 
Seems we really run out of diskspace there in /tmp

/tmp being tmpfs I tried a test run with more RAM associated to the VM - which in turn passes.

Will adjust the test to offer it that needed memory
(not clear why this started happening the day it did - but well)
Comment 4 Dominique Leuenberger 2023-08-10 09:03:44 UTC 
Adjusted test suite with QEMURAM=2048