Bug 1212824 - [Build 20230627] apptainer: Write failed because no space left on device
Summary: [Build 20230627] apptainer: Write failed because no space left on device
Status: RESOLVED FIXED
Alias: None
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Other (show other bugs)
Version: Current
Hardware: Other Other
: P5 - None : Normal (vote)
Target Milestone: ---
Assignee: Yiannis Bonatakis
QA Contact: E-mail List
URL: https://openqa.opensuse.org/tests/339...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-06-28 19:08 UTC by Dominique Leuenberger
Modified: 2023-08-10 09:03 UTC (History)
1 user (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Leuenberger 2023-06-28 19:08:39 UTC
## Observation

Since today, the hpc_apptainer test is failing with 'no space on device'

df shows sufficient space left:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           728M     0  728M   0% /dev/shm
tmpfs           292M  4.9M  287M   2% /run
/dev/vda2        28G  2.3G   26G   9% /
tmpfs           728M   24K  728M   1% /tmp
/dev/vda2        28G  2.3G   26G   9% /.snapshots
/dev/vda2        28G  2.3G   26G   9% /home
/dev/vda2        28G  2.3G   26G   9% /var
/dev/vda2        28G  2.3G   26G   9% /boot/grub2/i386-pc
/dev/vda2        28G  2.3G   26G   9% /boot/grub2/x86_64-efi
/dev/vda2        28G  2.3G   26G   9% /usr/local
/dev/vda2        28G  2.3G   26G   9% /opt
/dev/vda2        28G  2.3G   26G   9% /root
/dev/vda2        28G  2.3G   26G   9% /srv
tmpfs           146M     0  146M   0% /run/user/0

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-containers_hpc_apptainer@64bit fails in
[apptainer](https://openqa.opensuse.org/tests/3390518/modules/apptainer/steps/42)

## Test suite description
Maintainer: dheidler. Extra tests about CLI software in container module


## Reproducible

Fails since (at least) Build [20230627](https://openqa.opensuse.org/tests/3389026)


## Expected result

Last good: [20230626](https://openqa.opensuse.org/tests/3383456) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=containers_hpc_apptainer&version=Tumbleweed)
Comment 1 Egbert Eich 2023-06-30 19:35:30 UTC
Apparently, the command `apptainer build container.sif apptainer_container.def` died while it tried to create the squashfs. It is conceivable that apptainer temporarily needs some scratch space while creating the sif file. If space on the test system is too scares this needs to be remedied by QA. It is unrelated to the software.
Assigning to Ioannis.
Comment 2 Dominique Leuenberger 2023-06-30 20:30:19 UTC
(In reply to Egbert Eich from comment #1)
> Apparently, the command `apptainer build container.sif
> apptainer_container.def` died while it tried to create the squashfs. It is
> conceivable that apptainer temporarily needs some scratch space while
> creating the sif file. If space on the test system is too scares this needs
> to be remedied by QA. It is unrelated to the software.
> Assigning to Ioannis.

AS can be seen in the output of df in comment #0 we are on a 30GB disk with 2.3GB occupied. If we need more free space than that, everybody will be doomed (only chance would be for apptainer to use /tmp which is tmpfs and < 1GB free... but that should not have changed recently)
Comment 3 Dominique Leuenberger 2023-08-10 09:00:55 UTC
Seems we really run out of diskspace there in /tmp

/tmp being tmpfs I tried a test run with more RAM associated to the VM - which in turn passes.

Will adjust the test to offer it that needed memory
(not clear why this started happening the day it did - but well)
Comment 4 Dominique Leuenberger 2023-08-10 09:03:44 UTC
Adjusted test suite with QEMURAM=2048