Bug 1224130

Summary: [Build 2.2_spack-image] openQA test fails in bci_test_docker
Product: [openSUSE] PUBLIC SUSE Linux Enterprise Server 15 SP6 Reporter: Maurizio Dati <maurizio.dati>
Component: ContainersAssignee: Containers Team <containers-bugowner>
Status: RESOLVED FIXED QA Contact:
Severity: Major    
Priority: P5 - None CC: eich, felix.niederwanger
Version: unspecified   
Target Milestone: ---   
Hardware: Other   
OS: Other   
URL: https://openqa.suse.de/tests/14234953/modules/bci_test_docker/steps/31
Whiteboard:
Found By: openQA Services Priority:
Business Priority: Blocker: Yes
Marketing QA Status: --- IT Deployment: ---

Description Maurizio Dati 2024-05-10 14:30:37 UTC
## Observation

openQA test in scenario sle-15-SP6-BCI-Updates-ppc64le-spack_on_SLES_12-SP5@ppc64le fails in
[bci_test_docker](https://openqa.suse.de/tests/14234953/modules/bci_test_docker/steps/31)

## Test suite description
The base test suite is used for job templates defined in YAML documents. It has no settings of its own.


## Reproducible

Fails since (at least) Build [2.2_spack-image](https://openqa.suse.de/tests/14234925)


## Expected result

Last good: (unknown) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=ppc64le&distri=sle&flavor=BCI-Updates&machine=ppc64le&test=spack_on_SLES_12-SP5&version=15-SP6)

See failed tests in build "spack":

https://openqa.suse.de/tests/overview?result=failed&arch=&flavor=&machine=&test=&modules=&module_re=&group_glob=¬_group_glob=&comment=&distri=sle&version=15-SP6&build=2.2_spack-image&groupid=538#

---
Comment 1 Maurizio Dati 2024-05-10 14:32:40 UTC
- More details:

See these error messages in log of failed job i.e. https://openqa.suse.de/tests/14234953/logfile?filename=serial_terminal.txt:

- tests/test_all.py

"...
[gw0] [ 92%] FAILED tests/test_all.py::test_no_orphaned_packages[bci/spack:0.21 from registry.suse.de/suse/sle-15-sp6/update/cr/totest/images/bci/spack:0.21] 
tests/test_all.py::test_certificates_are_present[local-bci/spack:0.21 from registry.suse.de/suse/sle-15-sp6/update/cr/totest/images/bci/spack:0.21] 
...
>       assert not orphaned_packages.difference(known_orphaned_packages)
E       AssertionError: assert not {'clingo', 'libclingo4', 'patchelf', 'python3-clingo', 'spack', 'spack-recipes'}
E        +  where {'clingo', 'libclingo4', 'patchelf', 'python3-clingo', 'spack', 'spack-recipes'} = <built-in method difference of set object at 0x7fffb418bf28>({'ALP-dummy-release', 'kubic-locale-archive', 'skelcd-EULA-bci', 'sle-module-basesystem-release', 'sle-module-python3-release', 'sle-module-server-applications-release', ...})
E        +    where <built-in method difference of set object at 0x7fffb418bf28> = {'clingo', 'kubic-locale-archive', 'libclingo4', 'patchelf', 'python3-clingo', 'skelcd-EULA-bci', ...}.difference

tests/test_all.py:355: AssertionError
..."

- Then other errors also in other test module test_spack:

https://openqa.suse.de/tests/14234953/#step/bci_test_docker/59

"...
_ ERROR at setup of test_spack[local-bci/spack:0.21 from registry.suse.de/suse/sle-15-sp6/update/cr/totest/images/bci/spack:0.21] _
[gw0] linux -- Python 3.6.15 /root/BCI-tests/.tox/spack/bin/python

...
/usr/lib64/python3.6/subprocess.py:438: CalledProcessError
...

The command 'docker-shell cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y' returned a non-zero code: 3
...

docker: Error response from daemon: Duplicate mount point: /root/spack.yaml.
..."

---
Comment 2 Dirk Mueller 2024-05-10 17:10:24 UTC
these are two different issues, would have been useful to file them in two different bugs. anyhow, about a)

AssertionError: assert not {'clingo', 'libclingo4', 'patchelf', 'python3-clingo', 'spack', 'spack-recipes'}
E        +  where {'clingo', 'libclingo4', 'patchelf', 'python3-clingo', 'spack', 'spack-recipes'} = <built-in method difference of set object at 0x7fffb418bf28>({'ALP-dummy-release', 'kubic-locale-archive', 'skelcd-EULA-bci', 'sle-module-basesystem-release', 'sle-module-python3-release', 'sle-module-server-applications-release', ...})

this seems to be an outdated bci repo, I can see them in the repo: 
# zypper search -s clingo
Refreshing service 'container-suseconnect-zypp'.
Retrieving repository 'SLE_BCI' metadata ......................................................................................[done]
Building repository 'SLE_BCI' cache ...........................................................................................[done]
Loading repository data...
Reading installed packages...

S | Name           | Type    | Version            | Arch   | Repository
--+----------------+---------+--------------------+--------+-----------
  | clingo         | package | 5.5.0-150400.4.3.3 | x86_64 | SLE_BCI
  | libclingo4     | package | 5.5.0-150400.4.3.3 | x86_64 | SLE_BCI
  | python3-clingo | package | 5.5.0-150400.4.3.3 | x86_64 | SLE_BCI
Comment 3 Dirk Mueller 2024-05-10 17:11:55 UTC
ah, you're running on ppc64le. it's not supported there!.
Comment 7 Maurizio Dati 2024-05-13 08:56:27 UTC
(In reply to Dirk Mueller from comment #2)
> these are two different issues, would have been useful to file them in two
> different bugs. anyhow, about a)
> ...
> ...

Thank you, I will split that "issue b)" in another bug, if you prefer.
Comment 8 Maurizio Dati 2024-05-13 09:29:23 UTC
(In reply to Maurizio Dati from comment #7)
> (In reply to Dirk Mueller from comment #2)
> > these are two different issues, would have been useful to file them in two
> > different bugs. anyhow, about a)
> > ...
> > ...
> 
> Thank you, I will split that "issue b)" in another bug, if you prefer.

For more information:
impacted test in that build were mainly running on "ppc64le" and "s390x".

But in a last build, issue (b) appeared in a "x86_64" test "Build4.3_spack-image-spack_on_SLES_12-SP5":
https://openqa.suse.de/tests/14278399/#step/bci_test_docker/59
Comment 9 Egbert Eich 2024-05-13 17:20:17 UTC
Please forget any test results on ppc64le or s390x, these are irrelevant. These containers will no longer be built.
Also, we don't need any of this for SP5 as these containers will never be shipped. This feature will only be supported with SP5 onward.
Comment 10 Maurizio Dati 2024-05-14 07:55:48 UTC
Having been clarified that the "spack" images don't consider 'ppc64le' and 's390x' architectures and no related tests shall done, we can close this ticket.
Thank you all.
Comment 11 Felix Niederwanger 2024-05-16 07:51:59 UTC
To separate the spack issue from the underlying docker issue on 12-SP5 I created https://bugzilla.suse.com/show_bug.cgi?id=1224345.