Bug 1220746 - BCC tools fail with LLVM ERROR
Summary: BCC tools fail with LLVM ERROR
Status: VERIFIED FIXED
Alias: None
Product: PUBLIC SUSE Linux Enterprise Server 15 SP6
Classification: openSUSE
Component: Basesystem (show other bugs)
Version: unspecified
Hardware: Other Other
: P2 - High : Normal
Target Milestone: ---
Assignee: Shung-Hsi Yu
QA Contact:
URL: https://openqa.suse.de/tests/13647181...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-03-01 10:00 UTC by Andrea Cervesato
Modified: 2024-04-02 12:00 UTC (History)
7 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrea Cervesato 2024-03-01 10:00:46 UTC
## Observation

openQA test in scenario sle-15-SP6-Online-x86_64-bpftools@64bit fails in
[bcc](https://openqa.suse.de/tests/13647181/modules/bcc/steps/8).


## Test suite description
Compile and attach eBPF probes with BCC tools.


## Reproducible

Fails since (at least) Build [57.1](https://openqa.suse.de/tests/13548778)


## Expected result

Last good: [54.1](https://openqa.suse.de/tests/13493650) (or more recent)


## Further details

The tested tools are the following:
- btrfsdist
- btrfsslower
- filetop

All tools executions fail with "LLVM ERROR:" and return code 1, without providing any specific message. This suggests to take a look into LLVM version that has been used to build bcc-tools.

This happens in all tested architectures: aarch64, ppc64le, x86_64 on SLE15-SP6.
Comment 1 Petr Cervinka 2024-03-01 10:17:31 UTC
susetest:~ # /usr/share/bcc/tools/btrfsdist  5 2
LLVM ERROR:
susetest:~ # echo $?
1
susetest:~ # /usr/share/bcc/tools/btrfsslower  -d 10
LLVM ERROR:
susetest:~ # echo $?
1
susetest:~ # /usr/share/bcc/tools/filetop -a 5 10
LLVM ERROR:
susetest:~ # echo $?
1
Comment 2 Radoslav Tzvetkov 2024-03-01 14:25:26 UTC
Perhaps it is solved with https://build.suse.de/request/show/320592/changes ?
Comment 3 Petr Cervinka 2024-03-04 06:56:18 UTC
(In reply to Radoslav Tzvetkov from comment #2)
> Perhaps it is solved with https://build.suse.de/request/show/320592/changes ?


Thanks for pointing on that SR. It can't be solved by that as this change is responsible error. When digging in the history last working bcc version was 0.28 and first failing is 0.29.

Mentioned SR upgraded bcc to 0.29.
Comment 4 Shung-Hsi Yu 2024-03-11 11:10:37 UTC
Thanks for implementing the bcc test!

Looks like bcc ended up building against the (too old) LLVM 7 since SR320592

> $ zypper install bcc-tools
> ...
> The following 5 NEW packages are going to be installed:
>   bcc-tools libbcc0 libclang7 libLLVM7 python3-bcc

Likely due to my choice of using %product_libs_llvm_ver[1] that is present in openSUSE but not in SLES, so we ended up building against the default LLVM which was LLVM 7

> # Use the latest default LLVM version available, unless it is not yet
> # supported by bcc.
> %if 0%{?product_libs_llvm_ver} > 17
> %define llvm_major_version 17
> %else
> %define llvm_major_version %{nil}
>  %define llvm_major_version %{nil}
> %endif

I'll update the LLVM verion-chosing logic so it works on SLES

1: https://build.opensuse.org/request/show/1134405
Comment 6 Shung-Hsi Yu 2024-03-12 09:17:50 UTC
I ended up doing the following for a version-choosing logic that works both on OBS and IBS. For openSUSE just use the default unless it is too new to work (by looking at the %product_libs_llvm_ver macro that exists on OBS), for SLE15+ the default LLVM version is always 7 and doesn't work, so I just hard code the newest LLVM that's available.

> # Use default LLVM on openSUSE unless it is not yet supported
> %if 0%{?suse_version} >= 1600 || 0%{?is_opensuse}
>  %if 0%{?product_libs_llvm_ver} > 17
>  %define llvm_major_version 17
>  %else
>  %define llvm_major_version %{nil}
>  %endif
> %else
>  # Hard-code latest LLVM for SLES, the default version is too old
>  %if 0%{?sle_version} == 150600
>   %define llvm_major_version 17
>  %else
>  %if 0%{?sle_version} == 150500
>   %define llvm_major_version 15
>  %else
>  %if 0%{?sle_version} == 150400
>   %define llvm_major_version 11
>  %endif
>  %endif
>  %endif
> %endif

I've check that with the above change bcc indeed build against LLVM 17, so with with SR323752 problem should be fixed. But I'll rather wait for confirmation rather then closing this right away.
Comment 7 Swayammitra Tripathy 2024-03-20 14:55:24 UTC
Hello Shung ,

Could you please update the bug status to fix if the issue is already fixed.
Comment 8 Shung-Hsi Yu 2024-03-20 14:57:37 UTC
Fixed as per comment 6
Comment 9 Petr Cervinka 2024-03-25 15:13:13 UTC
https://build.suse.de/request/show/323752 is is still in review and fix didn't get to 15-SP6 yet.
Comment 10 Shung-Hsi Yu 2024-03-26 08:07:36 UTC
Thanks(In reply to Petr Cervinka from comment #9)
> https://build.suse.de/request/show/323752 is is still in review and fix
> didn't get to 15-SP6 yet.

Re-opening this issue.

@Eugenio do you think SR323752 could be accepted? As far as I can tell bcc seems to build fine in SUSE:SLE-15-SP6:GA:Staging:S.
Comment 11 Eugenio Paolantonio 2024-03-26 08:52:25 UTC
Hi Shung-Hsi,

apologies, I thought I pinged you but I didn't: bpftrace needs a matching update as it now becomes unresolvable due to


conflict for providers of llvm17-devel needed by bcc-devel, (provider llvm17-devel is in conflict with llvm15-devel)



could you please submit bpftrace as well? Thanks in advance (and sorry for the delay!)
Comment 12 Shung-Hsi Yu 2024-03-26 09:13:59 UTC
(In reply to Eugenio Paolantonio from comment #11)
> apologies, I thought I pinged you but I didn't: bpftrace needs a matching
> update as it now becomes unresolvable due to
> 
> conflict for providers of llvm17-devel needed by bcc-devel, (provider
> llvm17-devel is in conflict with llvm15-devel)
> 
> could you please submit bpftrace as well? Thanks in advance (and sorry for
> the delay!)

Understood and no worries, will work on it right now.
Comment 13 Shung-Hsi Yu 2024-03-26 14:10:36 UTC
Submitted bcc (with an additional fix in spec file) with SR#324892 and bpftrace with SR#324893
Comment 15 Radoslav Tzvetkov 2024-03-28 15:47:24 UTC
Staged will come after the RC1
Comment 16 Radoslav Tzvetkov 2024-04-02 08:40:31 UTC
A submit request mentioning this bug  was successfully integrated into build 77.1.

Please set it as resolved and eventually verify.
Comment 17 Radoslav Tzvetkov 2024-04-02 10:09:49 UTC
Please indicate the current expectation for fixing the bug using the Target Milestone field. What do we believe is possible? 

In the cases where we do not plan to deliver or cannot guess (!?) please do not enter anything, but you can comment if you wish to provide more details.
Comment 18 Petr Cervinka 2024-04-02 12:00:25 UTC
(In reply to Radoslav Tzvetkov from comment #16)
> A submit request mentioning this bug  was successfully integrated into build
> 77.1.
> 
> Please set it as resolved and eventually verify.

Build 77.1 is finally fine: https://openqa.suse.de/tests/13914887
Comment 19 Petr Cervinka 2024-04-02 12:00:56 UTC
Setting to verified per previous comment.