Bug 1175317 - [Build 20200815] openQA test fails in installation
[Build 20200815] openQA test fails in installation
Status: IN_PROGRESS
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: AutoYaST
Current
Other Other
: P2 - High : Normal (vote)
: ---
Assigned To: Ladislav Slezák
E-mail List
https://openqa.opensuse.org/tests/136...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-08-16 17:59 UTC by Dominique Leuenberger
Modified: 2022-09-12 10:39 UTC (History)
2 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Leuenberger 2020-08-16 17:59:03 UTC
## Observation

openQA test in scenario opensuse-Tumbleweed-NET-x86_64-autoyast_gnome@64bit fails in
[installation](https://openqa.opensuse.org/tests/1363405/modules/installation/steps/3)

Autotrader is stalling at configuring software step

## Test suite description
Testing autoyast installation with given profile and expect SLES with gnome. Please, see the profile for more details. Profile is available in distri git repo under data subdirectory. Same as autoyast_gnome but with product defined in the profile. Maintainers: riafarov, okurz


## Reproducible

Fails since (at least) Build [20200814](https://openqa.opensuse.org/tests/1362475)


## Expected result

Last good: [20200813](https://openqa.opensuse.org/tests/1361271) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=NET&machine=64bit&test=autoyast_gnome&version=Tumbleweed)
Comment 2 Ladislav Slezák 2020-08-18 16:44:08 UTC
Actually it's not stuck, it's iterating over a huge amount of packages.

It's related to the this change: https://github.com/yast/yast-autoinstallation/pull/665/files#diff-b1c37c173b1abb406de4f62a9888c301R46

I'll try adding some speed optimization...
Comment 3 Ladislav Slezák 2020-08-19 14:11:30 UTC
Unfortunately it turned out small optimizations do not help much.

The problem is that the OSS repository contains more than 60.000 (!) packages. Iterating over dependencies of all packages will take lot of time, speed optimizations can help only a bit.

In pkg-bindings we have Pkg.PkgQueryProvides() function (https://github.com/yast/yast-pkg-bindings/blob/3ed52e693ca6a79fd7ee6db41e024137268d517c/src/Package.cc#L115) which handles the "Provides" RPM dependency in a very efficient way.

I thought we could create a similar function (PkgQuerySupplements) which would handle the "Supplements" dependency used in by AutoYaST. Unfortunately libzypp does not support that, it only supports WhatProvides (https://github.com/openSUSE/libzypp/blob/master/zypp/sat/WhatProvides.h) and WhatObsoletes (https://github.com/openSUSE/libzypp/blob/master/zypp/sat/WhatObsoletes.h).



So my proposal is:

1) Switch from "Supplements: autoyast(foo)" to Provides:

    Provides: autoyast-tag()
    Provides: autoyast-tag(foo)

   ("autoyast-tag" is just a proposal, feel free to suggest anything better ;-))


2) In the code:

  a) first get packages providing "autoyast-tag()"
     using Pkg.PkgQueryProvides("autoyast-tag()"),
     that should return few packages (like a dozen or so)

  b) then for that few packages get all dependencies
     and find out that "autoyast-tag(.*)" details

  That should be both fast and memory efficient.
Comment 4 Ladislav Slezák 2020-08-19 15:03:10 UTC
Question to the libzypp team:

Is there a way how to efficiently query for packages providing a specific "Supplements"? Or do we need to switch to "Provides" as proposed in the previous comment?
Comment 5 Stefan Hundhammer 2020-08-19 15:13:15 UTC
I suggested on IRC that you libzypp maintainers might know of a completely different solution to this problem. While we tried a brute force approach (which does not scale anymore due to the sheer number of involved packages), maybe there is a different algorithm?

If you decide there is none, we can always fall back to Ladislav's suggested approach. But please check with your expert knowledge if that actually does what we need.
Comment 6 Michael Andres 2020-08-19 16:00:16 UTC
(In reply to Ladislav Slezák from comment #3)
> I thought we could create a similar function (PkgQuerySupplements) which
> would handle the "Supplements" dependency used in by AutoYaST. Unfortunately
> libzypp does not support that, it only supports WhatProvides

WhatProvides is an index built and maintained even inside the resolver. No one in the resolver is interested in supplements. But I guess you don't need an index. 

It's probably sufficient if a query is done inside the pkg-bindings rather than moving the whole haystack into ruby. Even searching for all supplements in 65995 packages with zypper is IMO reasonable fast (incl. refresh checks and screen output):

  ma@fibonacci:~ (0) $ zypper se -s | wc -l
  65995
  ma@fibonacci:~ (0) $ time zypper search -s -v --supplements /./
  ...
  real    0m5.519s
  user    0m3.797s
  sys     0m0.065s

In pkg-bindings you can do a PoolQuery 
   * PoolQuery q;
   * q.setMatchGlob();
   * q.addKind(ResKind::package);
   * q.addAttribute(sat::SolvAttr::supplements, "autoyast(*)");

The PoolQueryIterator visits all sat::Solavables that do contain matches and offers access to the Matches found within each solvable.

Example is in the PoolQuery / PoolQueryIterator class documentation. If you need help with the code, just ask.
Comment 7 Michael Andres 2020-08-19 17:04:59 UTC
// g++ -std=c++17 -O3 -Wall test.cc -lzypp  -o test 
#include <iostream>
#include <zypp/misc/DefaultLoadSystem.h>
#include <zypp/PoolQuery.h>

using std::cout;
using std::endl;
using namespace zypp;

int main( int argc, const char * argv[] )
{
  misc::defaultLoadSystem( "/", misc::LS_READONLY|misc::LS_NOREFRESH );

  //--- HERE.....
  PoolQuery q;
  q.setMatchGlob();
  q.addKind(ResKind::package);
  q.addAttribute(sat::SolvAttr::supplements, "autoyast(*)");

  for ( PoolQuery::const_iterator it = q.begin(), end = q.end(); it != end; ++it ) {
    sat::Solvable solv { *it };
    cout << "natch in Solvable " << solv << endl;

    for( PoolQuery::const_iterator::matches_iterator mit = it.matchesBegin(), end = it.matchesEnd(); mit != end; ++mit ) {
      sat::LookupAttr::iterator attr { *mit };
      cout << "  `" << attr.asString() << "´" << endl;
    }
  }
  //--- HERE.....
  return 0;
}
Comment 8 Ladislav Slezák 2020-08-19 17:11:33 UTC
Great! Thank you for the examples!
Comment 10 Ladislav Slezák 2020-08-21 14:57:17 UTC
Proposal for fix: https://github.com/yast/yast-pkg-bindings/pull/136