Bug 1175317

Summary: [Build 20200815] openQA test fails in installation
Product: [openSUSE] openSUSE Tumbleweed Reporter: Dominique Leuenberger <dimstar>
Component: AutoYaSTAssignee: Ladislav Slezák <lslezak>
Status: IN_PROGRESS --- QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P2 - High CC: dgonzalez, zypp-maintainers
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
URL: https://openqa.opensuse.org/tests/1363405/modules/installation/steps/3
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Dominique Leuenberger 2020-08-16 17:59:03 UTC
## Observation

openQA test in scenario opensuse-Tumbleweed-NET-x86_64-autoyast_gnome@64bit fails in
[installation](https://openqa.opensuse.org/tests/1363405/modules/installation/steps/3)

Autotrader is stalling at configuring software step

## Test suite description
Testing autoyast installation with given profile and expect SLES with gnome. Please, see the profile for more details. Profile is available in distri git repo under data subdirectory. Same as autoyast_gnome but with product defined in the profile. Maintainers: riafarov, okurz


## Reproducible

Fails since (at least) Build [20200814](https://openqa.opensuse.org/tests/1362475)


## Expected result

Last good: [20200813](https://openqa.opensuse.org/tests/1361271) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=NET&machine=64bit&test=autoyast_gnome&version=Tumbleweed)
Comment 2 Ladislav Slezák 2020-08-18 16:44:08 UTC
Actually it's not stuck, it's iterating over a huge amount of packages.

It's related to the this change: https://github.com/yast/yast-autoinstallation/pull/665/files#diff-b1c37c173b1abb406de4f62a9888c301R46

I'll try adding some speed optimization...
Comment 3 Ladislav Slezák 2020-08-19 14:11:30 UTC
Unfortunately it turned out small optimizations do not help much.

The problem is that the OSS repository contains more than 60.000 (!) packages. Iterating over dependencies of all packages will take lot of time, speed optimizations can help only a bit.

In pkg-bindings we have Pkg.PkgQueryProvides() function (https://github.com/yast/yast-pkg-bindings/blob/3ed52e693ca6a79fd7ee6db41e024137268d517c/src/Package.cc#L115) which handles the "Provides" RPM dependency in a very efficient way.

I thought we could create a similar function (PkgQuerySupplements) which would handle the "Supplements" dependency used in by AutoYaST. Unfortunately libzypp does not support that, it only supports WhatProvides (https://github.com/openSUSE/libzypp/blob/master/zypp/sat/WhatProvides.h) and WhatObsoletes (https://github.com/openSUSE/libzypp/blob/master/zypp/sat/WhatObsoletes.h).



So my proposal is:

1) Switch from "Supplements: autoyast(foo)" to Provides:

    Provides: autoyast-tag()
    Provides: autoyast-tag(foo)

   ("autoyast-tag" is just a proposal, feel free to suggest anything better ;-))


2) In the code:

  a) first get packages providing "autoyast-tag()"
     using Pkg.PkgQueryProvides("autoyast-tag()"),
     that should return few packages (like a dozen or so)

  b) then for that few packages get all dependencies
     and find out that "autoyast-tag(.*)" details

  That should be both fast and memory efficient.
Comment 4 Ladislav Slezák 2020-08-19 15:03:10 UTC
Question to the libzypp team:

Is there a way how to efficiently query for packages providing a specific "Supplements"? Or do we need to switch to "Provides" as proposed in the previous comment?
Comment 5 Stefan Hundhammer 2020-08-19 15:13:15 UTC
I suggested on IRC that you libzypp maintainers might know of a completely different solution to this problem. While we tried a brute force approach (which does not scale anymore due to the sheer number of involved packages), maybe there is a different algorithm?

If you decide there is none, we can always fall back to Ladislav's suggested approach. But please check with your expert knowledge if that actually does what we need.
Comment 6 Michael Andres 2020-08-19 16:00:16 UTC
(In reply to Ladislav Slezák from comment #3)
> I thought we could create a similar function (PkgQuerySupplements) which
> would handle the "Supplements" dependency used in by AutoYaST. Unfortunately
> libzypp does not support that, it only supports WhatProvides

WhatProvides is an index built and maintained even inside the resolver. No one in the resolver is interested in supplements. But I guess you don't need an index. 

It's probably sufficient if a query is done inside the pkg-bindings rather than moving the whole haystack into ruby. Even searching for all supplements in 65995 packages with zypper is IMO reasonable fast (incl. refresh checks and screen output):

  ma@fibonacci:~ (0) $ zypper se -s | wc -l
  65995
  ma@fibonacci:~ (0) $ time zypper search -s -v --supplements /./
  ...
  real    0m5.519s
  user    0m3.797s
  sys     0m0.065s

In pkg-bindings you can do a PoolQuery 
   * PoolQuery q;
   * q.setMatchGlob();
   * q.addKind(ResKind::package);
   * q.addAttribute(sat::SolvAttr::supplements, "autoyast(*)");

The PoolQueryIterator visits all sat::Solavables that do contain matches and offers access to the Matches found within each solvable.

Example is in the PoolQuery / PoolQueryIterator class documentation. If you need help with the code, just ask.
Comment 7 Michael Andres 2020-08-19 17:04:59 UTC
// g++ -std=c++17 -O3 -Wall test.cc -lzypp  -o test 
#include <iostream>
#include <zypp/misc/DefaultLoadSystem.h>
#include <zypp/PoolQuery.h>

using std::cout;
using std::endl;
using namespace zypp;

int main( int argc, const char * argv[] )
{
  misc::defaultLoadSystem( "/", misc::LS_READONLY|misc::LS_NOREFRESH );

  //--- HERE.....
  PoolQuery q;
  q.setMatchGlob();
  q.addKind(ResKind::package);
  q.addAttribute(sat::SolvAttr::supplements, "autoyast(*)");

  for ( PoolQuery::const_iterator it = q.begin(), end = q.end(); it != end; ++it ) {
    sat::Solvable solv { *it };
    cout << "natch in Solvable " << solv << endl;

    for( PoolQuery::const_iterator::matches_iterator mit = it.matchesBegin(), end = it.matchesEnd(); mit != end; ++mit ) {
      sat::LookupAttr::iterator attr { *mit };
      cout << "  `" << attr.asString() << "´" << endl;
    }
  }
  //--- HERE.....
  return 0;
}
Comment 8 Ladislav Slezák 2020-08-19 17:11:33 UTC
Great! Thank you for the examples!
Comment 10 Ladislav Slezák 2020-08-21 14:57:17 UTC
Proposal for fix: https://github.com/yast/yast-pkg-bindings/pull/136