Bug 1175317

Just in case could be helpful for a further diagnosis, it seems that the installation gets stuck (infinite loop?) when checking resolvables[1][2].

I was able to reproduce it using the latest NET image[3] but NOT using the latest DVD image[3].


[1]https://github.com/yast/yast-yast2/blob/799012eaa587b663a1cf4bc42eb0384a36d4ea65/library/packages/src/lib/y2packager/resolvable.rb#L149
[2] https://github.com/yast/yast-pkg-bindings/blob/6abebc1ed823ccd06495ad39a19a4f5b89279bcb/src/Resolvable_Properties.cc#L992
[3]https://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-NET-x86_64-Snapshot20200816-Media.iso
[4]https://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20200816-Media.iso

Actually it's not stuck, it's iterating over a huge amount of packages.

It's related to the this change: https://github.com/yast/yast-autoinstallation/pull/665/files#diff-b1c37c173b1abb406de4f62a9888c301R46

I'll try adding some speed optimization...

Unfortunately it turned out small optimizations do not help much.

The problem is that the OSS repository contains more than 60.000 (!) packages. Iterating over dependencies of all packages will take lot of time, speed optimizations can help only a bit.

In pkg-bindings we have Pkg.PkgQueryProvides() function (https://github.com/yast/yast-pkg-bindings/blob/3ed52e693ca6a79fd7ee6db41e024137268d517c/src/Package.cc#L115) which handles the "Provides" RPM dependency in a very efficient way.

I thought we could create a similar function (PkgQuerySupplements) which would handle the "Supplements" dependency used in by AutoYaST. Unfortunately libzypp does not support that, it only supports WhatProvides (https://github.com/openSUSE/libzypp/blob/master/zypp/sat/WhatProvides.h) and WhatObsoletes (https://github.com/openSUSE/libzypp/blob/master/zypp/sat/WhatObsoletes.h).



So my proposal is:

1) Switch from "Supplements: autoyast(foo)" to Provides:

    Provides: autoyast-tag()
    Provides: autoyast-tag(foo)

   ("autoyast-tag" is just a proposal, feel free to suggest anything better ;-))


2) In the code:

  a) first get packages providing "autoyast-tag()"
     using Pkg.PkgQueryProvides("autoyast-tag()"),
     that should return few packages (like a dozen or so)

  b) then for that few packages get all dependencies
     and find out that "autoyast-tag(.*)" details

  That should be both fast and memory efficient.

Question to the libzypp team:

Is there a way how to efficiently query for packages providing a specific "Supplements"? Or do we need to switch to "Provides" as proposed in the previous comment?

I suggested on IRC that you libzypp maintainers might know of a completely different solution to this problem. While we tried a brute force approach (which does not scale anymore due to the sheer number of involved packages), maybe there is a different algorithm?

If you decide there is none, we can always fall back to Ladislav's suggested approach. But please check with your expert knowledge if that actually does what we need.

(In reply to Ladislav Slezák from comment #3)
> I thought we could create a similar function (PkgQuerySupplements) which
> would handle the "Supplements" dependency used in by AutoYaST. Unfortunately
> libzypp does not support that, it only supports WhatProvides

WhatProvides is an index built and maintained even inside the resolver. No one in the resolver is interested in supplements. But I guess you don't need an index. 

It's probably sufficient if a query is done inside the pkg-bindings rather than moving the whole haystack into ruby. Even searching for all supplements in 65995 packages with zypper is IMO reasonable fast (incl. refresh checks and screen output):

  ma@fibonacci:~ (0) $ zypper se -s | wc -l
  65995
  ma@fibonacci:~ (0) $ time zypper search -s -v --supplements /./
  ...
  real    0m5.519s
  user    0m3.797s
  sys     0m0.065s

In pkg-bindings you can do a PoolQuery 
   * PoolQuery q;
   * q.setMatchGlob();
   * q.addKind(ResKind::package);
   * q.addAttribute(sat::SolvAttr::supplements, "autoyast(*)");

The PoolQueryIterator visits all sat::Solavables that do contain matches and offers access to the Matches found within each solvable.

Example is in the PoolQuery / PoolQueryIterator class documentation. If you need help with the code, just ask.

// g++ -std=c++17 -O3 -Wall test.cc -lzypp  -o test 
#include <iostream>
#include <zypp/misc/DefaultLoadSystem.h>
#include <zypp/PoolQuery.h>

using std::cout;
using std::endl;
using namespace zypp;

int main( int argc, const char * argv[] )
{
  misc::defaultLoadSystem( "/", misc::LS_READONLY|misc::LS_NOREFRESH );

  //--- HERE.....
  PoolQuery q;
  q.setMatchGlob();
  q.addKind(ResKind::package);
  q.addAttribute(sat::SolvAttr::supplements, "autoyast(*)");

  for ( PoolQuery::const_iterator it = q.begin(), end = q.end(); it != end; ++it ) {
    sat::Solvable solv { *it };
    cout << "natch in Solvable " << solv << endl;

    for( PoolQuery::const_iterator::matches_iterator mit = it.matchesBegin(), end = it.matchesEnd(); mit != end; ++mit ) {
      sat::LookupAttr::iterator attr { *mit };
      cout << "  `" << attr.asString() << "´" << endl;
    }
  }
  //--- HERE.....
  return 0;
}

Great! Thank you for the examples!

Proposal for fix: https://github.com/yast/yast-pkg-bindings/pull/136