maybe the most important thing is, I forgot, when simply zypper up on that situation from the command line, the whole zypper process wouldnt show any problems at all, it didnt complain that it wouldnt do anything or would abort agaon due to urlgrabber.

I only managed to find out by calling the online_update script (link) manually myself from the command line.

This machine did manage to update select packages and updates in the past weeks ever since the upgrade to 15.4, but apparently failed now for whatever reason during the past days or so. I wonder why it got bothered by urlgrabber problem as of recently and not before during early weeks of 15.4?

Also: no package or update would become installed related to this urlgrabber stuff, so I wonder why zypper onlineupdate script fails to begin with.

maybe hopefully this situation can be enhanced so that no leap installation would be left outside of the update cycle when coming to this kind of a situation.
thanks.

I'm forwarding it to the YAST maintainers. They maintain this scrip:
  opensuse.org-online_update -> /usr/lib/YaST2/bin/online_update


It appears to me that the script basically calls 'zypper --no-interactive patch'. AFAICS the command exits 4-ZYPPER_EXIT_ERR_ZYPP -as expected- if unsolved dependency issues prevent the installation. Executed unattended this command is suitable IFF it is asserted that errors are propagated to the user. Otherwise `zypper up` may be a better choice, because it at updates whatever is possible.

Or a combination "zypper up; zypper patch"...


(JFYI regarding 'for cat in $AOU_PATCH_CATEGORIES ; do runzypper $zcmd --category ...'. --category supports passing a ,-separated list of categories. Just in case you want to avoid executing multiple commands).

I'm afraid that we can't change the behavior of SLE 15, which is the base for Leap 15.X. In SLE it has to always call only `zypper patch` as it's expected to only really apply patches.

The problem here is IMO reporting as the user does not know that something went wrong and needs to be resolved manually.

So, this does package updates automatically in the background with a cron job / systemd timer. How is this supposed to work when that process is unattended? What's the intended concept behind this?

Wasn't the reason for all those desktop update applets to avoid exactly this situation: There is something that cannot be handled automatically, so user interaction is required? The applet can open a pop-up window or at least show a special icon in the task bar to raise the user's attention.

How is that supposed to work here?

So those cron jobs are created by the "Online Update Configuration" YaST module from the yast2-online-update-configuration package.

In its one interactive screen, it has an option "Skip Interactive Patches" (checked by default) which AFAICS is what Lukas mentioned in comment #3. But as Lukas wrote, it really only refers to patches like used on SLE releases.

Created attachment 860893 [details]
Screenshot: YaST Online Update Configuration

That "Online Update Configuration" YaST module does little more than write or modify this file with the settings for the online update:

/etc/sysconfig/automatic_online_update

AOU_ENABLE_CRONJOB="false"
AOU_SKIP_INTERACTIVE_PATCHES="true"
AOU_AUTO_AGREE_WITH_LICENSES="false"
AOU_INCLUDE_RECOMMENDS="false"
AOU_PATCH_CATEGORIES=""

The called script is quite simplistic:

https://github.com/yast/yast-online-update-configuration/blob/master/src/bin/online_update


zyppercmd="/usr/bin/zypper"
...
if [ -x ${zyppercmd}  ]
then
    zcmd="${zyppercmd} --non-interactive --quiet patch"
...

AFAICS this behaves as expected. It does apply the pending patches in that cron job; as far as that is possible.

But sometimes, human interaction might still be required; that's when it may fail because by the very nature of those cron jobs, there is no human interaction. It does not have any connection to a console or desktop to notify a human.

This is where the convenience of having this done in the background in an unattended way fails. In traditional Linux / Unix sys admin tasks, it may start sending mails to the root user; but those mails usually also remain unread because it's very uncommon these days to log in as the root user.

So how could this be solved? Start sending mails to root despite knowing that this is pretty pointless? Write to the journal / syslog in the (unrealistic) hope that somebody reads that and takes action?

Or do what we have done for many years: Expect that a responsible admin will check every once in a while how the machine is doing, e.g. with a command like "zypper list-updates" / "zypper list-patches"; or the GUI counterpart?

If you are using one of the major desktops, there will be an updater applet that should also have some sort of notification; like the number of pending updates.

my general problem with this sitation and also related bug in the updater stack / stalls:

> <https://bugzilla.opensuse.org/show_bug.cgi?id=1202498>

is that my understanding so far was that zypper up was similar / same to zypper patch.

I apparently never use zypper up (manually, interactively myself) but only zypper up and zypper dup when LEAPs EOL.

anyhow, so I didnt understand why the system was not up to date even though the script was activated, active, listed in cron.daily etc

> opensuse.org-online_update -> /usr/lib/YaST2/bin/online_update


I even went looking  for the file

/etc/sysconfig/cron

and its setting

MAX_NOT_RUN="1"

(and I set this to the lowest number I could think of) only to observe that the machine was still not updated (or some of my machines eventually many of them came into this kind of situation))

So I finally tried to manually execute that /etc/cron.daily/opensuse.org-online_update

from the command line  via sudo

which eventually brought up the many questions about problem packages etc.


When I am not interactively using the desktop of such a machine, and having still everything configured happily and neatly for automatic stuff, it still leads me as a user into situations where the machine would be left unprotected as it never calls up but only patch  for the update stack.

Looking at the other bug

> <https://bugzilla.opensuse.org/show_bug.cgi?id=1202498> it gives me the impression that even brand new and freshly installed 15.4 for example happen to arrive in this situation without any leftovers or unclean situations from maybe earlier LEAP upgrades etc.

It seems to be too easy for the opensuse project maintainers and software engineers to fail with packages and updates thus maneuvering the opensuse users into such situations. To be honest this is my moment of "windowsupdates" woes that I am familiar with from the microsoft windows world and their endless problems with their respective update stack and packages. I am not very happy about it.

so when I access my machines now remotely, from now on I would need to manually also trigger all sorts and variations of the zypper command and the onlineupdate script from yast etc just to make (more) sure that problematic stuff gets reported and patches updates etc executed and applied to the fullest extent possible.

I have the gut feeling that I "suffered" from these kind of symptoms on earlier LEAP versions and opensuse in the past as well, and I just never understood why some machines every now and then were not fully updated.

Wouldnt it be great if zypper up instead of zypper patch or in addition or both were called and executed or similar silly simple ideas?

thanks.

(In reply to andreas bittner from comment #10)
> Wouldnt it be great if zypper up instead of zypper patch or in addition or
> both were called and executed or similar silly simple ideas?

Long story short: It would be great if it was so simple. But zypper patch and zypper up are just two different commands with different behavior. The YaST Online Update is expected to apply patches (only) and does not have any ways to solve conflicts or report to the user as there is no user listening. It would not make sense to produce patches if zypper up was always called, but we do produce patches, so my expectation is that they are wanted and 'preferred'.

In fact, it's also expected that zypper patch should not produce conflicts, but that expectation is also false.

The same code is used on SLE and openSUSE and it IMO must not call zypper up at least on SLE. We may think about different solution between openSUSE and SLE, but then it's possible to migrate openSUSE to SLE and we have a problem that something would need to change `up` to `patch` there.

IMO there are two ways out:
1. Drop the functionality completely as it can't solve some special cases
2. Force zypper to resolve these conflicts automatically

#1 is probably something nobody wants
#2 is dangerous and needs MA's opinion

We may think about using `--force-resolution --solver-focus Update` but (if that works as I understand, it may uninstall something you want to keep.

When using `--force-resolution --solver-focus Installed` it may otherwise keep the installed software, but make your system insecure long-term.

This is just what I found in `man zypper`, both seem to be "insecure" in some way. I've never used these options ... and IMO the decision is on project management anyway.

In an ideal world, all those system updating tasks would always run without problems, without questions, and would be 100% reliable. In that case, a completely unattended automatic update in a cron job would never cause any problems.

But we don't live in an ideal world, and in Linux-land we give the admin choices, and the power to implement them. And sometimes, those choices can result in scenarios where we need to ask the user questions, because several solutions may be equally desirable or undesirable; we simply cannot know.

And of course there is also the human factor: Sometimes package maintainers make mistakes, or the upstream authors decide to change the behavior.

I can understand your frustration, but some things need some human to check, at least every once in a while.


If you have a large server farm with a number of unattended machines, you might consider a centralized server management solution to ease the pain.

(In reply to Lukas Ocilka from comment #11)
> In fact, it's also expected that zypper patch should not produce conflicts,
> but that expectation is also false.

No - exactly this is not to be expected with 'patch'!

'up' just updates what is possible and simply skips not applicable version updates. This will never produce conflicts.

'patch' however enforces that the defect or vulnerability will be resolved. Desired by simply updating the affected packages. If this is not possible, 
the user has to decide whether he want's to skip the patch or remove the packages. 

The typical conflict are installed packages not covered by the patch, which require the installed and vulnerable version. 

> 2. Force zypper to resolve these conflicts automatically

Don't even think about using '--force-resolution'!

Force-resolution allows the solver to remove packages in order to solve the conflict. So it will remove the installed packages that block the update. And of course all packages depending on them as well. What if an installed kernel or glibc is blocking the patch? Or the customers own or 3rd party software. 

Also --solver-focus will not help. It just tunes the job and says how 'agressive' the update should be. It will try to update as little as possible or as much as possible. But it can not resolve any conflict. If updating any package would resolve the conflict, the solver would do it. 

The problem here is that just deleting packages resolves the conflict. Or the patch has to be skipped. But there is no heuristic telling which is ok.

If we are actually unable to inform the user that certain patches are not applicable and need his decision (and supervision) - we have a problem.

We could add an option to 'zypper patch' telling to auto-resolve conflicts by sipping a patch - if this is a resolution option. This would at least try to apply as many patches as possible. If patches were skipped zypper could return 
ZYPPER_EXIT_INF_UPDATE_NEEDED(100). 

But it will not solve the basic issue. Needed patches will not be applied and the system might be vulnerable.

(In reply to Stefan Hundhammer from comment #13)
> If you have a large server farm with a number of unattended machines, you
> might consider a centralized server management solution to ease the pain.

In ancient times those issues were solved by sending mail to root. 
Cron sends mail - or logs to syslog - if tasks fail or write messages to the screen. If yast-online-update suppresses this - maybe there's an opt. to enable it.

(In reply to Michael Andres from comment #14)
> (In reply to Lukas Ocilka from comment #11)
> > In fact, it's also expected that zypper patch should not produce conflicts,
> > but that expectation is also false.
> 
> No - exactly this is not to be expected with 'patch'!

Let's call this a Devil's advocate approach ;)

I'll be more clear: User / Admin expects that there are no conflicts when applying patches (or that they are automatically resolved). Obviously, the implementation itself is kind of a "conflict" with insecure version of some software.

> > 2. Force zypper to resolve these conflicts automatically
> 
> Don't even think about using '--force-resolution'!
> ...
> The problem here is that just deleting packages resolves the conflict. Or
> the patch has to be skipped. But there is no heuristic telling which is ok.

Which is exactly my point, we can't do that automatically.

> If we are actually unable to inform the user that certain patches are not
> applicable and need his decision (and supervision) - we have a problem.

And that's why we have this bugreport.

> We could add an option to 'zypper patch' telling to auto-resolve conflicts
> by sipping a patch - if this is a resolution option. This would at least try
> to apply as many patches as possible. If patches were skipped zypper could
> return 
> ZYPPER_EXIT_INF_UPDATE_NEEDED(100). 

Looks like a nice feature request. But then we still don't have the reporting part. Anyway, it would already improve the security a lot.

Thinking about it ... there ARE ways to send e-mail from connadline, e.g., https://tecadmin.net/ways-to-send-email-from-linux-command-line/ but honestly, they just don't work unless you are sending e-mails to local users or unprotected mail domains. And nobody really reads root's e-mails.

> But it will not solve the basic issue. Needed patches will not be applied
> and the system might be vulnerable.

For that, the only real solution is a centralized way. SUSE Manager (there's also a free version available), Salt, ...

See, e.g., https://github.com/uyuni-project/uyuni

After reading the discussion, I feel that the only solution is to stop pretending that we're able to silently patch the system and remove the script.

Where to discuss it first? SLE / OS?

(In reply to Michal Filka from comment #18)
> After reading the discussion, I feel that the only solution is to stop
> pretending that we're able to silently patch the system 

Yes

> and remove the script.

Or find some way to report the issue. 
And clearly state and document how to supervise it.

An icon on the e.g. desktop showing basically the `zypper pchk` result at the end of the script. It's expected to be 0 then. (maybe '--no-refresh pchk' so you are not fooled by patches released between your 'patch' and 'pchk'.

For non desktop users: The online_update script could store the result of it's latest 'pchk'. Some online_update_result script could show the result and return the code. 

> Last online_update on Aug 25 13:51:53 did not complete.
> `zypper pch` returned: 101
> 
>  Found 299 applicable patches:
>  Category    | Updatestack | Patches | Locked | Included categories
>  ------------+-------------+---------+--------+--------------------
>  security    | -           | 126     | 1      | 
>  recommended | 4           | 165     | -      | 
>  optional    | -           | 3       | -      | feature
> 
>  1 patch locked
>  3 patches optional   (use '--with-optional' to include optional patches)
>  295 patches needed (126 security patches)

or

> Last online_update on Aug 25 13:51:53 completed.
> `zypper pch` returned: 0
> 
>  0 patches needed (0 security patches)

If that's clearly communicated, even unattended systems can be monitored. A cron job sending mail if online_update_result return !=0. Or pulling the status the 'ssh user@remote.host online_update_result'.

The basic idea of the online_update script is not bad.

Created attachment 861737 [details]
Screenshot how my firewall solves autoupdate issues

How? Reports them in UI. So unless someone gets in, it's blocked.
And this is auto-update with "do it every day".
As the FW does not have my personal e-mail address, there is no way to tell me.

not to derail, but meanwhile on 15.5 here, today (yesterday) the same situation occurred

apparently one unresolvable? situation hinders the whole cron daily update script to execute and install (decide) all of the amassing updates on the machine from that point on forwards:

-----------

 2023-07-21 12:24:04
$ sudo /etc/cron.daily/opensuse.org-online_update
[sudo] password for root:
Problem: the to be installed xfce4-panel-4.18.4-bp155.2.3.1.x86_64 requires 'xfce4-panel-branding = 4.18.4', but this requirement cannot be provided
  not installable providers: xfce4-panel-branding-upstream-4.18.4-bp155.2.3.1.noarch[repo-backports-update]
 Solution 1: Following actions will be done:
  deinstallation of xfce4-panel-4.18.3-bp155.1.5.x86_64
  deinstallation of xfce4-panel-lang-4.18.3-bp155.1.5.noarch
  deinstallation of xfce4-power-manager-plugin-4.18.2-bp155.2.3.1.x86_64
  deinstallation of xfce4-pulseaudio-plugin-0.4.7-bp155.2.3.1.x86_64
  deinstallation of xfce4-sensors-plugin-1.4.4-bp155.1.5.x86_64
  deinstallation of xfce4-datetime-plugin-0.8.3-bp155.1.5.x86_64
  deinstallation of xfce4-sensors-plugin-lang-1.4.4-bp155.1.5.noarch
  deinstallation of xfce4-datetime-plugin-lang-0.8.3-bp155.1.5.noarch
 Solution 2: deinstallation of xfce4-panel-branding-openSUSE-4.18.0+git0.9a2f754-lp155.1.6.noarch
 Solution 3: do not install patch:openSUSE-2023-175-1.noarch
 Solution 4: break xfce4-panel-4.18.4-bp155.2.3.1.x86_64 by ignoring some of its dependencies

Choose from above solutions by number or cancel [1/2/3/4/c/d/?] (c): c
[user01tux01 ~] 2023-07-21 12:24:24

any solution on situations such as this?
thanks

forgot: zypper lu

shows plenty of pending updates as a huge list, that didnt get applied. manual intervention is needed interactive handling of zypper

@andreas bittner one problem with your original suggestion to just skip the conflicting patch and continue with the others is that this would more likely bring you into a situation of a completely untested combination of patches. However that could be an option. Still it leaves the problem that any sysadmin needs to ensure that you see error reports eventually, either by trying to upgrade manually from time to time or forwarding the Cron reports to your external email or use one of the suggested management frameworks