Bug 1202448 - Unattended updates via cron job: One pending interactive question in zypper rendering all efforts of opensuse.org-online-update pointless, resulting in machine seemingly un-patched forever
Unattended updates via cron job: One pending interactive question in zypper r...
Status: CONFIRMED
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: YaST2
Leap 15.4
x86-64 openSUSE Leap 15.4
: P5 - None : Normal (vote)
: ---
Assigned To: YaST Team
Jiri Srain
https://trello.com/c/MCO6xoqy
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2022-08-16 14:13 UTC by andreas bittner
Modified: 2022-09-26 16:05 UTC (History)
1 user (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Screenshot: YaST Online Update Configuration (54.39 KB, image/png)
2022-08-18 09:19 UTC, Stefan Hundhammer
Details
Screenshot how my firewall solves autoupdate issues (48.69 KB, image/png)
2022-09-26 16:05 UTC, Lukas Ocilka
Details

Note You need to log in before you can comment on or make changes to this bug.
Description andreas bittner 2022-08-16 14:13:06 UTC
one pending interactive question in zypper rendering all efforts of opensuse.org-online-update pointless, resulting in machine seemingly un-patched forever


on a 15.4 which has been upgraded from previous opensuse leap versions there seemed to be a single package which blocked the successful execution of 

opensuse.org-online-update proceedings

via cron.daily

which links to

opensuse.org-online_update -> /usr/lib/YaST2/bin/online_update


I have noticed that cron (daily) calls that script in some ways with interactive prompts being answered by some parameters or similar but which eventually fails when there is something amiss like in this situation:

I have noticed as zypper lu showed plenty of updates which hadnt been applied even though the machine was activated for daily updates via yast (that script above in cron.daily)

first, manually checking brought up this:



 sudo /etc/cron.daily/opensuse.org-online_update
Problem: the to be installed patch:openSUSE-SLE-15.4-2022-2487-1.noarch conflicts with 'python-urlgrabber.noarch < 4.1.0-150400.4.3.                                                1' provided by the installed python-urlgrabber-3.9.1-1.37.noarch
 Solution 1: deinstallation of python-urlgrabber-3.9.1-1.37.noarch
 Solution 2: do not install patch:openSUSE-SLE-15.4-2022-2487-1.noarch

Choose from above solutions by number or cancel [1/2/c/d/?] (c): c

-----

it automatically excited / cancelled there. so this machine was missing plenty of days of updates for example shown by manually triggering zypper e.g.


sudo zypper up
Loading repository data...
Reading installed packages...

The following 16 packages are going to be upgraded:
  chromium kernel-macros libQt5Designer5 libQt5DesignerComponents5 libQt5Help5 libqt5-linguist libqt5-linguist-devel libqt5-qdbus
  libqt5-qtpaths libqt5-qttools libqt5-qttools-devel libqt5-qttools-doc libqt5-qttools-qhelpgenerator tar tar-lang tar-rmt

The following 5 NEW packages are going to be installed:
  kernel-default-5.14.21-150400.24.18.1 kernel-default-devel-5.14.21-150400.24.18.1 kernel-default-extra-5.14.21-150400.24.18.1
  kernel-default-optional-5.14.21-150400.24.18.1 kernel-devel-5.14.21-150400.24.18.1

The following package requires a system reboot:
  kernel-default-5.14.21-150400.24.18.1

16 packages to upgrade, 5 new.
Overall download size: 296.9 MiB. Already cached: 0 B. After the operation, additional 298.4 MiB will be used.

    Note: System reboot required.
Continue? [y/n/v/...? shows all options] (y): ^C

-------


python-urlgrabber package was causing the trouble. apparently there is a newer package in 15.4 which is called python3-urlgrabber or something, I dont understand just yet or at all why this hadnt been upgraded or handled when having moved from 15.3 to 15.4 though.

manual removal and manual re-installation of the newer urlgrabber package was needed. though i dont know what or if i need this urlgrabber package at all. is this some default package on leap? i didnt come across this bug behavior on many of my machines.

---------

sudo zypper rm python-urlgrabber
Reading installed packages...
Resolving package dependencies...

The following package is going to be REMOVED:
  python-urlgrabber

1 package to remove.
After the operation, 297.4 KiB will be freed.
Continue? [y/n/v/...? shows all options] (y): y
(1/1) Removing python-urlgrabber-3.9.1-1.37.noarch ...........................................................................[done]
There are running programs which still use files and libraries deleted or updated by recent upgrades. They should be restarted to be                                                nefit from the latest updates. Run 'zypper ps -s' to list these programs.


 
--------------

after the manual removal of that urlgrabber package, the cron.daily script call would go through properly


sudo /etc/cron.daily/opensuse.org-online_update

The following 16 packages are going to be upgraded:
  chromium kernel-macros libQt5Designer5 libQt5DesignerComponents5 libQt5Help5 libqt5-linguist libqt5-linguist-devel libqt5-qdbus
  libqt5-qtpaths libqt5-qttools libqt5-qttools-devel libqt5-qttools-doc libqt5-qttools-qhelpgenerator tar tar-lang tar-rmt

The following 5 NEW packages are going to be installed:
  kernel-default-5.14.21-150400.24.18.1 kernel-default-devel-5.14.21-150400.24.18.1 kernel-default-extra-5.14.21-150400.24.18.1
  kernel-default-optional-5.14.21-150400.24.18.1 kernel-devel-5.14.21-150400.24.18.1

The following 4 NEW patches are going to be installed:
  openSUSE-2022-10086 openSUSE-SLE-15.4-2022-2735 openSUSE-SLE-15.4-2022-2736 openSUSE-SLE-15.4-2022-2803

The following patch requires a system reboot:
  openSUSE-SLE-15.4-2022-2803

The following package requires a system reboot:
  kernel-default-5.14.21-150400.24.18.1

16 packages to upgrade, 5 new.
Overall download size: 296.9 MiB. Already cached: 0 B. After the operation, additional 298.4 MiB will be used.

Continue? [y/n/v/...? shows all options] (y): y

Warning: One of the installed patches requires a reboot of your machine. Reboot as soon as possible.

---------
Comment 1 andreas bittner 2022-08-16 14:17:20 UTC
maybe the most important thing is, I forgot, when simply zypper up on that situation from the command line, the whole zypper process wouldnt show any problems at all, it didnt complain that it wouldnt do anything or would abort agaon due to urlgrabber.

I only managed to find out by calling the online_update script (link) manually myself from the command line.

This machine did manage to update select packages and updates in the past weeks ever since the upgrade to 15.4, but apparently failed now for whatever reason during the past days or so. I wonder why it got bothered by urlgrabber problem as of recently and not before during early weeks of 15.4?

Also: no package or update would become installed related to this urlgrabber stuff, so I wonder why zypper onlineupdate script fails to begin with.

maybe hopefully this situation can be enhanced so that no leap installation would be left outside of the update cycle when coming to this kind of a situation.
thanks.
Comment 2 Michael Andres 2022-08-16 16:30:38 UTC
I'm forwarding it to the YAST maintainers. They maintain this scrip:
  opensuse.org-online_update -> /usr/lib/YaST2/bin/online_update


It appears to me that the script basically calls 'zypper --no-interactive patch'. AFAICS the command exits 4-ZYPPER_EXIT_ERR_ZYPP -as expected- if unsolved dependency issues prevent the installation. Executed unattended this command is suitable IFF it is asserted that errors are propagated to the user. Otherwise `zypper up` may be a better choice, because it at updates whatever is possible.

Or a combination "zypper up; zypper patch"...


(JFYI regarding 'for cat in $AOU_PATCH_CATEGORIES ; do runzypper $zcmd --category ...'. --category supports passing a ,-separated list of categories. Just in case you want to avoid executing multiple commands).
Comment 3 Lukas Ocilka 2022-08-17 08:33:02 UTC
I'm afraid that we can't change the behavior of SLE 15, which is the base for Leap 15.X. In SLE it has to always call only `zypper patch` as it's expected to only really apply patches.

The problem here is IMO reporting as the user does not know that something went wrong and needs to be resolved manually.
Comment 4 Stefan Hundhammer 2022-08-18 08:33:06 UTC
So, this does package updates automatically in the background with a cron job / systemd timer. How is this supposed to work when that process is unattended? What's the intended concept behind this?

Wasn't the reason for all those desktop update applets to avoid exactly this situation: There is something that cannot be handled automatically, so user interaction is required? The applet can open a pop-up window or at least show a special icon in the task bar to raise the user's attention.

How is that supposed to work here?
Comment 5 Stefan Hundhammer 2022-08-18 09:18:37 UTC
So those cron jobs are created by the "Online Update Configuration" YaST module from the yast2-online-update-configuration package.

In its one interactive screen, it has an option "Skip Interactive Patches" (checked by default) which AFAICS is what Lukas mentioned in comment #3. But as Lukas wrote, it really only refers to patches like used on SLE releases.
Comment 6 Stefan Hundhammer 2022-08-18 09:19:10 UTC
Created attachment 860893 [details]
Screenshot: YaST Online Update Configuration
Comment 7 Stefan Hundhammer 2022-08-18 09:29:16 UTC
That "Online Update Configuration" YaST module does little more than write or modify this file with the settings for the online update:

/etc/sysconfig/automatic_online_update

AOU_ENABLE_CRONJOB="false"
AOU_SKIP_INTERACTIVE_PATCHES="true"
AOU_AUTO_AGREE_WITH_LICENSES="false"
AOU_INCLUDE_RECOMMENDS="false"
AOU_PATCH_CATEGORIES=""
Comment 8 Stefan Hundhammer 2022-08-18 09:37:31 UTC
The called script is quite simplistic:

https://github.com/yast/yast-online-update-configuration/blob/master/src/bin/online_update


zyppercmd="/usr/bin/zypper"
...
if [ -x ${zyppercmd}  ]
then
    zcmd="${zyppercmd} --non-interactive --quiet patch"
...
Comment 9 Stefan Hundhammer 2022-08-18 10:03:00 UTC
AFAICS this behaves as expected. It does apply the pending patches in that cron job; as far as that is possible.

But sometimes, human interaction might still be required; that's when it may fail because by the very nature of those cron jobs, there is no human interaction. It does not have any connection to a console or desktop to notify a human.

This is where the convenience of having this done in the background in an unattended way fails. In traditional Linux / Unix sys admin tasks, it may start sending mails to the root user; but those mails usually also remain unread because it's very uncommon these days to log in as the root user.

So how could this be solved? Start sending mails to root despite knowing that this is pretty pointless? Write to the journal / syslog in the (unrealistic) hope that somebody reads that and takes action?

Or do what we have done for many years: Expect that a responsible admin will check every once in a while how the machine is doing, e.g. with a command like "zypper list-updates" / "zypper list-patches"; or the GUI counterpart?

If you are using one of the major desktops, there will be an updater applet that should also have some sort of notification; like the number of pending updates.
Comment 10 andreas bittner 2022-08-18 14:31:52 UTC
my general problem with this sitation and also related bug in the updater stack / stalls:

> <https://bugzilla.opensuse.org/show_bug.cgi?id=1202498>

is that my understanding so far was that zypper up was similar / same to zypper patch.

I apparently never use zypper up (manually, interactively myself) but only zypper up and zypper dup when LEAPs EOL.

anyhow, so I didnt understand why the system was not up to date even though the script was activated, active, listed in cron.daily etc

> opensuse.org-online_update -> /usr/lib/YaST2/bin/online_update


I even went looking  for the file

/etc/sysconfig/cron

and its setting

MAX_NOT_RUN="1"

(and I set this to the lowest number I could think of) only to observe that the machine was still not updated (or some of my machines eventually many of them came into this kind of situation))

So I finally tried to manually execute that /etc/cron.daily/opensuse.org-online_update

from the command line  via sudo

which eventually brought up the many questions about problem packages etc.


When I am not interactively using the desktop of such a machine, and having still everything configured happily and neatly for automatic stuff, it still leads me as a user into situations where the machine would be left unprotected as it never calls up but only patch  for the update stack.

Looking at the other bug

> <https://bugzilla.opensuse.org/show_bug.cgi?id=1202498> it gives me the impression that even brand new and freshly installed 15.4 for example happen to arrive in this situation without any leftovers or unclean situations from maybe earlier LEAP upgrades etc.

It seems to be too easy for the opensuse project maintainers and software engineers to fail with packages and updates thus maneuvering the opensuse users into such situations. To be honest this is my moment of "windowsupdates" woes that I am familiar with from the microsoft windows world and their endless problems with their respective update stack and packages. I am not very happy about it.

so when I access my machines now remotely, from now on I would need to manually also trigger all sorts and variations of the zypper command and the onlineupdate script from yast etc just to make (more) sure that problematic stuff gets reported and patches updates etc executed and applied to the fullest extent possible.

I have the gut feeling that I "suffered" from these kind of symptoms on earlier LEAP versions and opensuse in the past as well, and I just never understood why some machines every now and then were not fully updated.

Wouldnt it be great if zypper up instead of zypper patch or in addition or both were called and executed or similar silly simple ideas?

thanks.
Comment 11 Lukas Ocilka 2022-08-18 15:11:41 UTC
(In reply to andreas bittner from comment #10)
> Wouldnt it be great if zypper up instead of zypper patch or in addition or
> both were called and executed or similar silly simple ideas?

Long story short: It would be great if it was so simple. But zypper patch and zypper up are just two different commands with different behavior. The YaST Online Update is expected to apply patches (only) and does not have any ways to solve conflicts or report to the user as there is no user listening. It would not make sense to produce patches if zypper up was always called, but we do produce patches, so my expectation is that they are wanted and 'preferred'.

In fact, it's also expected that zypper patch should not produce conflicts, but that expectation is also false.

The same code is used on SLE and openSUSE and it IMO must not call zypper up at least on SLE. We may think about different solution between openSUSE and SLE, but then it's possible to migrate openSUSE to SLE and we have a problem that something would need to change `up` to `patch` there.

IMO there are two ways out:
1. Drop the functionality completely as it can't solve some special cases
2. Force zypper to resolve these conflicts automatically

#1 is probably something nobody wants
#2 is dangerous and needs MA's opinion
Comment 12 Lukas Ocilka 2022-08-18 15:23:07 UTC
We may think about using `--force-resolution --solver-focus Update` but (if that works as I understand, it may uninstall something you want to keep.

When using `--force-resolution --solver-focus Installed` it may otherwise keep the installed software, but make your system insecure long-term.

This is just what I found in `man zypper`, both seem to be "insecure" in some way. I've never used these options ... and IMO the decision is on project management anyway.
Comment 13 Stefan Hundhammer 2022-08-18 15:42:44 UTC
In an ideal world, all those system updating tasks would always run without problems, without questions, and would be 100% reliable. In that case, a completely unattended automatic update in a cron job would never cause any problems.

But we don't live in an ideal world, and in Linux-land we give the admin choices, and the power to implement them. And sometimes, those choices can result in scenarios where we need to ask the user questions, because several solutions may be equally desirable or undesirable; we simply cannot know.

And of course there is also the human factor: Sometimes package maintainers make mistakes, or the upstream authors decide to change the behavior.

I can understand your frustration, but some things need some human to check, at least every once in a while.


If you have a large server farm with a number of unattended machines, you might consider a centralized server management solution to ease the pain.
Comment 14 Michael Andres 2022-08-18 16:16:07 UTC
(In reply to Lukas Ocilka from comment #11)
> In fact, it's also expected that zypper patch should not produce conflicts,
> but that expectation is also false.

No - exactly this is not to be expected with 'patch'!

'up' just updates what is possible and simply skips not applicable version updates. This will never produce conflicts.

'patch' however enforces that the defect or vulnerability will be resolved. Desired by simply updating the affected packages. If this is not possible, 
the user has to decide whether he want's to skip the patch or remove the packages. 

The typical conflict are installed packages not covered by the patch, which require the installed and vulnerable version. 

> 2. Force zypper to resolve these conflicts automatically

Don't even think about using '--force-resolution'!

Force-resolution allows the solver to remove packages in order to solve the conflict. So it will remove the installed packages that block the update. And of course all packages depending on them as well. What if an installed kernel or glibc is blocking the patch? Or the customers own or 3rd party software. 

Also --solver-focus will not help. It just tunes the job and says how 'agressive' the update should be. It will try to update as little as possible or as much as possible. But it can not resolve any conflict. If updating any package would resolve the conflict, the solver would do it. 

The problem here is that just deleting packages resolves the conflict. Or the patch has to be skipped. But there is no heuristic telling which is ok.

If we are actually unable to inform the user that certain patches are not applicable and need his decision (and supervision) - we have a problem.

We could add an option to 'zypper patch' telling to auto-resolve conflicts by sipping a patch - if this is a resolution option. This would at least try to apply as many patches as possible. If patches were skipped zypper could return 
ZYPPER_EXIT_INF_UPDATE_NEEDED(100). 

But it will not solve the basic issue. Needed patches will not be applied and the system might be vulnerable.
Comment 15 Michael Andres 2022-08-18 16:46:05 UTC
(In reply to Stefan Hundhammer from comment #13)
> If you have a large server farm with a number of unattended machines, you
> might consider a centralized server management solution to ease the pain.

In ancient times those issues were solved by sending mail to root. 
Cron sends mail - or logs to syslog - if tasks fail or write messages to the screen. If yast-online-update suppresses this - maybe there's an opt. to enable it.
Comment 16 Lukas Ocilka 2022-08-19 08:22:47 UTC
(In reply to Michael Andres from comment #14)
> (In reply to Lukas Ocilka from comment #11)
> > In fact, it's also expected that zypper patch should not produce conflicts,
> > but that expectation is also false.
> 
> No - exactly this is not to be expected with 'patch'!

Let's call this a Devil's advocate approach ;)

I'll be more clear: User / Admin expects that there are no conflicts when applying patches (or that they are automatically resolved). Obviously, the implementation itself is kind of a "conflict" with insecure version of some software.

> > 2. Force zypper to resolve these conflicts automatically
> 
> Don't even think about using '--force-resolution'!
> ...
> The problem here is that just deleting packages resolves the conflict. Or
> the patch has to be skipped. But there is no heuristic telling which is ok.

Which is exactly my point, we can't do that automatically.

> If we are actually unable to inform the user that certain patches are not
> applicable and need his decision (and supervision) - we have a problem.

And that's why we have this bugreport.

> We could add an option to 'zypper patch' telling to auto-resolve conflicts
> by sipping a patch - if this is a resolution option. This would at least try
> to apply as many patches as possible. If patches were skipped zypper could
> return 
> ZYPPER_EXIT_INF_UPDATE_NEEDED(100). 

Looks like a nice feature request. But then we still don't have the reporting part. Anyway, it would already improve the security a lot.

Thinking about it ... there ARE ways to send e-mail from connadline, e.g., https://tecadmin.net/ways-to-send-email-from-linux-command-line/ but honestly, they just don't work unless you are sending e-mails to local users or unprotected mail domains. And nobody really reads root's e-mails.

> But it will not solve the basic issue. Needed patches will not be applied
> and the system might be vulnerable.

For that, the only real solution is a centralized way. SUSE Manager (there's also a free version available), Salt, ...
Comment 17 Lukas Ocilka 2022-08-19 08:31:33 UTC
See, e.g., https://github.com/uyuni-project/uyuni
Comment 18 Michal Filka 2022-08-24 07:52:33 UTC
After reading the discussion, I feel that the only solution is to stop pretending that we're able to silently patch the system and remove the script.

Where to discuss it first? SLE / OS?
Comment 19 Michael Andres 2022-08-25 12:06:30 UTC
(In reply to Michal Filka from comment #18)
> After reading the discussion, I feel that the only solution is to stop
> pretending that we're able to silently patch the system 

Yes

> and remove the script.

Or find some way to report the issue. 
And clearly state and document how to supervise it.

An icon on the e.g. desktop showing basically the `zypper pchk` result at the end of the script. It's expected to be 0 then. (maybe '--no-refresh pchk' so you are not fooled by patches released between your 'patch' and 'pchk'.

For non desktop users: The online_update script could store the result of it's latest 'pchk'. Some online_update_result script could show the result and return the code. 

> Last online_update on Aug 25 13:51:53 did not complete.
> `zypper pch` returned: 101
> 
>  Found 299 applicable patches:
>  Category    | Updatestack | Patches | Locked | Included categories
>  ------------+-------------+---------+--------+--------------------
>  security    | -           | 126     | 1      | 
>  recommended | 4           | 165     | -      | 
>  optional    | -           | 3       | -      | feature
> 
>  1 patch locked
>  3 patches optional   (use '--with-optional' to include optional patches)
>  295 patches needed (126 security patches)

or

> Last online_update on Aug 25 13:51:53 completed.
> `zypper pch` returned: 0
> 
>  0 patches needed (0 security patches)

If that's clearly communicated, even unattended systems can be monitored. A cron job sending mail if online_update_result return !=0. Or pulling the status the 'ssh user@remote.host online_update_result'.

The basic idea of the online_update script is not bad.
Comment 20 Lukas Ocilka 2022-09-26 16:05:20 UTC
Created attachment 861737 [details]
Screenshot how my firewall solves autoupdate issues

How? Reports them in UI. So unless someone gets in, it's blocked.
And this is auto-update with "do it every day".
As the FW does not have my personal e-mail address, there is no way to tell me.