Bug 893659

Summary: Patch for apache2-2.4.6-6.27.1 restarts before updating MPM to same version
Product: [openSUSE] openSUSE 13.1 Reporter: James Carter <jimc>
Component: ApacheAssignee: E-mail List <apache-bugs>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P3 - Medium CC: astieger, bruno, fcrozat, maint-coord, pgajdos, security-team
Version: Final   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE 13.1   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description James Carter 2014-08-26 19:47:14 UTC
In the recent patch updating Apache2-2.4.6 from build ID 6.23.1 to 6.27.1, apparently it restarts apache immediately, and it fails with this error message (this is with apache2-prefork-2.4.6-6.23.1.x86_64 as the MPM):

Cannot load /usr/lib64/apache2-prefork/mod_proxy.so into server: 
    /usr/lib64/apache2-prefork/mod_proxy.so : 
    undefined symbol: ap_parse_token_list_strict

That library belongs to apache2-prefork-2.4.6-6.23.1.x86_64 which evidently did not get updated until after the restart.  Workarounds:

On most machines with webservers, just restart Apache again and it will be fine, because the MPM does eventually get updated.  However, on my main webserver I host an enterprise mirror of the SuSE distro, and after Apache fails to restart, Zypper cannot download any more packages including the MPM that it's going to need to restart itself.  One fix that would have avoided the problem: if I had used --download-in-advance rather than --download-as-needed, the MPM would already have been downloaded and could be installed.  

I just did rpm -U /s1/SuSE/SuSE/update/13.1/x86_64/apache2-prefork-2.4.6-6.27.1.x86_64.rpm (taking it directly from the enterprise mirror) and restarted Apache.  You can't use "zypper install /path/to.rpm" because it checks the repo status anyway (there might be dependencies), which fails because the webserver cannot be started.  

I imagine that few people will encounter this problem, but if there were some way to defer the restart until all patches were installed, that would help a lot.
Comment 1 Petr Gajdos 2015-10-16 11:52:15 UTC
I guess that original problem persists in Factory, too.

If I understand correctly, the issue is caused by calling httpd restart in wrong time (%postun of apache2). 

apache2-prefork requires apache2 directly and apache2 requires apache2-prefork indirectly trough apache2-MPM symbol (similarly for apache2-worker and apache2-event MPM packages). If apache2 gets installed first, then its %postun is called before the update of apache2-worker, which can lead to various errors.

How can we ensure the restart of the apache will happen after update of all apache2, apache2-event, apache2-prefork, apache2-worker?

instances
=========
More over, for Factory, we introduced instances. That means, on the system can be active several instances run with systemctl start apache2<instance>. How do we restart them all automatically during rpm update?

Of course, we can control individual apache instances manually trough /var/run/httpd<instance>.pid pid number, but isn't there a better way?
Comment 2 Bruno Friedmann 2015-10-17 07:57:04 UTC
I will just add my own experience: there's also the case of "linked" languages or modules for example python, php, and I guess (cause never tried) ruby too.

We need to have only one restart trigger for the whole stack.
But I honestly have no idea of what is the best universal solution.

For myself the no-recomment download-in-advance have been the rule for my installation. 
But still I'm seeing package trying to be smarted than admin like php-fpm which restart after update, but there's still 20 other php module to update, forcing the admin the restart again at the end. Same for libvirtd etc ...

Please, when we do update as admin. we know we will have daemon or process to restart, but first we will be able to ajust config file, and apply the restart at best time.

The ideal tool would allow package to write down what need to be done after install, to a global place. Once everything is done, zypper execute the bunch of command, after sorting and uniq them. (Nobody want to see its apache2 instance restarted 3 or 4 times).
Comment 3 Petr Gajdos 2015-11-02 14:48:23 UTC
(In reply to Petr Gajdos from comment #1)
> instances
> =========

This should be achieved via apache2@.service restart, and this is already done.
Comment 4 Petr Gajdos 2015-11-02 14:56:00 UTC
(In reply to Bruno Friedmann from comment #2)
> The ideal tool would allow package to write down what need to be done after
> install, to a global place. Once everything is done, zypper execute the
> bunch of command, after sorting and uniq them. (Nobody want to see its
> apache2 instance restarted 3 or 4 times).

Even if not so straitforwardly, it is already implemented in zypper and used for a while. For example, there was a need to make this true for font packages (after SuSEconfig.fonts was dropped) to not run fonts-config several times after each font package install or update, but only once after the whole transaction.

Let me think about it ..
Comment 5 Petr Gajdos 2015-11-10 12:43:38 UTC
I have divided the problem into two steps:

(1) ensure that httpd is restarted after all of apache2{,-worker,-prefork,-event} update (should solve comment 0)

This is already done so in home:pgajdos/apache2.

I will explain how, to be more clear in point (2). Each of apache2{,-worker,-prefork,-event}:
  a. immediately after package update: sets a flag (touches
     /var/run/httpd.restart.flag)
  b. after whole zypper or rpm transaction: restarts apache if 
     /var/run/httpd.restart.flag exists and removes the flag (therefore restart
     runs only once)

(2) create rpm macros which will allow other packages to order apache restart after its installation or update (should solve comment 2)

How I plan to do: the work flow would be similar to point (1). module could:
  a. immediately after package transaction (install/update):     
     request apache restart setting by touching /var/run/httpd.restart.flag
  b. after whole zypper/rpm transaction: it would restart apache if 
     /var/run/httpd.restart.flag exists and remove the flag in that case

To lower unneeded apache restarts, we can figure out which instances are running, figure out in which instances use the module and then restart only these instances.

While (1) and (2) will share same flag (/var/run/httpd.restart.flag), in case that apache2 gets updated together with the module gets installation/update in one zypper or rpm transaction, the restart happens only once.

_When_ the module requests the package restart (its installation, update or both) depends on the module and its packager.

Unfortunately, to my knowledge, there is no similar way to delay apache restart for module uninstall. So packager of the module is left to not restart apache after module uninstall at all or restart it immediately after package uninstall even if that could lead to more restarts in one zypper or rpm transaction.

Does that sound correct?
Comment 6 Bruno Friedmann 2015-11-10 12:53:36 UTC
Hi Petr!

A big +1 from my side. 

Still it would be nice to include here (once done) some line of comment for packager that want to achieve this correctly. Even if it's quite easy to understand. So it can serve as reference too.
Comment 7 Petr Gajdos 2015-11-12 08:05:22 UTC
(In reply to Petr Gajdos from comment #5)
> While (1) and (2) will share same flag (/var/run/httpd.restart.flag), in

One little correction, we need 'absolute' restart flag (/var/run/httpd.restart.flag), which will trigger restart all instances and then 'instance' restart flags (/var/run/httpd.restart.flag@<instancename>) which will run restart of particular instance. When /var/run/httpd.restart.flag is used for overall restart, instance flags are removed too, so there should be again at most one restart for each instance.

Whole thing is submitted for no into Apache project (involved packages: apache2 and new apache-rpm-macros-control) and two modules from Apache:Modules was experimentally involved too (apache2-mod_bmx, apache2-mod_bw).
Comment 8 Petr Gajdos 2015-11-12 08:06:37 UTC
s/for no/for now/
Comment 14 Petr Gajdos 2016-01-11 14:10:47 UTC
Fixed in Factory, will be fixed in Leap 42.2.