Bug 927750

Summary: Xen-tools loads non-existent kernel module 'xen-acpi-processor', causes fail of 'systemd-modules-load' at boot
Product: [openSUSE] openSUSE Distribution Reporter: Forgotten User ev5s19PAVW <forgotten_ev5s19PAVW>
Component: XenAssignee: systemd maintainers <systemd-maintainers>
Status: RESOLVED WORKSFORME QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: ohering
Version: 13.2   
Target Milestone: ---   
Hardware: All   
OS: openSUSE 13.2   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Forgotten User ev5s19PAVW 2015-04-19 15:21:22 UTC
Booting to

	uname -r
		3.19.4-1.g74c332b-xen

Systemd service fails at 'Load Kernel Modules'

	dmesg | grep -i modules
		[    0.144000] Modules linked in:
		[    4.357642] systemd[1]: Failed to start Load Kernel Modules.
		[   25.284199] systemd-modules-load[779]: Inserted module 'pciback'
		[   35.284122] systemd-modules-load[779]: Failed to find module 'xen-acpi-processor'
		[   45.312110] systemd-modules-load[779]: Inserted module 'blktap2'
		[   47.972022] systemd[1]: Failed to start Load Kernel Modules.

	systemctl status systemd-modules-load.service
		systemd-modules-load.service - Load Kernel Modules
		   Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static)
		   Active: failed (Result: exit-code) since Sat 2015-04-18 14:59:05 PDT; 27min ago
		     Docs: man:systemd-modules-load.service(8)
		           man:modules-load.d(5)
		  Process: 779 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
		 Main PID: 779 (code=exited, status=1/FAILURE)

The module is not loaded after boot

	lsmod | grep acpi
		acpi_pad               12960  0 

Can't be loaded by the given name

	modprobe xen-acpi-processor
		modprobe: FATAL: Module xen-acpi-processor not found.

Doesn't exist as namrd

	locate xen | grep acpi | grep proc | grep 3.19.4
		/lib/modules/3.19.4-1.g74c332b-xen/kernel/drivers/acpi/processor.ko

And isn't mentioned (as CONFIG_XEN_ACPI_PROCESSOR) in the kernel config

	grep PROCESSOR  /boot/config-3.19.4-1.g74c332b-xen 
		# CONFIG_PROCESSOR_SELECT is not set
		CONFIG_ACPI_PROCESSOR=m
		CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
		CONFIG_PROCESSOR_EXTERNAL_CONTROL=y

On this server PM is available

	xenpm get-cpufreq-states 
		cpu id               : 0
		total P-states       : 16
		usable P-states      : 16
		current frequency    : 800 MHz
		P0         [3101 MHz]: transition [                 314]
		                       residency  [               13730 ms]
		...

	xenpm get-cpufreq-para 
		cpu id               : 0
		affected_cpus        : 0
		cpuinfo frequency    : max [3101000] min [800000] cur [3100000]
		scaling_driver       : acpi-cpufreq
		scaling_avail_gov    : userspace performance powersave ondemand
		current_governor     : ondemand
		  ondemand specific  :
		    sampling_rate    : max [10000000] min [10000] cur [20000]
		    up_threshold     : 80
		scaling_avail_freq   : 3101000 3100000 2900000 2800000 2600000 2400000 2300000 2100000 1900000 1800000 1600000 1500000 1300000 1100000 1000000 *800000
		scaling frequency    : max [3101000] min [800000] cur [800000]
		turbo mode           : enabled
		...

Checking what loads the module

	cat /usr/lib/systemd/system/systemd-modules-load.service
		#  This file is part of systemd.
		#
		#  systemd is free software; you can redistribute it and/or modify it
		#  under the terms of the GNU Lesser General Public License as published by
		#  the Free Software Foundation; either version 2.1 of the License, or
		#  (at your option) any later version.

		[Unit]
		Description=Load Kernel Modules
		Documentation=man:systemd-modules-load.service(8) man:modules-load.d(5)
		DefaultDependencies=no
		Conflicts=shutdown.target
		After=systemd-readahead-collect.service systemd-readahead-replay.service
		Before=sysinit.target shutdown.target
		ConditionCapability=CAP_SYS_MODULE
		ConditionDirectoryNotEmpty=|/lib/modules-load.d
		ConditionDirectoryNotEmpty=|/usr/lib/modules-load.d
		ConditionDirectoryNotEmpty=|/usr/local/lib/modules-load.d
		ConditionDirectoryNotEmpty=|/etc/modules-load.d
		ConditionDirectoryNotEmpty=|/run/modules-load.d
		ConditionKernelCommandLine=|modules-load
		ConditionKernelCommandLine=|rd.modules-load

		[Service]
		Type=oneshot
		RemainAfterExit=yes
		ExecStart=/usr/lib/systemd/systemd-modules-load

The only source is only '/usr/lib/modules-load.d/xen.conf'

	grep -rlni acpi /lib/modules-load.d /usr/lib/modules-load.d /usr/local/lib/modules-load.d /etc/modules-load.d /run/modules-load.d `which modules-load` `which rd.modules-load`
		/usr/lib/modules-load.d/xen.conf

	grep acpi /usr/lib/modules-load.d/xen.conf
		xen-acpi-processor

Provided by 'xen-tools' pkg

	rpm -q --whatprovides /usr/lib/modules-load.d/xen.conf
		xen-tools-4.5.0_03-366.5.x86_64

Not clear to me if the fix should be in xen-tools, the missing module, or systemd's loader.
Comment 1 Olaf Hering 2015-04-20 15:03:19 UTC
Three possible solutions:

1) ignore any errors from such badly implemented systemd-modules-load.service
2) adjust systemd-modules-load.service to report no such noise
3) adjust all provides of *.conf files to not rely on that .service
Comment 2 Dr. Werner Fink 2015-04-21 07:16:33 UTC
(In reply to Olaf Hering from comment #1)

> 1) ignore any errors from such badly implemented systemd-modules-load.service
> 2) adjust systemd-modules-load.service to report no such noise
> 3) adjust all provides of *.conf files to not rely on that .service

4) Correct the broken configuration!

Instead of bashing systemd for its features I'd like to know *what* is wrong with the implemention and *how* change this?

If there is a wrong configuration this does not belong to systemd, that is
and AFAICS the binary /usr/lib/systemd/systemd-modules-load called in /usr/lib/systemd/system/systemd-modules-load.service goes through the configurations

 /lib/modules-load.d
 /usr/lib/modules-load.d
 /usr/local/lib/modules-load.d
 /etc/modules-load.d
 /run/modules-load.d

and does its job.  IMHO this is fully correct ad indeed depends on the configuration.   But this configuration has *nothing* to do with systemd its self.

See man:systemd-modules-load(8)

       systemd-modules-load.service is an early-boot service that loads
       kernel modules from static configuration.

See man:modules-load.d(5)

       modules-load.d - Configure kernel modules to load at boot
Comment 3 Olaf Hering 2015-04-21 07:26:15 UTC
(In reply to Dr. Werner Fink from comment #2)
> (In reply to Olaf Hering from comment #1)
> 
> > 1) ignore any errors from such badly implemented systemd-modules-load.service
> > 2) adjust systemd-modules-load.service to report no such noise
> > 3) adjust all provides of *.conf files to not rely on that .service
> 
> 4) Correct the broken configuration!

It is not broken. Just look at it from this perspective:

A third party app (any non-systemd|udev package) which provides drivers that can not be autoloaded has to place a X.conf into the relevant directories. Just to support many possible kernel configurations. Since the names of the drivers differ across kernel releases there have to be more names listed that a given kernel can provide. And such unavailable drivers will cause the reported failures.

That service provides zero conditionals to possibly reduce the number of reported failures. 

Of course I can implement the conditionals myself, but then the service is not needed. Not sure what purpose the service serves anyway if it cant be used in a generic fashion. 

To solve that for our distro I suggest to go for #2: make it a dumb slave, it has no rights to report anything.
Comment 4 Dr. Werner Fink 2015-04-21 08:04:32 UTC
(In reply to Olaf Hering from comment #3)

Sorry but I do not buy this: If you do not want that systemd-modules-load.service does load a module at boot then do not specify this module in
anyone of the configuration directories below /etc/modules-load.d/ or /usr/lib/modules-load.d/.  This has *nothing* to do with modprobe and the /lib/modprobe.d/ and /etc/modprobe.d/ directories!

The bug is /usr/lib/modules-load.d/xen.conf
Comment 5 Forgotten User ev5s19PAVW 2015-04-22 03:01:39 UTC
Upgrade to

	rpm -qa | grep ^xen-4
		xen-4.5.0_03-367.1.x86_64

	rpm -q --changelog xen-4.5.0_03-367.1.x86_64 | head
		* Tue Apr 21 2015 ohering@suse.de
>>		- bnc#927750 - Avoid errors reported by system-modules-load.service

		* Wed Apr 08 2015 rguenther@suse.com
		- Add xen-no-array-bounds.patch and blktap-no-uninit.patch to selectively
		  turn errors back to warnings to fix build with GCC 5.
		- Amend xen.stubdom.newlib.patch to pull in declaration of strcmp to
		  avoid implicit-fortify-decl rpmlint error.
		- Fix quoting of __SMBIOS_DATE__ in xen.build-compare.smbiosdate.patch.

After reboot

	journalctl -b | egrep -i "Load Kern|modules-load"
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'xen-evtchn'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'xen-gntalloc'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'xen-blkback'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'xen-netback'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'xen-pciback'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Inserted module 'evtchn'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Inserted module 'gntdev'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'netbk'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'blkbk'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'xen-scsibk'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'usbbk'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'pciback'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'xen-acpi-processor'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'blktap2'
		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module 'blktap'
		Apr 21 19:49:43 xen01 systemd[1]: Failed to start Load Kernel Modules.

	systemctl status systemd-modules-load
		systemd-modules-load.service - Load Kernel Modules
		   Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static)
		   Active: active (exited) since Tue 2015-04-21 19:49:54 PDT; 9min ago
		     Docs: man:systemd-modules-load.service(8)
		           man:modules-load.d(5)
		  Process: 773 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=0/SUCCESS)
		 Main PID: 773 (code=exited, status=0/SUCCESS)
		   CGroup: /system.slice/systemd-modules-load.service
Comment 6 Dr. Werner Fink 2015-04-22 06:53:43 UTC
(In reply to lynda t from comment #5)

You may run, e.g.

   find /lib/modules/$(uname -r) -name 'xen-evtchn*'

to see if the module(s) (here xen-evtchn) is installed at all.

The systemd-modules-load.service of systemd is the same feature of loading modules from a static configuration as in the former SysVinit based boot scheme with the MODULES_LOADED_ON_BOOT variable in /etc/sysconfig/kernel and the script /etc/init.d/boot.loadmodules.

Normally the almost all modules will be loaded on demand (e.g. by udev) and also by dependencies (by modprobe) and not from a static configuration as the systemd-modules-load.service or MODULES_LOADED_ON_BOOT is.
Comment 7 Forgotten User ev5s19PAVW 2015-04-22 07:13:16 UTC
(In reply to Werner Fink from comment #6)
> (In reply to lynda t from comment #5)
> 
> You may run, e.g.
> 
>    find /lib/modules/$(uname -r) -name 'xen-evtchn*'
> 
> to see if the module(s) (here xen-evtchn) is installed at all.
> 
> The systemd-modules-load.service of systemd is the same feature of loading
> modules from a static configuration as in the former SysVinit based boot
> scheme with the MODULES_LOADED_ON_BOOT variable in /etc/sysconfig/kernel and
> the script /etc/init.d/boot.loadmodules.
> 
> Normally the almost all modules will be loaded on demand (e.g. by udev) and
> also by dependencies (by modprobe) and not from a static configuration as
> the systemd-modules-load.service or MODULES_LOADED_ON_BOOT is.

find /lib/modules/$(uname -r) -name 'xen-evtchn*'
    (null)

but,

find /lib/modules/$(uname -r) -name '*evtchn*'
    /lib/modules/3.19.4-1.g74c332b-xen/kernel/drivers/xen/evtchn.ko

lsmod | grep evtchn
    evtchn                 13033  1 

I'd guess that this may be due to module name changes with new pvops kernels, vs opensuse's legacy kernel-xen?
Comment 8 Dr. Werner Fink 2015-04-22 07:24:54 UTC
(In reply to lynda t from comment #7)

Hmmm .... IMHO any xen.conf below modules-load.d diretory should be part of the specific package which includes the module.

The old MODULES_LOADED_ON_BOOT SysVinit way had the disadvantage that not existing modules are ignored, that is that a system adminstrator had never realized that a module name has changed due to an update.  Now with systemd and the modular modules-load.d there are error messages on such changes.  One reason more that static module configuration should be part of the specific package.

Btw: This bug report was for openSUSE 13.2, I was not aware that pvops kernels are part of the 13.2.
Comment 9 Olaf Hering 2015-04-22 07:44:34 UTC
(In reply to lynda t from comment #5)
> 		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module
> 'xen-evtchn'

This is from initrd. Try to regenerate it and they should be gone.


Try to understand what paragraph 2 of comment #3 means.
xen.conf used to contain all possible modules of all possible kernels. Now they are all loaded manually, and errors are ignored because errors from modprobe are not relevant.
Comment 10 Forgotten User ev5s19PAVW 2015-04-22 13:10:56 UTC
> Btw: This bug report was for openSUSE 13.2, I was not aware that pvops kernels 
> are part of the 13.2.

It is against 13.2, with kernel from KernelStable.  In either case, IIUC, there's NO pvops kernels -- still the -legacy.

I was just making an unsubstantiated guess that some presumptions may have been made from the Xen-pkgs side ...

(In reply to Olaf Hering from comment #9)
> (In reply to lynda t from comment #5)
> > 		Apr 21 19:49:43 xen01 systemd-modules-load[124]: Failed to find module
> > 'xen-evtchn'
> 
> This is from initrd. Try to regenerate it and they should be gone.


Confirmed.  

dmesg | egrep -i "modules|load kern"
[    0.144000] Modules linked in:
[    4.774432] systemd[1]: Failed to start Load Kernel Modules.
[   14.821837] systemd[1]: Found dependency on xen-dom0-modules.service/start

Still the "failed to start" message ...