Bugzilla – Bug 1215282
systemd-vconsole-setup fails on a gnome scenario
Last modified: 2024-03-08 08:51:50 UTC
## Observation openQA test in scenario opensuse-Tumbleweed-DVD-aarch64-system_performance@aarch64 fails in [consoletest_setup](https://openqa.opensuse.org/tests/3568610/modules/consoletest_setup/steps/52) consoletest_setup fails on gnome scenario since aarch64 Tumbleweed snapshot 20230911. The problem is we have broken fonts: https://openqa.opensuse.org/tests/3568601#step/consoletest_setup/54 So, '/usr/lib/systemd/systemd-vconsole-setup' is called (and returns 0), but prints some errors: https://openqa.opensuse.org/tests/3568601#step/consoletest_setup/55 ********** Sep 12 02:25:59.709634 localhost systemd-vconsole-setup[376]: KD_FONT_OP_GET failed while trying to get the font metadata: Invalid argument Sep 12 02:25:59.709640 localhost systemd-vconsole-setup[376]: Fonts will not be copied to remaining consoles ********** And does not display new fonts as expected by openQA: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/utils.pm#L436 ## Test suite description Maintainer: QE Core An explicit system performance test. https://progress.opensuse.org/issues/36100 ## Reproducible Fails since (at least) Build [20230911](https://openqa.opensuse.org/tests/3568153) ## Expected result Last good: [20230910](https://openqa.opensuse.org/tests/3565572) (or more recent) ## Further details Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=aarch64&distri=opensuse&flavor=DVD&machine=aarch64&test=system_performance&version=Tumbleweed)
Created attachment 869470 [details] Main changes in the broken snapshot
Also, please note that sometimes the test runs just fine.
Any idea how to move forward on this error?
Why has this been assigned to me? I doubt this is related to console-setup, which I'm maintainer of for historical reasons (no relevant changes since the update to 1.215 on Fri Dec 30 12:16:39 UTC 2022.
"consoletest_setup" != "console-setup package" ...
Yes, this is not console-setup package, I reassign it to screening team and will investigate how to assign it to the correct bugowner, apologize and thanks for your remainder.
*** Bug 1215283 has been marked as a duplicate of this bug. ***
Can we get access to a system where the problem occurs ?
This seems not to be limited to ARM only - we see the same error on x86_64 using the live images: https://openqa.opensuse.org/tests/3769667#step/consoletest_setup/46 That might make it easier to reproduce
Finally reproduced it locally after downloading the iso. The problem seems due to plymouth that leaves tty1 in "graphics" mode when the boot process is finished and systemd-vconsole-setup doesn't handle this case very well. It looks like plymouth doesn't behave consistently when it exits because unlike when the test is done via libvirtd the problem disappears when libvirtd is replaced by a plain qemu. I also noticed a third issue: getty@tty1.service fails to start a login prompt when tty1 is left in graphic mode. I'll look into the 2 issues related to systemd but the remaining one about plymouth should be looked at by someone who knows plymouth. Cliff Zhao, maybe you can explain this inconsistent behavior of plymouth ?
BTW I also noticed another suspicious thing when looking at the systemd debug logs: > [ 14.701685] localhost systemd[1]: YaST2-Second-Stage.service: Looking at job YaST2-Second-Stage.service/start conflicted_by=no > [ 14.701706] localhost systemd[1]: YaST2-Second-Stage.service: Looking at job YaST2-Second-Stage.service/stop conflicted_by=no > [ 14.701730] localhost systemd[1]: YaST2-Second-Stage.service: Fixing conflicting jobs YaST2-Second-Stage.service/start,YaST2-Second-Stage.service/stop by deleting job YaST2-Second-Stage.service/stop > [ 14.701812] localhost systemd[1]: plymouth-start.service: Looking at job plymouth-start.service/stop conflicted_by=yes > [ 14.701833] localhost systemd[1]: plymouth-start.service: Looking at job plymouth-start.service/start conflicted_by=no > [ 14.701853] localhost systemd[1]: plymouth-start.service: Fixing conflicting jobs plymouth-start.service/stop,plymouth-start.service/start by deleting job plymouth-start.service/start So basically YaST2-Second-Stage.service conflicts with plymouth-start.service. That means that if for any reason plymouth is not included in initrd then plymouth will stop working even if YaST2-Second-Stage.service isn't needed. Currently this conflicts has no side effect due to the fact that plymouthd started from within initrd survives from the initrd -> host transition but I guess it would be better if YaST uses a different approach. For example replacing the conflict with `ExecStartPre=-/usr/bin/plymouth --wait quit` might do the trick.
And if plymouth is supposed to always be started from within initrd then not including it in the host initial transaction might be a good idea.
Franck, we always fight with systemd dependencies with every release. You probably remember that ;) It was not a problem so far and as it seems systemd was able to live with that. Is this 100% secure and will produce no other issues? Or is it safer to keep it complaining, but being sure it still works the same?
(In reply to Lukas Ocilka from comment #13) > Franck, we always fight with systemd dependencies with every release. You > probably remember that ;) How can I forget ? ;) > It was not a problem so far and as it seems systemd was able to live with > that. Is this 100% secure and will produce no other issues? Or is it safer > to keep it complaining, but being sure it still works the same? The thing is that I don't see how the conflict works. It probably does otherwise plymouth would interfere with yast2 second stage service badly. But as described previously, plymouth is started from within initrd and this instance is the one that is used during the whole boot process. The fact that plymouth-start.service is not started again (because yast2 second stage conflicts with it) after the switch root transition should have no effect it seems.
Trying some software archaeology... https://github.com/yast/yast-installation/blob/master/package/yast2-installation.changes#L1820-L1830 > ------------------------------------------------------------------- > Thu Jan 18 12:48:55 UTC 2018 - qzhao@suse.com > . > - Update YaST2-Firstboot.service: Depreciate plymouth --wait, add > conflict to plymouth start service. > - Update YaST2-Second-Stage.service: Depreciate plymouth deactivate > command and add conflict to plymouth start service. For that will > block TTY device activation(bsc#1042554). > - 4.0.23 > . > ------------------------------------------------------------------- https://github.com/yast/yast-installation/blob/master/package/yast2-installation.changes#L2896-L2903 > ------------------------------------------------------------------- > Fri Feb 26 08:36:55 UTC 2016 - ancor@suse.com > . > - Ensure plymouth does not interfere with X11 when executing > yast2-firstboot (bsc#966874) > - 3.1.170 > . > -------------------------------------------------------------------
The PR for the 01/2018 change log entry: https://github.com/yast/yast-installation/pull/642 based on: https://github.com/yast/yast-installation/pull/584 Not much explanation what it does and why, sadly.
(In reply to Stefan Hundhammer from comment #16) > Not much explanation what it does and why, sadly. That's a pity. Maybe I'm missing something but I currently don't see how this works.
I've tried to reproduce the issue with a branched image from: https://build.opensuse.org/package/show/openSUSE:Factory:Live/livecd-tumbleweed-gnome But the issue doesn't happen therewith. Still it is present in: https://openqa.opensuse.org/tests/3787994 Has anyone an idea what is the difference between image from the openSUSE:Factory:Live project and the openQA image?
(In reply to Thomas Blume from comment #18) > Has anyone an idea what is the difference between image from the > openSUSE:Factory:Live project and the openQA image? There is no difference: the image from openSUSE:Factory:Live is copied to openSUSE:Factory:ToTest when the snapshot is completed and heading for openQA (copied, not rebuilt); and from there then sync'ed to openQA for testing.
So back to the initial issue, I have a patch [1] that improves systemd-vconsole-setup behavior when tty1 is in graphics mode. The reason why /dev/tty1 left in graphics mode is still unknown but at least systemd-vconsole-setup should handle the case more gracefully. Is there any way to check that the patch really helps before I submit it ? [1] https://github.com/systemd/systemd/commit/58161db61ba933f1e3e7395d1f50fe0213bb82f2
(In reply to Franck Bui from comment #11) > BTW I also noticed another suspicious thing when looking at the systemd > debug logs: > > > [ 14.701685] localhost systemd[1]: YaST2-Second-Stage.service: Looking at job YaST2-Second-Stage.service/start conflicted_by=no > > [ 14.701706] localhost systemd[1]: YaST2-Second-Stage.service: Looking at job YaST2-Second-Stage.service/stop conflicted_by=no > > [ 14.701730] localhost systemd[1]: YaST2-Second-Stage.service: Fixing conflicting jobs YaST2-Second-Stage.service/start,YaST2-Second-Stage.service/stop by deleting job YaST2-Second-Stage.service/stop > > [ 14.701812] localhost systemd[1]: plymouth-start.service: Looking at job plymouth-start.service/stop conflicted_by=yes > > [ 14.701833] localhost systemd[1]: plymouth-start.service: Looking at job plymouth-start.service/start conflicted_by=no > > [ 14.701853] localhost systemd[1]: plymouth-start.service: Fixing conflicting jobs plymouth-start.service/stop,plymouth-start.service/start by deleting job plymouth-start.service/start > > So basically YaST2-Second-Stage.service conflicts with > plymouth-start.service. That means that if for any reason plymouth is not > included in initrd then plymouth will stop working even if > YaST2-Second-Stage.service isn't needed. Currently this conflicts has no > side effect due to the fact that plymouthd started from within initrd > survives from the initrd -> host transition but I guess it would be better > if YaST uses a different approach. > > For example replacing the conflict with `ExecStartPre=-/usr/bin/plymouth > --wait quit` might do the trick. This has already been submitted. it is in the review of Yast team. [1] https://github.com/yast/yast-installation/pull/1070/files
(In reply to Cliff Zhao from comment #21) > (In reply to Franck Bui from comment #11) > > BTW I also noticed another suspicious thing when looking at the systemd > > debug logs: > > > > > [ 14.701685] localhost systemd[1]: YaST2-Second-Stage.service: Looking at job YaST2-Second-Stage.service/start conflicted_by=no > > > [ 14.701706] localhost systemd[1]: YaST2-Second-Stage.service: Looking at job YaST2-Second-Stage.service/stop conflicted_by=no > > > [ 14.701730] localhost systemd[1]: YaST2-Second-Stage.service: Fixing conflicting jobs YaST2-Second-Stage.service/start,YaST2-Second-Stage.service/stop by deleting job YaST2-Second-Stage.service/stop > > > [ 14.701812] localhost systemd[1]: plymouth-start.service: Looking at job plymouth-start.service/stop conflicted_by=yes > > > [ 14.701833] localhost systemd[1]: plymouth-start.service: Looking at job plymouth-start.service/start conflicted_by=no > > > [ 14.701853] localhost systemd[1]: plymouth-start.service: Fixing conflicting jobs plymouth-start.service/stop,plymouth-start.service/start by deleting job plymouth-start.service/start > > > > So basically YaST2-Second-Stage.service conflicts with > > plymouth-start.service. That means that if for any reason plymouth is not > > included in initrd then plymouth will stop working even if > > YaST2-Second-Stage.service isn't needed. Currently this conflicts has no > > side effect due to the fact that plymouthd started from within initrd > > survives from the initrd -> host transition but I guess it would be better > > if YaST uses a different approach. > > > > For example replacing the conflict with `ExecStartPre=-/usr/bin/plymouth > > --wait quit` might do the trick. > > This has already been submitted. it is in the review of Yast team. > > [1] https://github.com/yast/yast-installation/pull/1070/files I mean the "YaST2-Second-Stage.service" part of the request.
(In reply to Franck Bui from comment #10) > Finally reproduced it locally after downloading the iso. > > The problem seems due to plymouth that leaves tty1 in "graphics" mode when > the boot process is finished and systemd-vconsole-setup doesn't handle this > case very well. > > It looks like plymouth doesn't behave consistently when it exits because > unlike when the test is done via libvirtd the problem disappears when > libvirtd is replaced by a plain qemu. > > I also noticed a third issue: getty@tty1.service fails to start a login > prompt when tty1 is left in graphic mode. > > I'll look into the 2 issues related to systemd but the remaining one about > plymouth should be looked at by someone who knows plymouth. > > Cliff Zhao, maybe you can explain this inconsistent behavior of plymouth ? For this problem, I should take a look the environment. When you said that "plymouth that leaves tty1 in "graphics" mode when the boot process is finished", Do you mean that plymouth progress is quit, but the screen is still shows the booting animation?
(In reply to Cliff Zhao from comment #21) > This has already been submitted. it is in the review of Yast team. > > [1] https://github.com/yast/yast-installation/pull/1070/files I'm not following this PR actually does the opposite of what I suggested.
(In reply to Cliff Zhao from comment #23) > For this problem, I should take a look the environment. You can download the iso image of the live system. It's easy to reproduce if the iso is run with libvirtd. > When you said that "plymouth that leaves tty1 in "graphics" mode when the > boot process is finished", Do you mean that plymouth progress is quit, but > the screen is still shows the booting animation? I meant that after plymouth quits, /dev/tty1 is in KD_GRAPHICS mode (see ioctl_console(2) man page). I suspected plymouth because when booting with 'plymouth.enable=0' the problem disappears.
(In reply to Franck Bui from comment #24) > (In reply to Cliff Zhao from comment #21) > > This has already been submitted. it is in the review of Yast team. > > > > [1] https://github.com/yast/yast-installation/pull/1070/files > > I'm not following this PR actually does the opposite of what I suggested. Then, Since you said "For example replacing the conflict with `ExecStartPre=-/usr/bin/plymouth --wait quit` might do the trick.", Do you mean to launch "/usr/bin/plymouth --wait quit" before the YaST2-Second-Stage.service?
(In reply to Cliff Zhao from comment #26) > Then, Since you said "For example replacing the conflict with > `ExecStartPre=-/usr/bin/plymouth --wait quit` might do the trick.", Do you > mean to launch "/usr/bin/plymouth --wait quit" before the > YaST2-Second-Stage.service? I meant that YaST2-Second-Stage.service should have "ExecStartPre=-/usr/bin/plymouth quit" but without conflicting itself with plymouth-start.service. But this is getting out of topic as this bug is about systemd-vconsole misbehaving when /dev/tty1 is in KD_GRAPHICS mode. It's probably better to discuss this topic in a separate bug report or you can contact me directly.
(In reply to Franck Bui from comment #20) > https://github.com/systemd/systemd/commit/ > 58161db61ba933f1e3e7395d1f50fe0213bb82f2 I backported this commit that I believe will fix the reported issue. It will be available in the next update. Feel free to re-open if it isn't the case.
Fix submitted to Factory via sr#1134212, hence closing.
This is an autogenerated message for OBS integration: This bug (1215282) was mentioned in https://build.opensuse.org/request/show/1134212 Factory / systemd
Comment11 indeed uncovered a risk, I have submitted a request, and they are not the opposite. The fix(1134212) is not on the plymouth, but I will continue to forward the plymouth-related suggestion change to the Yast team.
*** Bug 1217971 has been marked as a duplicate of this bug. ***
*** Bug 1218710 has been marked as a duplicate of this bug. ***
This is an autogenerated message for OBS integration: This bug (1215282) was mentioned in https://build.opensuse.org/request/show/1152118 Factory / systemd
I installed Leap 15.6 Beta and when the system boots I get an error "failed to start virtual console setup" When I need to unlock the partition, the English layout works, but the German one is displayed. I changed the font in /etc/vconsole conf, but after reboot it stops working. Did I write here correctly?