Bug 1212101

Summary: Firefox 114.0 and 114.0.1 immediately crashes at startup.
Product: [openSUSE] openSUSE Tumbleweed Reporter: Paul Tannington <paul.pgp-7>
Component: FirefoxAssignee: Factory Mozilla <factory-mozilla>
Status: IN_PROGRESS --- QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P5 - None CC: Andreas.Stieger, aplanas, bingmybong, danilo.spinella, ed.davis, fstrba, Gerhard, giecrilj, gtx.swift, ilgaz, jan.hubicka, jayjayjazz, jslaby, kaykaykay123, kevin.legouguec, lsalzman, Markus.Elfring, martin.sirringhaus, mikeyw, mjambor, non.space.1, noreply.section+dev, paul.pgp-7, rguenther, sachse, stakanov, victorhck, wolfgang
Version: CurrentFlags: jslaby: needinfo?
lsalzman: SHIP_STOPPER?
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
See Also: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110334
https://bugzilla.opensuse.org/show_bug.cgi?id=1212610
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: save-temps
build log of firefox with LTO
Preprocessed SkOpts_avx

Description Paul Tannington 2023-06-07 11:31:07 UTC
Just updated Firefox 113.0.2 -> 114.0 (from https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/) on a TW system (20230605).

Firefox now consistently crashes at startup, no GUI is shown it just immediately brings up the Firefox Crash Reporter dialogue.

Tested with a new user and new default Firefox profile, same result, immediate crash.

FF 114.o is working OK on Leap 15.4, this appears restricted only to TW.
Comment 1 Wolfgang Rosenauer 2023-06-07 11:57:45 UTC
I'm running FF114 on TW from that repo and it's working fine.
Do you have more information?
Comment 2 Paul Tannington 2023-06-07 13:08:27 UTC
> Do you have more information?

Unsure of what other information, or where to obtain it, would be useful :(


The machine in question is quite elderly hardware:
Operating System: openSUSE Tumbleweed 20230605
KDE Plasma Version: 5.27.5
KDE Frameworks Version: 5.106.0
Qt Version: 5.15.9
Kernel Version: 6.3.4-1-default (64-bit)
Graphics Platform: X11
Processors: 2 × AMD Athlon(tm) 64 X2 Dual Core Processor 5600+
Memory: 3.8 GiB of RAM
Graphics Processor: NV84
Manufacturer: Gigabyte Technology Co., Ltd.
Product Name: GA-MA770-DS3

(I have a second machine which is hardware identical, also running TW20230605. Updating FF 113.0.2 -> 114.0 also results in the immediate crash.  Reverting both machines to FF 113.0.2 using a saved 113 profile and FF starts as normal.)


"journalctl -f" yields the following upon attempting to launch FF:

Jun 07 13:51:27 Orion-15.openSUSE systemd[1095]: Started Firefox - Web Browser.
Jun 07 13:51:28 Orion-15.openSUSE kmozillahelper[2148]: kf.i18n: KLocalizedString: Using an empty domain, fix the code. msgid: "Mozilla Firefox" msgid_plural: "" msgctxt: ""
Jun 07 13:51:29 Orion-15.openSUSE firefox[2111]: Theme parsing error: gtk-dark.css:1:50: Failed to import: Error opening file /usr/share/themes/Breeze-Dark/gtk-3.20/gtk.css: No such file or directory
Jun 07 13:51:29 Orion-15.openSUSE firefox[2111]: Theme parsing error: gtk-dark.css:1:50: Failed to import: Error opening file /usr/share/themes/Breeze-Dark/gtk-3.20/gtk.css: No such file or directory
Jun 07 13:51:29 Orion-15.openSUSE firefox[2111]: Theme parsing error: gtk.css:68:35: The style property GtkButton:child-displacement-x is deprecated and shouldn't be used anymore. It will be removed in a future version
Jun 07 13:51:29 Orion-15.openSUSE firefox[2111]: Theme parsing error: gtk.css:69:35: The style property GtkButton:child-displacement-y is deprecated and shouldn't be used anymore. It will be removed in a future version
Jun 07 13:51:29 Orion-15.openSUSE firefox[2111]: Theme parsing error: gtk.css:71:36: The style property GtkCheckMenuItem:indicator-size is deprecated and shouldn't be used anymore. It will be removed in a future version
Jun 07 13:51:29 Orion-15.openSUSE firefox[2111]: Theme parsing error: gtk.css:73:46: The style property GtkScrolledWindow:scrollbars-within-bevel is deprecated and shouldn't be used anymore. It will be removed in a future version
Jun 07 13:51:29 Orion-15.openSUSE firefox[2111]: Theme parsing error: gtk.css:76:30: The style property GtkExpander:expander-size is deprecated and shouldn't be used anymore. It will be removed in a future version
Jun 07 13:51:29 Orion-15.openSUSE plasmashell[2111]: ATTENTION: default value of option mesa_glthread overridden by environment.
Jun 07 13:51:29 Orion-15.openSUSE plasmashell[2111]: ATTENTION: default value of option mesa_glthread overridden by environment.
Jun 07 13:51:29 Orion-15.openSUSE plasmashell[2111]: ATTENTION: default value of option mesa_glthread overridden by environment.
Jun 07 13:51:29 Orion-15.openSUSE plasmashell[2111]: ExceptionHandler::GenerateDump cloned child 2174
Jun 07 13:51:29 Orion-15.openSUSE plasmashell[2111]: ExceptionHandler::SendContinueSignalToChild sent continue signal to child
Jun 07 13:51:29 Orion-15.openSUSE plasmashell[2174]: ExceptionHandler::WaitForContinueSignal waiting for continue signal...
Jun 07 13:51:30 Orion-15.openSUSE plasmashell[1221]: file:///usr/share/plasma/plasmoids/org.kde.plasma.taskmanager/contents/ui/Task.qml:286: Unable to assign [undefined] to QString
Jun 07 13:51:38 Orion-15.openSUSE systemd[1095]: app-firefox-e557337bebaa4a8283013fac9b8f22d0.scope: Consumed 2.750s CPU time.


What other information would be useful?
Comment 3 hui 2023-06-08 16:50:06 UTC
Basics like:
- output when startet from terminal?
- Troubleshooting mode? https://support.mozilla.org/en-US/kb/diagnose-firefox-issues-using-troubleshoot-mode
Comment 4 Paul Tannington 2023-06-08 17:50:03 UTC
(In reply to hui from comment #3)
> Basics like:
> - output when startet from terminal?

Provides nothing over and above that shown in comment #2 from "journalctl -f" and starting FF

paul@Orion-15:~$ firefox

(firefox:1455): Gtk-WARNING **: 18:19:56.281: Theme parsing error: gtk-dark.css:1:50: Failed to import: Error opening file /usr/share/themes/Breeze-Dark/gtk-3.20/gtk.css: No such file or directory

(firefox:1455): Gtk-WARNING **: 18:19:56.346: Theme parsing error: gtk-dark.css:1:50: Failed to import: Error opening file /usr/share/themes/Breeze-Dark/gtk-3.20/gtk.css: No such file or directory

(firefox:1455): Gtk-WARNING **: 18:19:56.350: Theme parsing error: gtk.css:68:35: The style property GtkButton:child-displacement-x is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1455): Gtk-WARNING **: 18:19:56.351: Theme parsing error: gtk.css:69:35: The style property GtkButton:child-displacement-y is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1455): Gtk-WARNING **: 18:19:56.351: Theme parsing error: gtk.css:71:36: The style property GtkCheckMenuItem:indicator-size is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1455): Gtk-WARNING **: 18:19:56.351: Theme parsing error: gtk.css:73:46: The style property GtkScrolledWindow:scrollbars-within-bevel is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1455): Gtk-WARNING **: 18:19:56.351: Theme parsing error: gtk.css:76:30: The style property GtkExpander:expander-size is deprecated and shouldn't be used anymore. It will be removed in a future version
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
ExceptionHandler::GenerateDump cloned child ExceptionHandler::WaitForContinueSignal waiting for continue signal...
1506
ExceptionHandler::SendContinueSignalToChild sent continue signal to child
paul@Orion-15:~$

I think the Gtk warnings about "No such file or directory" are rather a red herring, as the referenced file is not present on my Leap 15.4 systems on which FF 114 is working without this problem.


> - Troubleshooting mode?
> https://support.mozilla.org/en-US/kb/diagnose-firefox-issues-using-
> troubleshoot-mode

As I wrote in the initial post, FF is crashing before it even displays it's GUI, it goes straight to the "Mozilla Crash Reporter" dialogue.

Starting FF from the command line with "firefox -safe-mode" also immediately crashes.

The "real" information of use is probably the crash dump file produced which is submitted to Mozilla via the "Mozilla Crash Reporter" - it's an approx 500k binary file...
Comment 5 Gerhard Maier 2023-06-08 18:20:40 UTC
Same here... after Update 113 -> 114 Firefox doesn't work. Go back to 113 helps
Comment 6 Wolfgang Rosenauer 2023-06-09 05:30:14 UTC
(In reply to Gerhard  Maier from comment #5)
> Same here... after Update 113 -> 114 Firefox doesn't work. Go back to 113
> helps

Is this also KDE? Or is it nvidia (assuming that the reporter has nvidia (NV84)?
Comment 7 Gerhard Maier 2023-06-09 06:14:19 UTC
(In reply to Wolfgang Rosenauer from comment #6)
> (In reply to Gerhard  Maier from comment #5)
> > Same here... after Update 113 -> 114 Firefox doesn't work. Go back to 113
> > helps
> 
> Is this also KDE? Or is it nvidia (assuming that the reporter has nvidia
> (NV84)?

Yes, it's also KDE
Comment 8 Wolfgang Rosenauer 2023-06-09 06:39:10 UTC
Did Firefox create a crashreport which was sent to mozilla for any of you?
Comment 9 Gerhard Maier 2023-06-09 06:42:39 UTC
(In reply to Wolfgang Rosenauer from comment #8)
> Did Firefox create a crashreport which was sent to mozilla for any of you?

Yes, i sent various reports to mozilla
Comment 10 Paul Tannington 2023-06-09 07:35:47 UTC
(In reply to Wolfgang Rosenauer from comment #8)
> Did Firefox create a crashreport which was sent to mozilla for any of you?

Yes, also submitted.
Comment 11 Paul Tannington 2023-06-09 09:04:43 UTC
(In reply to Paul Tannington from comment #10)
> (In reply to Wolfgang Rosenauer from comment #8)
> > Did Firefox create a crashreport which was sent to mozilla for any of you?
> 
> Yes, also submitted.

https://crash-stats.mozilla.org/report/index/ec7a98ca-eef4-4ee7-879a-309e80230609
Comment 12 Paul Tannington 2023-06-09 09:42:01 UTC
(In reply to Gerhard  Maier from comment #9)
> (In reply to Wolfgang Rosenauer from comment #8)
> > Did Firefox create a crashreport which was sent to mozilla for any of you?
> 
> Yes, i sent various reports to mozilla

@Gerhard Maier

You might want to check to see if your crash report was actually submitted, as in my instance it appears not to have been.


"Mozilla Crash Reporter" indicated the report was sent, but I was unable to find it at "https://crash-stats.mozilla.org" and locally ".mozilla/firefox/Crash Reports/submitted/" was empty.


I copied the entire "/.mozilla/firefox/Crash Reports" directory structure from FF 114 to a working FF 113 and then used "about:crashes" to submit the report, this time successfully.
Comment 13 Markus Elfring 2023-06-10 05:06:09 UTC
(In reply to Paul Tannington from comment #0)
I became also curious when the program can be properly started again after the software package “MozillaFirefox 114.0-2.1”.
Comment 14 Andreas Stieger 2023-06-10 14:51:27 UTC
Possibly https://bugzilla.mozilla.org/show_bug.cgi?id=1837201
Comment 15 Paul Tannington 2023-06-10 15:48:33 UTC
(In reply to Andreas Stieger from comment #14)
> Possibly https://bugzilla.mozilla.org/show_bug.cgi?id=1837201

I don't believe that is the reason for the crash reported here:

I had seen that bug report, however looking at the linked (Mozilla) bug reports the reason for the crash is either "EXCEPTION_BREAKPOINT" or "EXC_BAD_ACCESS / KERN_INVALID_ADDRESS".

Whereas in this (Mozilla) bug report instance, link in comment #11, all reports give the reason as "SIGILL / ILL_ILLOPN".

Also, interestingly, disregarding 5 duplicated reports, 16 out of 17 list the OS as "openSUSE Tumbleweed" (the 17th being a generic "Linux"). 16/17 seems a little more than coincidence maybe?
Comment 16 Paul Tannington 2023-06-10 16:05:01 UTC
(In reply to Paul Tannington from comment #15)
...
> ... however looking at the linked (Mozilla) bug
> reports the reason for the crash is either "EXCEPTION_BREAKPOINT" or
> "EXC_BAD_ACCESS / KERN_INVALID_ADDRESS".
...
> Whereas in this (Mozilla) bug report instance, link in comment #11, all
> reports give the reason as "SIGILL / ILL_ILLOPN".

Sorry, I meant "crash report", not "bug report" ...
Comment 17 Andreas Stieger 2023-06-10 16:43:35 UTC
Confirm still crashing on start-up with 114.0.1
Comment 18 Paul Tannington 2023-06-10 17:52:41 UTC
(In reply to Andreas Stieger from comment #17)
> Confirm still crashing on start-up with 114.0.1

Hmmm. Yes, just updated one of my TW machines to FF 114.0.1 and indeed FF still crashes at start-up.

Also, as before with 114.0, on Leap 15.4 FF 114.0.1-lp154.1.1 starts OK.

Don't know if this is a FF or TW issue, initially I wondered if it could be hardware (CPU / GPU) related, but looking at the Mozilla crash reports submitted  for 114 there is a mix of hardware, AMD and Intel CPUs, and AMD, Intel & nVidia graphics...
Comment 19 Paul Tannington 2023-06-10 18:13:16 UTC
Let's get the ball rolling again...

Submitted Crash Report to Mozilla:

https://crash-stats.mozilla.org/report/index/78361b89-e993-4de7-a59e-d808a0230610

As with the 114.0 crashes, the reason given in the Crash Report is "SIGILL / ILL_ILLOPN"
Comment 20 Jazz 2023-06-12 14:03:26 UTC
Also happening on my Tumbleweed machine:
https://crash-stats.mozilla.org/report/index/699d1007-3129-4c78-b137-6bc300230612

- KDE: yes
- nvidia: no
Comment 21 Nikolai Nikolaevskii 2023-06-13 09:28:58 UTC
Users with troubles - please try to use Firefox tar from Mozilla: https://www.mozilla.org/en-US/firefox/all/#product-desktop-release
I have some suspicions.
FF 114 & 114.0.1 works OK on Leap 15.4.
Comment 22 Paul Tannington 2023-06-13 09:47:31 UTC
(In reply to Nikolai Nikolaevskii from comment #21)
> Users with troubles - please try to use Firefox tar from Mozilla:
> https://www.mozilla.org/en-US/firefox/all/#product-desktop-release
> I have some suspicions.
> FF 114 & 114.0.1 works OK on Leap 15.4.

Tried that already, FF provided by "firefox-114.0.1.tar.bz2" still crashes in the same manner on my TW system.

To clarify your post. Are you saying that on your TW system 114.0.1 (from https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/) crashes, but that provide by "firefox-114.0.1.tar.bz2" doesn't crash?
Comment 23 Martin Sirringhaus 2023-06-13 11:26:01 UTC
I just contacted upstream, and they don't have the symbols for those builds yet.
Which is why the crash-reports are more or less empty (all red).

They'll scrape them now and hopefully are able to reprocess the crash-reports to contain actual stack traces and proper info, soon.
Comment 24 Paul Tannington 2023-06-13 11:50:02 UTC
(In reply to Paul Tannington from comment #22)
> (In reply to Nikolai Nikolaevskii from comment #21)
> > Users with troubles - please try to use Firefox tar from Mozilla:
> > https://www.mozilla.org/en-US/firefox/all/#product-desktop-release
...
> 
> Tried that already, FF provided by "firefox-114.0.1.tar.bz2" still crashes
> in the same manner on my TW system.
> 
...

Further to that, I've just retried once more to confirm:

Using "https://ftp.mozilla.org/pub/firefox/releases/114.0.1/linux-x86_64/en-GB/firefox-114.0.1.tar.bz2" (MD5: 4ab794c5200c65f67cab6f9db1340d78)

FF Immediately crashes at start-up in the same manner.

Mozilla Crash Report:

https://crash-stats.mozilla.org/report/index/81f45207-9b1e-44d8-b727-7594a0230613


(In reply to Martin Sirringhaus from comment #23)
> I just contacted upstream, and they don't have the symbols for those builds
> yet.
> Which is why the crash-reports are more or less empty (all red).
> 
> They'll scrape them now and hopefully are able to reprocess the
> crash-reports to contain actual stack traces and proper info, soon.

Excellent, many thanks. We will await the outcome.
Comment 25 Nikolai Nikolaevskii 2023-06-13 21:58:17 UTC
ILL problem is in using SSE instruction in AVX notation.
This bug is distinct from bmo #1837201.

My bug report in bmo: https://bugzilla.mozilla.org/show_bug.cgi?id=1838323

Copy from bmo #1838323:
-----------------------------------------------------------
ILL bug persist only on the newest Linux distributions, such as openSUSE Tumbleweed.
openSUSE bugzilla: https://bugzilla.opensuse.org/show_bug.cgi?id=1212101
Possibly this is not Firefox bug, but gcc or compile options or something other.

Batch of crash reports:
https://crash-stats.mozilla.org/signature/?product=Firefox&signature=libxul.so%400x3c912e0%20%7C%20libxul.so%400x3da095a%20%7C%20firefox%400x17eee&date=%3E%3D2023-06-06T20%3A12%3A00.000Z&date=%3C2023-06-13T20%3A12%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#summary

Info from "Crash Report" - "Raw Data and Minidumps":

"crash_info": {

"address": "0x00007fe6fbe912e0",
"assertion": null,
"crashing_thread": 0,
"instruction": "vshufps xmm0, xmm0, xmm0, 0x0",
"memory_accesses": [ ],
"type": "SIGILL / ILL_ILLOPN"

Instruction vshufps is a SSE instruction coded in AVX notation, that is why it requires AVX support: https://www.felixcloutier.com/x86/shufps
CPUs without AVX support will hang.
Hint: older Intel Celerons & Pentiums don't support AVX, it begins with Alder Lake for desktops: https://en.wikipedia.org/wiki/List_of_Intel_Celeron_processors
Intel Atom CPUs support - since Gracemont microarchitecture (end of 2022 year): https://en.wikipedia.org/wiki/Gracemont_(microarchitecture)

Users need SSE instruction in SSE coding.

Instead of AVX-styled
 vshufps xmm0, xmm0, xmm0, 0x0

I expect SSE-styled
 shufps xmm0, xmm0, 0x0
-----------------------------------------------------------
Comment 26 Ilgaz Öcal 2023-06-14 09:14:13 UTC
I have Tumbleweed installed on Powerbook 5.1 (core2 duo/nvidia 9400/nouveau) and HP i5/HD5500. 

I run zypper dup on both systems. 

On core2 duo/nv9400 openSUSE Firefox 114.0.1-1.1 crashes (wayland/kde) however stock Firefox 114.0.1 from getfirefox.com runs fine. Stock Firefox says: "ATTENTION: default value of option mesa_glthread overridden by environment."

On i5/Intel HD5500, openSUSE Tumbleweed stock Firefox launched fine.
Comment 27 Luca Billi 2023-06-14 11:38:48 UTC
Happens to me too on one machine.

I have two similarly configured tumbleweed KDE/X11 machines and on one of them Firefox started crashing on startup after the latest 114.0.1 update:
- CPU: Intel i3-7100        ok
- CPU: Intel Celeron N3450  crash

The only output is:
ExceptionHandler::GenerateDump cloned child 19663
ExceptionHandler::SendContinueSignalToChild sent continue signal to child
ExceptionHandler::WaitForContinueSignal waiting for continue signal...
Comment 28 Andreas Stieger 2023-06-14 19:37:12 UTC
*** Bug 1212384 has been marked as a duplicate of this bug. ***
Comment 29 geheim geheim 2023-06-14 20:08:26 UTC
Same problem here:

Desktop: XFCE
VGA: Intel Corporation Xeon E3-1200
CPU: Intel(R) Pentium(R) CPU G2130 @ 3.20GHz, microcode	: 0x21
Comment 30 Christopher Yeleighton 2023-06-14 21:29:21 UTC
Work-around:

1. Install MozillaFirefox, MozillaFirefox-translations-common of 20230610!
2. Tell Firefox to make a new profile!
3. Tell Firefox to quit!
4. Copy the following files to the new profile! places.sqlite key4.db logins.json permissions.sqlite content-prefs.sqlite search.json.mozlz4 persdict.dat formhistory.sqlite cookies.sqlite cert9.db handlers.json

This will probably make your new old Firefox more or less usable.
Comment 31 Paul Tannington 2023-06-15 08:45:21 UTC
(In reply to Paul Tannington from comment #24)
> (In reply to Paul Tannington from comment #22)
> > (In reply to Nikolai Nikolaevskii from comment #21)
> ...
> 
> Further to that, I've just retried once more to confirm:
> 
> Using
> "https://ftp.mozilla.org/pub/firefox/releases/114.0.1/linux-x86_64/en-GB/
> firefox-114.0.1.tar.bz2" (MD5: 4ab794c5200c65f67cab6f9db1340d78)
> 
> FF Immediately crashes at start-up in the same manner.
> 
> Mozilla Crash Report:
> 
> https://crash-stats.mozilla.org/report/index/81f45207-9b1e-44d8-b727-
> 7594a0230613
> 
> 

My bad...

Firefox provided by "firefox-114.0.1.tar.bz2" *does* in fact start *without* crashing.

I had launched "firefox" from the directory containing the extracted executable, failing to realise it had actually launched (the openSUSE) "firefox" having followed the execution order defined by $PATH.

Launching "Firefox" with the absolute path to the extracted executable and "firefox" starts without crashing.

Sincere apologies for the misleading information, and how embarrassing on my part!


So I guess this is looking like something specific to the openSUSE build?
Comment 32 Martin Sirringhaus 2023-06-15 09:06:42 UTC
(In reply to Paul Tannington from comment #31)
> (In reply to Paul Tannington from comment #24)
> > (In reply to Paul Tannington from comment #22)
> > > (In reply to Nikolai Nikolaevskii from comment #21)
> > ...
> > 
> > Further to that, I've just retried once more to confirm:
> > 
> > Using
> > "https://ftp.mozilla.org/pub/firefox/releases/114.0.1/linux-x86_64/en-GB/
> > firefox-114.0.1.tar.bz2" (MD5: 4ab794c5200c65f67cab6f9db1340d78)
> > 
> > FF Immediately crashes at start-up in the same manner.
> > 
> > Mozilla Crash Report:
> > 
> > https://crash-stats.mozilla.org/report/index/81f45207-9b1e-44d8-b727-
> > 7594a0230613
> > 
> > 
> 
> My bad...
> 
> Firefox provided by "firefox-114.0.1.tar.bz2" *does* in fact start *without*
> crashing.
> 
> I had launched "firefox" from the directory containing the extracted
> executable, failing to realise it had actually launched (the openSUSE)
> "firefox" having followed the execution order defined by $PATH.
> 
> Launching "Firefox" with the absolute path to the extracted executable and
> "firefox" starts without crashing.
> 
> Sincere apologies for the misleading information, and how embarrassing on my
> part!
> 
> 
> So I guess this is looking like something specific to the openSUSE build?

Thanks for correcting this! Me and upstream where scratching our heads about this specific crash-report :-D
This clears it up.

But we have still trouble getting a clear stack-trace from the submitted reports.

Could any of you install the debug-packages and start firefox in gdb (`firefox -d gdb`), to give us a trace?
Comment 33 Victor hck 2023-06-15 10:09:36 UTC
Same issue here after update Tumbleweed+Plasma yesterday. I tried also run firefox from Xfce desktop and get the same issue.

My machine specification here:
https://paste.opensuse.org/pastes/416b637cd3d8

Error info when I run firefox or firefox -safe-mode command form terminal:

ExceptionHandler::GenerateDump cloned child 5285
ExceptionHandler::SendContinueSignalToChild sent continue signal to child
ExceptionHandler::WaitForContinueSignal waiting for continue signal...


Report content that I send to Mozilla:

AdapterDeviceID: 0x0106
AdapterDriverVendor: mesa/crocus
AdapterDriverVersion: 23.1.2.0
AdapterVendorID: 0x8086
AvailablePageFile: 752132096
AvailablePhysicalMemory: 2446344192
AvailableSwapMemory: 4292866048
AvailableVirtualMemory: 4856406016
BackgroundTaskMode: 0
BuildID: 20230608214645
CrashTime: 1686823682
DesktopEnvironment: kde
HeadlessMode: 0
InstallTime: 1686773525
IsWayland: 0
Notes: FP(D00-L1000-W0000000-T010) WR? WR+ 
ProductID: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
ProductName: Firefox
ReleaseChannel: release
SafeMode: 0
SecondsSinceLastCrash: 44
StartupCrash: 1
StartupTime: 1686823680
SubmittedFrom: Client
Throttleable: 1
TotalPageFile: 12417294336
TotalPhysicalMemory: 8124428288
UptimeTS: 1.65052749
Vendor: Mozilla
Version: 114.0.1

This report also contains technical information about the state of the application when it crashed.
Comment 34 Paul Tannington 2023-06-15 12:06:53 UTC
(In reply to Martin Sirringhaus from comment #32)
> ...
> 
> Could any of you install the debug-packages and start firefox in gdb
> (`firefox -d gdb`), to give us a trace?
> ...

Ah! - This is getting above, way above, my pay scale :)

I *think* I've installed all of the necessary debug packages, but...

gdb pauses with:

Thread 1 "firefox" received signal SIGILL, Illegal instruction.
skvx::Vec<4, float>::VecStorage () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:274
274     /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h: No such file or directory.

which makes me think it's not ran to completion?


Full gdb output to that point:


paul@Orion-15:~$ firefox -d gdb
gdb /usr/lib64/firefox/firefox -x /tmp/mozargs.LNPT5m
GNU gdb (GDB; openSUSE Tumbleweed) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib64/firefox/firefox...
Reading symbols from /usr/lib/debug/usr/lib64/firefox/firefox.debug...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff77ff6c0 (LWP 1594)]
[Thread 0x7ffff77ff6c0 (LWP 1594) exited]
[Detaching after fork from child process 1595]
[New Thread 0x7ffff77ff6c0 (LWP 1596)]
[New Thread 0x7fffeb9ca6c0 (LWP 1597)]
[Detaching after vfork from child process 1598]
[New Thread 0x7fffeb7c96c0 (LWP 1599)]
[New Thread 0x7fffeb7886c0 (LWP 1601)]
[New Thread 0x7fffeb7476c0 (LWP 1602)]
[New Thread 0x7fffeb7066c0 (LWP 1603)]
[New Thread 0x7fffeb6c56c0 (LWP 1604)]
[New Thread 0x7fffeb6846c0 (LWP 1605)]
[New Thread 0x7fffe97ff6c0 (LWP 1606)]
[New Thread 0x7fffe6dff6c0 (LWP 1607)]
[New Thread 0x7fffebd096c0 (LWP 1608)]
[Detaching after fork from child process 1609]
[Thread 0x7fffe6dff6c0 (LWP 1607) exited]
[New Thread 0x7fffe65c46c0 (LWP 1615)]
[New Thread 0x7fffe97be6c0 (LWP 1616)]
[New Thread 0x7fffe65836c0 (LWP 1617)]
[New Thread 0x7fffe65426c0 (LWP 1618)]
[New Thread 0x7fffe5aff6c0 (LWP 1619)]
[New Thread 0x7fffe59006c0 (LWP 1620)]
[Thread 0x7fffe65c46c0 (LWP 1615) exited]
[New Thread 0x7fffe65c46c0 (LWP 1621)]
[New Thread 0x7fffe51776c0 (LWP 1622)]
[Thread 0x7fffeb6846c0 (LWP 1605) exited]
[New Thread 0x7fffeb6846c0 (LWP 1623)]
[New Thread 0x7fffe50ac6c0 (LWP 1624)]
[New Thread 0x7fffe506b6c0 (LWP 1625)]
[New Thread 0x7fffe4dff6c0 (LWP 1626)]
[New Thread 0x7fffe4dbe6c0 (LWP 1627)]
[New Thread 0x7fffe4d7d6c0 (LWP 1628)]
[New Thread 0x7fffe4d3c6c0 (LWP 1629)]
[New Thread 0x7fffe4cfb6c0 (LWP 1630)]
[New Thread 0x7fffe4cba6c0 (LWP 1631)]
[New Thread 0x7fffe4c796c0 (LWP 1632)]
[Thread 0x7fffe51776c0 (LWP 1622) exited]
[Thread 0x7fffe4dff6c0 (LWP 1626) exited]
[Thread 0x7fffe50ac6c0 (LWP 1624) exited]
[Thread 0x7fffe4cba6c0 (LWP 1631) exited]
[Thread 0x7fffe4dbe6c0 (LWP 1627) exited]
[Thread 0x7fffe4c796c0 (LWP 1632) exited]
[New Thread 0x7fffe4c796c0 (LWP 1633)]
[New Thread 0x7fffe4dbe6c0 (LWP 1634)]
[Detaching after fork from child process 1635]
kf.i18n: KLocalizedString: Using an empty domain, fix the code. msgid: "Mozilla Firefox" msgid_plural: "" msgctxt: ""
[New Thread 0x7fffe4cba6c0 (LWP 1639)]
[New Thread 0x7fffe6dff6c0 (LWP 1640)]
[New Thread 0x7fffe3fff6c0 (LWP 1641)]
[New Thread 0x7fffe31ff6c0 (LWP 1642)]
[Thread 0x7fffe31ff6c0 (LWP 1642) exited]
[New Thread 0x7fffe31ff6c0 (LWP 1643)]
[New Thread 0x7fffe29c76c0 (LWP 1644)]
[Thread 0x7fffe29c76c0 (LWP 1644) exited]
[New Thread 0x7fffe29c76c0 (LWP 1645)]
[Thread 0x7fffe31ff6c0 (LWP 1643) exited]
[New Thread 0x7fffe31ff6c0 (LWP 1646)]
[Thread 0x7fffe31ff6c0 (LWP 1646) exited]
[New Thread 0x7fffe31ff6c0 (LWP 1647)]
[Thread 0x7fffe29c76c0 (LWP 1645) exited]
[New Thread 0x7fffe29c76c0 (LWP 1648)]
[Thread 0x7fffe31ff6c0 (LWP 1647) exited]
[Thread 0x7fffe29c76c0 (LWP 1648) exited]
[New Thread 0x7fffe29c76c0 (LWP 1649)]

(firefox:1591): Gtk-WARNING **: 12:57:23.070: Theme parsing error: gtk-dark.css:1:50: Failed to import: Error opening file /usr/share/themes/Breeze-Dark/gtk-3.20/gtk.css: No such file or directory

(firefox:1591): Gtk-WARNING **: 12:57:23.116: Theme parsing error: gtk-dark.css:1:50: Failed to import: Error opening file /usr/share/themes/Breeze-Dark/gtk-3.20/gtk.css: No such file or directory

(firefox:1591): Gtk-WARNING **: 12:57:23.118: Theme parsing error: gtk.css:68:35: The style property GtkButton:child-displacement-x is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1591): Gtk-WARNING **: 12:57:23.118: Theme parsing error: gtk.css:69:35: The style property GtkButton:child-displacement-y is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1591): Gtk-WARNING **: 12:57:23.119: Theme parsing error: gtk.css:71:36: The style property GtkCheckMenuItem:indicator-size is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1591): Gtk-WARNING **: 12:57:23.119: Theme parsing error: gtk.css:73:46: The style property GtkScrolledWindow:scrollbars-within-bevel is deprecated and shouldn't be used anymore. It will be removed in a future version

(firefox:1591): Gtk-WARNING **: 12:57:23.119: Theme parsing error: gtk.css:76:30: The style property GtkExpander:expander-size is deprecated and shouldn't be used anymore. It will be removed in a future version
[Thread 0x7fffe29c76c0 (LWP 1649) exited]
ATTENTION: default value of option mesa_glthread overridden by environment.
[New Thread 0x7fffe29c76c0 (LWP 1654)]
[New Thread 0x7fffe50ac6c0 (LWP 1655)]
ATTENTION: default value of option mesa_glthread overridden by environment.
[New Thread 0x7fffe31ff6c0 (LWP 1656)]
[New Thread 0x7fffd843e6c0 (LWP 1657)]
[New Thread 0x7fffd823d6c0 (LWP 1658)]
[New Thread 0x7fffd7eff6c0 (LWP 1659)]
[New Thread 0x7fffd7aff6c0 (LWP 1660)]
[New Thread 0x7fffd7cfe6c0 (LWP 1661)]
libEGL warning: failed to get driver name for fd -1

libEGL warning: MESA-LOADER: failed to retrieve device information

libEGL warning: failed to get driver name for fd -1

ATTENTION: default value of option mesa_glthread overridden by environment.

Thread 1 "firefox" received signal SIGILL, Illegal instruction.
skvx::Vec<4, float>::VecStorage () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:274
274         using VecStorage<N,T>::VecStorage;
(gdb) q
A debugging session is active.

        Inferior 1 [process 1591] will be killed.

Quit anyway? (y or n) y
paul@Orion-15:~$
Comment 35 Nikolai Nikolaevskii 2023-06-15 13:47:38 UTC
ILL as gcc or code optimiser have changed instruction. With that change code works faster on compatible CPUs by using AVX, avoiding penalties for hopping between ordinary and VEX coding scheme https://en.wikipedia.org/wiki/VEX_prefix
Comment 36 Martin Sirringhaus 2023-06-15 13:50:24 UTC
Upstream did update skia in 114, which has a bunch of low-level code in SkVx.h from the trace Paul provided. So that is consistent. Not yet sure how to fix this. 
Could be either in the code or by using different gcc-flags.
Comment 37 Fritz Hudnut 2023-06-17 00:41:08 UTC
Adding my FF crash incident to this bug report . . . in my case running Gecko rolling (stack on TW) MATE DE on '12 Mac Pro with GTX 780 nvidia card after running a zypper dup. . . same scenario, upon trying to launch FF immediate crash reporter opens .. . .

Rebooted the machine into a Debian Bookworm partition and no problems with FF.

I have a straight TW and Leap 15.5 install on the machine that I can cycle through to see what happens . . . .

As posted over on the OpenSUSE list serve, I'm now running my Sys76 laptop with nvidia card in Pop!_OS and yesterday I saw a number of FF named packages in tghe apt full-upgrade . . . and yet, FF is working fine to post this data.
Comment 38 Wolfgang Rosenauer 2023-06-17 04:29:09 UTC
Could some people here try 
https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/x86_64/MozillaFirefox-114.0.1-3.2.x86_64.rpm
either by direct download or using the mozilla repository?

There should be a small change in compile optimizations in there which could make a difference but as I do not have any machine which seems to be affected I have no idea if it does. Please let us know.
Comment 39 Fridrich Strba 2023-06-17 06:04:10 UTC
(In reply to Wolfgang Rosenauer from comment #38)
> Could some people here try 
> https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/
> x86_64/MozillaFirefox-114.0.1-3.2.x86_64.rpm
> either by direct download or using the mozilla repository?
> 
> There should be a small change in compile optimizations in there which could
> make a difference but as I do not have any machine which seems to be
> affected I have no idea if it does. Please let us know.

I have the same problem on one of my machines. I took that version of firefox using "osc getbinaries", but the problem persists :(
Comment 40 Paul Tannington 2023-06-17 08:04:15 UTC
(In reply to Wolfgang Rosenauer from comment #38)
> Could some people here try 
> https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/
> x86_64/MozillaFirefox-114.0.1-3.2.x86_64.rpm
> either by direct download or using the mozilla repository?
> 
> There should be a small change in compile optimizations in there which could
> make a difference but as I do not have any machine which seems to be
> affected I have no idea if it does. Please let us know.


Version      : 114.0.1-3.2
Build Time   : Sat 17 Jun 2023 02:01:52 BST
Install Time : Sat 17 Jun 2023 08:50:20 BST

paul@Orion-15:~$ firefox -d gdb
...
...

Thread 1 "firefox" received signal SIGILL, Illegal instruction.
skvx::Vec<4, float>::VecStorage () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:274
274         using VecStorage<N,T>::VecStorage;
(gdb) bt 10
#0  skvx::Vec<4, float>::VecStorage () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:274
#1  0x00007ffff01a095b in operator*<4, float, float> () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:539
#2  map_rect_affine () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/core/SkM44.cpp:154
#3  0x00007ffff01a1851 in SkMatrixPriv::MapRect () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/core/SkM44.cpp:220
#4  SkCanvas::computeDeviceClipBounds () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/core/SkCanvas.cpp:1709
#5  0x00007ffff01a24a4 in SkCanvas::init () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/core/SkCanvas.cpp:425
#6  0x00007ffff00ec8ce in SkCanvas::SkCanvas () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/core/SkCanvas_Raster.cpp:31
#7  SkSurface_Raster::onNewCanvas () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/image/SkSurface_Raster.cpp:76
#8  0x00007fffed94bc9e in SkSurface_Base::getCachedCanvas () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/image/SkSurface_Base.h:210
#9  SkSurface::getCanvas () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/image/SkSurface.cpp:80
(More stack frames follow...)

...
...
Comment 41 Schmidt 2023-06-17 09:06:45 UTC
How can I go back to 113 in Tumbleweed?
Comment 42 Paul Tannington 2023-06-17 09:40:45 UTC
(In reply to Schmidt from comment #41)
> How can I go back to 113 in Tumbleweed?

You can download FF 113.0.2 from:

https://download.opensuse.org/history/20230608/tumbleweed/repo/oss/x86_64/MozillaFirefox-113.0.2-1.1.x86_64.rpm

If you've not a back-up copy of your old FF 113 profile then after installation initially start FF from the command line with the "--allow-downgrade" parameter.
Comment 43 Ilgaz Öcal 2023-06-18 10:14:13 UTC
I downloaded the new binary and it crashes with
Thread 1 "firefox" received signal SIGILL, Illegal instruction.
0x00007ffff00912e0 in ?? () from /usr/lib64/firefox/libxul.so

My CPU is:
https://www.intel.com/content/www/us/en/products/sku/36750/intel-core2-duo-processor-p7350-3m-cache-2-00-ghz-1066-mhz-fsb/specifications.html
Comment 44 Lee Seymour 2023-06-18 11:04:49 UTC
I also have x86 (0) Intel C2D CPU.
Comment 45 Christopher Yeleighton 2023-06-18 12:48:54 UTC
[ 2121s] + echo 'file /home/abuild/rpmbuild/BUILD/obj/toolkit/library/build/libxul.so
[ 2121s] disassemble '\''skvx::Vec<4, float>::VecStorage(float)'\''
[ 2121s] quit'
[ 2121s] + gdb -batch -x no-avx.gdb
[ 2127s] + grep -F vshufps VecStorage.s
[ 2127s]    0x0000000003cbfda0 <+0>:	vshufps $0x0,%xmm0,%xmm0,%xmm0
[ 2127s] + false
Comment 46 Jiri Slaby 2023-06-19 05:12:23 UTC
(In reply to Christopher Yeleighton from comment #45)
> [ 2121s] + echo 'file
> /home/abuild/rpmbuild/BUILD/obj/toolkit/library/build/libxul.so
> [ 2121s] disassemble '\''skvx::Vec<4, float>::VecStorage(float)'\''
> [ 2121s] quit'
> [ 2121s] + gdb -batch -x no-avx.gdb
> [ 2127s] + grep -F vshufps VecStorage.s
> [ 2127s]    0x0000000003cbfda0 <+0>:	vshufps $0x0,%xmm0,%xmm0,%xmm0

This should be a "x86-64-v3" library residing in /usr/lib64/glibc-hwcaps/x86-64-v3/. Something is wrong with how libxul is built.
Comment 47 Paul Tannington 2023-06-19 07:52:25 UTC
Quote from post 10 of mozilla bug report https://bugzilla.mozilla.org/show_bug.cgi?id=1838323

"My offhand guess as to why our official builds work and these distro builds don't is they're building with AVX flags in the binary. This causes SkVx to include them in the build and use them regardless.

Official builds on x86 are only built with SSE2, with higher SIMD levels only explicitly enabled selectively in various files. I would encourage the distro build maintainers to fix this on their end by not supplying AVX flags to the build. Acceleration for higher levels of SIMD is still used in Mozilla builds, but again, we only selectively enable that in various parts of the moz.build where it is actually safe and properly gated by CPU feature-level checks.

There is no nice way to work around this inside our code, but the downstream build fixes on the distro end should be simple."
Comment 48 Jiri Slaby 2023-06-19 08:05:41 UTC
(In reply to Paul Tannington from comment #47)
> Official builds on x86 are only built with SSE2, with higher SIMD levels
> only explicitly enabled selectively in various files. I would encourage the
> distro build maintainers to fix this on their end by not supplying AVX flags
> to the build. Acceleration for higher levels of SIMD is still used in
> Mozilla builds, but again, we only selectively enable that in various parts
> of the moz.build where it is actually safe and properly gated by CPU
> feature-level checks.

Sure, I assume -march=native (or similar) is used somewhere (and maybe some of the build hosts started exposing AVX and friends to the build VMs).

But I failed to find where as the build scripts are non-standard and the build log is not helping (e.g. there is no build line for SkM44.cpp -- it's likely one of those Unified_cpp's). I hope Wolfgang knows better...
Comment 49 Wolfgang Rosenauer 2023-06-19 10:35:16 UTC
Since it's still quite some guesswork I have built Firefox for TW using gcc12 (instead of 13). If someone wants to test it's
MozillaFirefox-114.0.1-4.1.x86_64.rpm
https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/x86_64/MozillaFirefox-114.0.1-4.1.x86_64.rpm
in the mozilla repository.
Comment 50 Martin Sirringhaus 2023-06-19 10:37:43 UTC
(In reply to Jiri Slaby from comment #48)
> 
> Sure, I assume -march=native (or similar) is used somewhere (and maybe some
> of the build hosts started exposing AVX and friends to the build VMs).
> 
> But I failed to find where as the build scripts are non-standard and the
> build log is not helping (e.g. there is no build line for SkM44.cpp -- it's
> likely one of those Unified_cpp's). I hope Wolfgang knows better...

Should be inside one of the `Unified_cpp_gfx_skia*`-files. In my local build-dir, it's Unified_cpp_gfx_skia5.cpp, but build-flags are identical for all of them, as far as I can see. And I also fail to see anything that could specify -mavx directly or indirectly for those calls.
Comment 51 Jiri Slaby 2023-06-19 11:09:46 UTC
(In reply to Wolfgang Rosenauer from comment #49)
> Since it's still quite some guesswork I have built Firefox for TW using
> gcc12 (instead of 13).

Easily reproducible with an old "CPU":
qemu-kvm ... -cpu Nehalem

Thread 1 "firefox" received signal SIGILL, Illegal instruction.
(gdb) x/i $rip
=> 0x7ffff006a850:      vshufps $0x0,%xmm0,%xmm0,%xmm0

(In reply to Martin Sirringhaus from comment #50)
> Should be inside one of the `Unified_cpp_gfx_skia*`-files. In my local
> build-dir, it's Unified_cpp_gfx_skia5.cpp, but build-flags are identical for
> all of them, as far as I can see. And I also fail to see anything that could
> specify -mavx directly or indirectly for those calls.

Can you paste here a preprocessed SkM44.cpp (generated with -E instead of -c)? And the gcc command line?
Comment 52 Paul Tannington 2023-06-19 12:01:23 UTC
(In reply to Wolfgang Rosenauer from comment #49)
> Since it's still quite some guesswork I have built Firefox for TW using
> gcc12 (instead of 13). If someone wants to test it's
> MozillaFirefox-114.0.1-4.1.x86_64.rpm
> https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/
> x86_64/MozillaFirefox-114.0.1-4.1.x86_64.rpm
> in the mozilla repository.

Version      : 114.0.1-4.1
Build Time   : Mon 19 Jun 2023 10:26:36 BST
Install Time : Mon 19 Jun 2023 12:51:04 BST

Thread 1 "firefox" received signal SIGILL, Illegal instruction.
skvx::Vec<4, float>::VecStorage () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:274
274         using VecStorage<N,T>::VecStorage;
(gdb) x/i $rip
=> 0x7ffff006a850 <_ZN4skvx3VecILi4EfECI2NS_10VecStorageILi4EfEEEf>:    vshufps $0x0,%xmm0,%xmm0,%xmm0
(gdb) bt 4
#0  skvx::Vec<4, float>::VecStorage () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:274
#1  0x00007ffff015c36b in operator*<4, float, float> () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/base/SkVx.h:539
#2  map_rect_affine () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/core/SkM44.cpp:154
#3  0x00007ffff015d28a in SkMatrixPriv::MapRect () at /usr/src/debug/firefox-114.0.1/gfx/skia/skia/src/core/SkM44.cpp:220
(More stack frames follow...)
Comment 53 Martin Sirringhaus 2023-06-19 13:42:55 UTC
Created attachment 867672 [details]
save-temps

Copied the gcc-call of OBS and added `-save-temps`:

/usr/bin/g++ -save-temps -c -o Unified_cpp_gfx_skia5.o -flto -flifetime-dse=1 -I/home/abuild/rpmbuild/BUILD/obj/dist/stl_wrappers -I/home/abuild/rpmbuild/BUILD/obj/dist/system_wrappers -include /home/abuild/rpmbuild/BUILD/firefox-114.0.1/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DNDEBUG=1 -DTRIMMED=1 -DSKIA_IMPLEMENTATION=1 -DSK_PDF_USE_HARFBUZZ_SUBSETTING=1 -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/home/abuild/rpmbuild/BUILD/firefox-114.0.1/gfx/skia -I/home/abuild/rpmbuild/BUILD/obj/gfx/skia -I/home/abuild/rpmbuild/BUILD/firefox-114.0.1/gfx/skia/skia -I/home/abuild/rpmbuild/BUILD/firefox-114.0.1/gfx/cairo/cairo/src -I/home/abuild/rpmbuild/BUILD/obj/dist/include -I/usr/include/nspr4 -I/usr/include/nss3 -I/usr/include/nspr4 -I/home/abuild/rpmbuild/BUILD/obj/dist/include/nss -DMOZILLA_CLIENT -include /home/abuild/rpmbuild/BUILD/obj/mozilla-config.h -fno-sized-deallocation -fno-aligned-new -O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -g -fimplicit-constexpr -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g1 -freorder-blocks -O2 -fomit-frame-pointer -funwind-tables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wno-invalid-offsetof -Wc++2a-compat -Wcomma-subscript -Wvolatile -Wno-error=deprecated -Wno-error=deprecated-enum-enum-conversion -Wduplicated-cond -Wimplicit-fallthrough -Wlogical-op -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wformat -Wformat-overflow=2 -Wno-psabi -Wno-deprecated-declarations -Wno-overloaded-virtual -Wno-sign-compare -Wno-unreachable-code -Wno-unused-function -Wno-logical-op -Wno-maybe-uninitialized -I/usr/include/freetype2 -I/usr/include/freetype2 -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/harfbuzz -I/usr/include/freetype2 -pthread -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/fribidi -I/usr/include/cairo -I/usr/include/libpng16 -I/usr/include/pixman-1 -fno-strict-aliasing -ffp-contract=off  -MD -MP  Unified_cpp_gfx_skia5.cpp

All files are in the attachment.

If the stack traces posted here are correct, it points to the default-constructor of VecStorage. The preprocessed file doesn't show more than that.
Comment 54 Wolfgang Rosenauer 2023-06-19 20:14:26 UTC
There is another build with a different build change:
https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/x86_64/MozillaFirefox-114.0.1-5.1.x86_64.rpm

Please try and report if it changes anything.
Comment 55 Edward Davis 2023-06-19 21:01:39 UTC
(In reply to Wolfgang Rosenauer from comment #54)
> There is another build with a different build change:
> https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/
> x86_64/MozillaFirefox-114.0.1-5.1.x86_64.rpm
> 
> Please try and report if it changes anything.

I upgraded my laptop to this version and it opens OK now.

My laptop is an ASUS with an Intel(R) Celeron(R) N4500 @ 1.10GHz
Running tumbleweed:20230617 and firefox failed to open with the 114
update from the other day.

Thanks,
Edward
Comment 56 Andreas Stieger 2023-06-19 21:42:06 UTC
-ac_add_options --enable-lto
+#ac_add_options --enable-lto

that did the trick.
Comment 57 Gerhard Maier 2023-06-20 06:14:10 UTC
MozillaFirefox-114.0.1-5.1 works again. For me, the problem is solved!
Comment 58 Wolfgang Rosenauer 2023-06-20 06:44:55 UTC
The workaround has been submitted for Tumbleweed.

I hope our gcc experts (or mozilla folks) find the underlying reason so we can eventually turn it on again.
Comment 59 OBSbugzilla Bot 2023-06-20 07:05:02 UTC
This is an autogenerated message for OBS integration:
This bug (1212101) was mentioned in
https://build.opensuse.org/request/show/1093874 Factory / MozillaFirefox
Comment 60 Martin Sirringhaus 2023-06-20 07:57:47 UTC
Then my hunch was correct. On the one hand good, at least we now have a functioning binary again. On the other hand not so good, as upstream suspects a gcc-bug, which might be difficult to pinpoint.
Comment 61 Paul Tannington 2023-06-20 08:15:12 UTC
As confirmed already, Version: 114.0.1-5.1 Build Time: Mon 19 Jun 2023 19:49:03 BST, starts without crashing.

Many thanks to all involved in this issue.
Comment 62 Jiri Slaby 2023-06-20 08:22:32 UTC
(In reply to Andreas Stieger from comment #56)
> -ac_add_options --enable-lto
> +#ac_add_options --enable-lto
> 
> that did the trick.

Guys, any idea how to debug this LTO issue?

It looks like without any -m for gfx/skia/skia/src/core/SkM44.cpp, the final ELF still receives AVX instructions through immintrin.h. And that crashes on older CPUs like Nehalem, of course.

Note that there are some objects where AVX instructions are expected (like gfx/skia/SkOpts_avx.o). So maybe the flags are inherited from that (or other) file by mistake when linking the final ELF?

See also comment #53.
Comment 63 Jiri Slaby 2023-06-20 08:25:42 UTC
Created attachment 867687 [details]
build log of firefox with LTO

(In reply to Jiri Slaby from comment #62)
> Note that there are some objects where AVX instructions are expected (like
> gfx/skia/SkOpts_avx.o). So maybe the flags are inherited from that (or
> other) file by mistake when linking the final ELF?

FTR, the command line contains this for that file (see the attachment for the full log/command line):
-O3 -mavx ...gfx/skia/skia/src/opts/SkOpts_avx.cpp

I had to pack it due to size limits in bsc.
Comment 64 Martin Sirringhaus 2023-06-20 08:38:30 UTC
(In reply to Jiri Slaby from comment #63)
> 
> FTR, the command line contains this for that file (see the attachment for
> the full log/command line):
> -O3 -mavx ...gfx/skia/skia/src/opts/SkOpts_avx.cpp
> 
> I had to pack it due to size limits in bsc.

And also FTR: This was the case for older Firefox-versions as well (and still is for 102ESR), which work(ed) fine.
Maybe, because there was no ` = default`-implementation in Skia that could (by chance) use AVX-instructions, and the Skia-update now brought this to light. Or something else entirely.
But it is (probably?) not going to be as simple as changing some flags, I'm afraid.
Comment 65 Richard Biener 2023-06-20 08:59:13 UTC
(In reply to Martin Sirringhaus from comment #64)
> (In reply to Jiri Slaby from comment #63)
> > 
> > FTR, the command line contains this for that file (see the attachment for
> > the full log/command line):
> > -O3 -mavx ...gfx/skia/skia/src/opts/SkOpts_avx.cpp
> > 
> > I had to pack it due to size limits in bsc.
> 
> And also FTR: This was the case for older Firefox-versions as well (and
> still is for 102ESR), which work(ed) fine.
> Maybe, because there was no ` = default`-implementation in Skia that could
> (by chance) use AVX-instructions, and the Skia-update now brought this to
> light. Or something else entirely.
> But it is (probably?) not going to be as simple as changing some flags, I'm
> afraid.

skvx::Vec<4, float>::VecStorage

looks like a generic C++ name not in any way mangled with a AVX2 template ID or so.  Since we are dealing with C++ template instantiations end up in ELF comdat
sections and the C++ ODR guarantees the linker can pick up any of the instantiations done.  I'm not sure how firefox makes that SkOpts_avx object
work - I suppose they have some wrapping API in there.  But if that CU ends
up exporting any of the generic template instantiations then the linker can
pick an AVX2 variant up when seeing a reference to it.

Using LTO can subtly change the order of the comdats the linker sees in the
end so if Mozilla carefully crafted the linker command-line to avoid this
issue (relying on implementation defined behavior in linkers) then this might
explain why -flto "breaks" it.

I don't see preprocessed source of that SkOpts_avx.cpp file, can somebody
produce it?  Is that the only CU where -mavx is passed?

Can somebody try if appending -fno-lto to the compile command of
SkOpts_avx.cpp helps?  Making everything hidden in that CU besides the
initAVX API entry might also help (-fdefault-visibility=hidden and adding
visibility attributes to the exported API)
Comment 66 Richard Biener 2023-06-20 09:02:41 UTC
I see a few more -mavx uses for mozglue/misc/SIMD_avx2.cpp, gfx/2d/ConvolutionFilterAVX2.cpp, gfx/2d/SwizzleAVX2.cpp, gfx/skia/skia/src/opts/SkOpts_hsw.cpp, gfx/skia/skia/src/opts/SkOpts_skx.cpp (AVX512),
third_party/aom/aom_dsp/x86/aom_subpixel_8t_intrin_avx2.c, ...
and the list continues.  If any of them include the skvx instantiation
you are doomed.
Comment 67 Martin Sirringhaus 2023-06-20 09:28:16 UTC
SkOpts_avx.cpp and where it's only function is being used: https://searchfox.org/mozilla-central/search?q=symbol:_ZN6SkOpts8Init_avxEv&redirect=false

The only place is behind an ifdef that boils down to `__AVX__`

Where VecStorage is being used (first order): https://searchfox.org/mozilla-central/search?q=VecStorage&path=&case=false&regexp=false

None of them are the avx-files, as far as I can tell.

I'll try to upload a preprocessed SkOpts_avx.cpp shortly.
Comment 68 Martin Sirringhaus 2023-06-20 09:42:18 UTC
Created attachment 867688 [details]
Preprocessed SkOpts_avx

Here is the preprocessed SkOpts_avx with -flto. 
Although, fwiw I suspect this file is a red herring.
Comment 69 Richard Biener 2023-06-20 10:45:05 UTC
(In reply to Martin Sirringhaus from comment #68)
> Created attachment 867688 [details]
> Preprocessed SkOpts_avx
> 
> Here is the preprocessed SkOpts_avx with -flto. 
> Although, fwiw I suspect this file is a red herring.

A quick check doesn't show any symbols instantiated besides the obviously
desired ones when using -O3 -mavx.

As said there are other CUs that use ISA flags upon compilation so we'd have to double check all their object files, like do readelf -s on them and grep for
VecStorage for example.
Comment 70 Richard Biener 2023-06-20 11:05:49 UTC
(In reply to Richard Biener from comment #69)
> (In reply to Martin Sirringhaus from comment #68)
> > Created attachment 867688 [details]
> > Preprocessed SkOpts_avx
> > 
> > Here is the preprocessed SkOpts_avx with -flto. 
> > Although, fwiw I suspect this file is a red herring.
> 
> A quick check doesn't show any symbols instantiated besides the obviously
> desired ones when using -O3 -mavx.

Oh, but when building with -O0 I can clearly see the C++ frontend instantiating

skvx::Vec<4, unsigned long>::VecStorage(unsigned long)

for example.  That means these will eventually leak into the LTO link
where they also get "random" chosen.  In fact I can see that with optimization
these symbols are only removed during IPA which means for LTO at LTO WPA
time.

Now I don't think we "clone" the comdats that were built with different ISA
flags to make them prevail.  But I might be wrong.  I know we promote them
local when possible, but I don't think that helps here since it's the linker
deciding which one prevails.  I also don't think we diagnose mismatched
optimization/target flags on cgraph nodes we merge.

> As said there are other CUs that use ISA flags upon compilation so we'd have
> to double check all their object files, like do readelf -s on them and grep
> for
> VecStorage for example.
Comment 71 Martin Sirringhaus 2023-06-20 11:52:20 UTC
(In reply to Richard Biener from comment #69)
> (In reply to Martin Sirringhaus from comment #68)
> > Created attachment 867688 [details]
> > Preprocessed SkOpts_avx
> > 
> > Here is the preprocessed SkOpts_avx with -flto. 
> > Although, fwiw I suspect this file is a red herring.
> 
> A quick check doesn't show any symbols instantiated besides the obviously
> desired ones when using -O3 -mavx.
> 
> As said there are other CUs that use ISA flags upon compilation so we'd have
> to double check all their object files, like do readelf -s on them and grep
> for
> VecStorage for example.

I used the build-log that is attached here.

No hits:

/var/tmp/build-root/openSUSE_Factory-x86_64 ❯ for objf in $(grep mavx .build.log | split --column='-1' | sed 's|\.c.*$|\.o|g'); do OBJFILE="$(basename $objf)"; echo $OBJFILE; fd $OBJFILE "./home/abuild/rpmbuild/BUILD/obj/" -x readelf -s | grep VecStorage; done

SIMD_avx2.o
ConvolutionFilterAVX2.o
SwizzleAVX2.o
SkOpts_avx.o
SkOpts_hsw.o
SkOpts_skx.o
aom_subpixel_8t_intrin_avx2.o
blend_a64_mask_avx2.o
fft_avx2.o
highbd_convolve_avx2.o
highbd_loopfilter_avx2.o
intrapred_avx2.o
cdef_block_avx2.o
av1_inv_txfm_avx2.o
cfl_avx2.o
convolve_2d_avx2.o
convolve_avx2.o
highbd_convolve_2d_avx2.o
highbd_inv_txfm_avx2.o
highbd_jnt_convolve_avx2.o
highbd_wiener_convolve_avx2.o
jnt_convolve_avx2.o
reconinter_avx2.o
selfguided_avx2.o
wiener_convolve_avx2.o
vp9_diamond_search_sad_avx.o
vp9_error_avx2.o
vp9_quantize_avx2.o
avg_intrin_avx2.o
fwd_txfm_avx2.o
loopfilter_avx2.o
quantize_avx.o
quantize_avx2.o
sad4d_avx2.o
sad_avx2.o
subtract_avx2.o
variance_avx2.o
vpx_subpixel_8t_intrin_avx2.o
Unified_cpp_common_audio_avx2_gn0.o
Unified_cpp_aec3_aec3_avx2_gn0.o
Unified_cpp_vector_math_avx2_gn0.o
Comment 72 Richard Biener 2023-06-20 12:22:29 UTC
(In reply to Martin Sirringhaus from comment #71)
> (In reply to Richard Biener from comment #69)
> > (In reply to Martin Sirringhaus from comment #68)
> > > Created attachment 867688 [details]
> > > Preprocessed SkOpts_avx
> > > 
> > > Here is the preprocessed SkOpts_avx with -flto. 
> > > Although, fwiw I suspect this file is a red herring.
> > 
> > A quick check doesn't show any symbols instantiated besides the obviously
> > desired ones when using -O3 -mavx.
> > 
> > As said there are other CUs that use ISA flags upon compilation so we'd have
> > to double check all their object files, like do readelf -s on them and grep
> > for
> > VecStorage for example.
> 
> I used the build-log that is attached here.
> 
> No hits:
> 
> /var/tmp/build-root/openSUSE_Factory-x86_64 ❯ for objf in $(grep mavx
> .build.log | split --column='-1' | sed 's|\.c.*$|\.o|g'); do
> OBJFILE="$(basename $objf)"; echo $OBJFILE; fd $OBJFILE
> "./home/abuild/rpmbuild/BUILD/obj/" -x readelf -s | grep VecStorage; done

That's the LTO built objects or the ones with LTO disabled?  With LTO
objects you have to resort to gcc-nm instead (that will inspect the LTO
symbol table).

> SIMD_avx2.o
> ConvolutionFilterAVX2.o
> SwizzleAVX2.o
> SkOpts_avx.o
> SkOpts_hsw.o
> SkOpts_skx.o
> aom_subpixel_8t_intrin_avx2.o
> blend_a64_mask_avx2.o
> fft_avx2.o
> highbd_convolve_avx2.o
> highbd_loopfilter_avx2.o
> intrapred_avx2.o
> cdef_block_avx2.o
> av1_inv_txfm_avx2.o
> cfl_avx2.o
> convolve_2d_avx2.o
> convolve_avx2.o
> highbd_convolve_2d_avx2.o
> highbd_inv_txfm_avx2.o
> highbd_jnt_convolve_avx2.o
> highbd_wiener_convolve_avx2.o
> jnt_convolve_avx2.o
> reconinter_avx2.o
> selfguided_avx2.o
> wiener_convolve_avx2.o
> vp9_diamond_search_sad_avx.o
> vp9_error_avx2.o
> vp9_quantize_avx2.o
> avg_intrin_avx2.o
> fwd_txfm_avx2.o
> loopfilter_avx2.o
> quantize_avx.o
> quantize_avx2.o
> sad4d_avx2.o
> sad_avx2.o
> subtract_avx2.o
> variance_avx2.o
> vpx_subpixel_8t_intrin_avx2.o
> Unified_cpp_common_audio_avx2_gn0.o
> Unified_cpp_aec3_aec3_avx2_gn0.o
> Unified_cpp_vector_math_avx2_gn0.o
Comment 73 Martin Sirringhaus 2023-06-20 13:42:39 UTC
Ok, using `gcc-nm -C {} | grep VecStorage` I get two hits:

SkOpts_hsw.o
00000000 W skvx::Vec<16, float>::VecStorage(float)
00000000 W skvx::Vec<16, float>::VecStorage(float)
00000000 W skvx::Vec<16, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<16, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<16, unsigned short>::VecStorage(unsigned short)
00000000 W skvx::Vec<16, unsigned short>::VecStorage(unsigned short)
00000000 W skvx::Vec<2, float>::VecStorage(float)
00000000 W skvx::Vec<2, float>::VecStorage(float)
00000000 W skvx::Vec<2, unsigned char>::VecStorage(unsigned char)
00000000 W skvx::Vec<2, unsigned char>::VecStorage(unsigned char)
00000000 W skvx::Vec<2, int>::VecStorage(int)
00000000 W skvx::Vec<2, int>::VecStorage(int)
00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<2, unsigned long>::VecStorage(unsigned long)
00000000 W skvx::Vec<2, unsigned long>::VecStorage(unsigned long)
00000000 W skvx::Vec<2, unsigned short>::VecStorage(unsigned short)
00000000 W skvx::Vec<2, unsigned short>::VecStorage(unsigned short)
00000000 W skvx::Vec<32, int>::VecStorage(int)
00000000 W skvx::Vec<32, int>::VecStorage(int)
00000000 W skvx::Vec<4, float>::VecStorage(float)
00000000 W skvx::Vec<4, float>::VecStorage(float)
00000000 W skvx::Vec<4, unsigned char>::VecStorage(unsigned char)
00000000 W skvx::Vec<4, unsigned char>::VecStorage(unsigned char)
00000000 W skvx::Vec<4, int>::VecStorage(int)
00000000 W skvx::Vec<4, int>::VecStorage(int)
00000000 W skvx::Vec<4, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<4, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<4, unsigned long>::VecStorage(unsigned long)
00000000 W skvx::Vec<4, unsigned long>::VecStorage(unsigned long)
00000000 W skvx::Vec<4, unsigned short>::VecStorage(unsigned short)
00000000 W skvx::Vec<4, unsigned short>::VecStorage(unsigned short)
00000000 W skvx::Vec<8, unsigned char>::VecStorage(unsigned char)
00000000 W skvx::Vec<8, unsigned char>::VecStorage(unsigned char)
00000000 W skvx::Vec<8, int>::VecStorage(int)
00000000 W skvx::Vec<8, int>::VecStorage(int)
00000000 W skvx::Vec<8, unsigned short>::VecStorage(unsigned short)
00000000 W skvx::Vec<8, unsigned short>::VecStorage(unsigned short)
SkOpts_skx.o
00000000 W skvx::Vec<16, int>::VecStorage(int)
00000000 W skvx::Vec<16, int>::VecStorage(int)
00000000 W skvx::Vec<2, int>::VecStorage(int)
00000000 W skvx::Vec<2, int>::VecStorage(int)
00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
00000000 W skvx::Vec<4, int>::VecStorage(int)
00000000 W skvx::Vec<4, int>::VecStorage(int)
00000000 W skvx::Vec<8, int>::VecStorage(int)
00000000 W skvx::Vec<8, int>::VecStorage(int)

But why would upstreams binary work, while ours crashes? Shouldn't that be a general problem, then?
Comment 74 Victor hck 2023-06-20 15:19:03 UTC
(In reply to Wolfgang Rosenauer from comment #54)
> There is another build with a different build change:
> https://download.opensuse.org/repositories/mozilla/openSUSE_Tumbleweed/
> x86_64/MozillaFirefox-114.0.1-5.1.x86_64.rpm
> 
> Please try and report if it changes anything.

Downloades the rpm with wget and updated with sudo rpm -U MozillaFirefox-114.0.1-5.1.x86_64.rpm
and now works fine again!
Thanks a lot for your work!

Greetings!
Comment 75 Richard Biener 2023-06-21 07:00:03 UTC
(In reply to Martin Sirringhaus from comment #73)
> Ok, using `gcc-nm -C {} | grep VecStorage` I get two hits:
> 
> SkOpts_hsw.o
> 00000000 W skvx::Vec<16, float>::VecStorage(float)
> 00000000 W skvx::Vec<16, float>::VecStorage(float)
> 00000000 W skvx::Vec<16, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<16, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<16, unsigned short>::VecStorage(unsigned short)
> 00000000 W skvx::Vec<16, unsigned short>::VecStorage(unsigned short)
> 00000000 W skvx::Vec<2, float>::VecStorage(float)
> 00000000 W skvx::Vec<2, float>::VecStorage(float)
> 00000000 W skvx::Vec<2, unsigned char>::VecStorage(unsigned char)
> 00000000 W skvx::Vec<2, unsigned char>::VecStorage(unsigned char)
> 00000000 W skvx::Vec<2, int>::VecStorage(int)
> 00000000 W skvx::Vec<2, int>::VecStorage(int)
> 00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<2, unsigned long>::VecStorage(unsigned long)
> 00000000 W skvx::Vec<2, unsigned long>::VecStorage(unsigned long)
> 00000000 W skvx::Vec<2, unsigned short>::VecStorage(unsigned short)
> 00000000 W skvx::Vec<2, unsigned short>::VecStorage(unsigned short)
> 00000000 W skvx::Vec<32, int>::VecStorage(int)
> 00000000 W skvx::Vec<32, int>::VecStorage(int)
> 00000000 W skvx::Vec<4, float>::VecStorage(float)
> 00000000 W skvx::Vec<4, float>::VecStorage(float)
> 00000000 W skvx::Vec<4, unsigned char>::VecStorage(unsigned char)
> 00000000 W skvx::Vec<4, unsigned char>::VecStorage(unsigned char)
> 00000000 W skvx::Vec<4, int>::VecStorage(int)
> 00000000 W skvx::Vec<4, int>::VecStorage(int)
> 00000000 W skvx::Vec<4, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<4, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<4, unsigned long>::VecStorage(unsigned long)
> 00000000 W skvx::Vec<4, unsigned long>::VecStorage(unsigned long)
> 00000000 W skvx::Vec<4, unsigned short>::VecStorage(unsigned short)
> 00000000 W skvx::Vec<4, unsigned short>::VecStorage(unsigned short)
> 00000000 W skvx::Vec<8, unsigned char>::VecStorage(unsigned char)
> 00000000 W skvx::Vec<8, unsigned char>::VecStorage(unsigned char)
> 00000000 W skvx::Vec<8, int>::VecStorage(int)
> 00000000 W skvx::Vec<8, int>::VecStorage(int)
> 00000000 W skvx::Vec<8, unsigned short>::VecStorage(unsigned short)
> 00000000 W skvx::Vec<8, unsigned short>::VecStorage(unsigned short)
> SkOpts_skx.o
> 00000000 W skvx::Vec<16, int>::VecStorage(int)
> 00000000 W skvx::Vec<16, int>::VecStorage(int)
> 00000000 W skvx::Vec<2, int>::VecStorage(int)
> 00000000 W skvx::Vec<2, int>::VecStorage(int)
> 00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<2, unsigned int>::VecStorage(unsigned int)
> 00000000 W skvx::Vec<4, int>::VecStorage(int)
> 00000000 W skvx::Vec<4, int>::VecStorage(int)
> 00000000 W skvx::Vec<8, int>::VecStorage(int)
> 00000000 W skvx::Vec<8, int>::VecStorage(int)
> 
> But why would upstreams binary work, while ours crashes? Shouldn't that be a
> general problem, then?

It's a problem that's somewhat specific to LTO as with LTO the linker is
involved before these symbols are removed as unneeded and it gets the
chance to pick up the "wrong" one as prevailing.
Comment 76 Christopher Yeleighton 2023-06-22 08:42:45 UTC
(In reply to Jiri Slaby from comment #46)
> (In reply to Christopher Yeleighton from comment #45)
> > [ 2121s] + echo 'file
> > /home/abuild/rpmbuild/BUILD/obj/toolkit/library/build/libxul.so
> > [ 2121s] disassemble '\''skvx::Vec<4, float>::VecStorage(float)'\''
> > [ 2121s] quit'
> > [ 2121s] + gdb -batch -x no-avx.gdb
> > [ 2127s] + grep -F vshufps VecStorage.s
> > [ 2127s]    0x0000000003cbfda0 <+0>:	vshufps $0x0,%xmm0,%xmm0,%xmm0
> 
> This should be a "x86-64-v3" library residing in
> /usr/lib64/glibc-hwcaps/x86-64-v3/. Something is wrong with how libxul is
> built.

There is nothing wrong, I just checked the result in the build tree.
Comment 77 Christopher Yeleighton 2023-06-22 09:14:43 UTC
(In reply to Richard Biener from comment #70)
> (In reply to Richard Biener from comment #69)
> > (In reply to Martin Sirringhaus from comment #68)
> > > Created attachment 867688 [details]
> > > Preprocessed SkOpts_avx
> > > 
> > > Here is the preprocessed SkOpts_avx with -flto. 
> > > Although, fwiw I suspect this file is a red herring.
> > 
> > A quick check doesn't show any symbols instantiated besides the obviously
> > desired ones when using -O3 -mavx.
> 
> Oh, but when building with -O0 I can clearly see the C++ frontend
> instantiating
> 
> skvx::Vec<4, unsigned long>::VecStorage(unsigned long)
> 
> for example.  That means these will eventually leak into the LTO link
> where they also get "random" chosen.  In fact I can see that with
> optimization
> these symbols are only removed during IPA which means for LTO at LTO WPA
> time.

To make your explanation less cryptic: these constructors should be expanded inline and forgotten and not end up as separate symbols.  However, both -O0 and -flto cause them to be used as stand-alone functions, one for debugging and the other for binary optimisation.  I have checked that libxul.so built without -flto does not contain the offending constructor at all.
Comment 78 Christopher Yeleighton 2023-06-22 09:22:20 UTC
(In reply to Richard Biener from comment #70)
> Now I don't think we "clone" the comdats that were built with different ISA
> flags to make them prevail.  But I might be wrong.  I know we promote them
> local when possible, but I don't think that helps here since it's the linker
> deciding which one prevails.  I also don't think we diagnose mismatched
> optimization/target flags on cgraph nodes we merge.

Upstream could solve this problem by putting AVX-enabled instantiations into a separate namespace.
Comment 79 Richard Biener 2023-06-22 11:04:30 UTC
Btw, the code actually instantiates skvx::Vec<> stuff via

namespace avx {
    
    template <typename T> 
    static void memsetT(T buffer[], T value, int count) {

        static constexpr int N = 32 / sizeof(T);
    


        static_assert(N > 0, "T is too big for memsetT");

        skvx::Vec<N,T> wideValue(value);
        while (count >= N) {

            wideValue.store(buffer);
            buffer += N;
            count -= N;
        }

        while (count --> 0) {
            *buffer++ = value;
        }
    }

                   inline void memset16(uint16_t buffer[], uint16_t value, int count) {
        memsetT(buffer, value, count);
    }
...
}

and in SkOpts_avx.cpp:

namespace SkOpts {
    void Init_avx() {
        memset16 = avx::memset16;
...

and it expects everything to be fully optimized, eliminating all used
template instantiations.  But at the time we stream out for LTO that
does not have happened.

So I don't see how we can avoid this issue besides inventing some "DWIM"
mechanism here.

I would suggest to add -fno-lto to all TUs that are built with extra
machine specific flags as workaround.
Comment 80 Richard Biener 2023-06-22 11:17:22 UTC
Oh, btw, removing all __attribute__((always_inline))s from the TU makes the inline heuristics work and we get everything optimized early.

The alternative is to place __attribute__((flatten)) on memsetT and rect_memsetT
Comment 81 Nikolai Nikolaevskii 2023-06-27 22:11:43 UTC
Google creates skia and uses clang to compile it. 
Mozilla uses its own fork of skia, and uses clang for Firefox.
So, one of solution is in using clang to compile Firefox.

I wonder why openSUSE uses 'gnu99' C Dialect Options together with 'c++17' https://gcc.gnu.org/onlinedocs/gcc/C-Dialect-Options.html
IMHO 'gnu99' is rather outdated and may cause incompatibilities with code written for clang https://clang.llvm.org/compatibility.html.
Try deleting these options and using default settings.

Letters 'hsw' in SkOpts_hsw.cpp name hint on Haswell "https://en.wikipedia.org/wiki/Haswell_(microarchitecture)", which supports AVX2 and older, and this corresponds to gcc's x86-64-v3. 
Letters 'skx' in SkOpts_skx.cpp name hint on Skylake X https://en.wikichip.org/wiki/intel/cores/skylake_x, which supports AVX512 and older, and this corresponds to gcc's x86-64-v4.

For Leap 15.4 openSUSE with Mozilla repo uses gcc 7.5 together with 11.3?
Comment 82 Lee Salzman 2023-07-05 03:23:51 UTC
(In reply to Richard Biener from comment #79)
> Btw, the code actually instantiates skvx::Vec<> stuff via
> 
> namespace avx {
>     
>     template <typename T> 
>     static void memsetT(T buffer[], T value, int count) {
> 
>         static constexpr int N = 32 / sizeof(T);
>     
> 
> 
>         static_assert(N > 0, "T is too big for memsetT");
> 
>         skvx::Vec<N,T> wideValue(value);
>         while (count >= N) {
> 
>             wideValue.store(buffer);
>             buffer += N;
>             count -= N;
>         }
> 
>         while (count --> 0) {
>             *buffer++ = value;
>         }
>     }
> 
>                    inline void memset16(uint16_t buffer[], uint16_t value,
> int count) {
>         memsetT(buffer, value, count);
>     }
> ...
> }
> 
> and in SkOpts_avx.cpp:
> 
> namespace SkOpts {
>     void Init_avx() {
>         memset16 = avx::memset16;
> ...
> 
> and it expects everything to be fully optimized, eliminating all used
> template instantiations.  But at the time we stream out for LTO that
> does not have happened.
> 
> So I don't see how we can avoid this issue besides inventing some "DWIM"
> mechanism here.
> 
> I would suggest to add -fno-lto to all TUs that are built with extra
> machine specific flags as workaround.

Richard, would just adding -fno-lto to the flags for the SkOpts_foo.cpp files be sufficient to work around this on your end for GCC builds? I am willing to add this to Firefox's moz.builds to just make this easy. If GCC will let us selectively disable LTO on certain files like that, I am on board.
Comment 83 Richard Biener 2023-07-05 07:43:31 UTC
(In reply to Lee Salzman from comment #82)
> (In reply to Richard Biener from comment #79)
> > Btw, the code actually instantiates skvx::Vec<> stuff via
> > 
> > namespace avx {
> >     
> >     template <typename T> 
> >     static void memsetT(T buffer[], T value, int count) {
> > 
> >         static constexpr int N = 32 / sizeof(T);
> >     
> > 
> > 
> >         static_assert(N > 0, "T is too big for memsetT");
> > 
> >         skvx::Vec<N,T> wideValue(value);
> >         while (count >= N) {
> > 
> >             wideValue.store(buffer);
> >             buffer += N;
> >             count -= N;
> >         }
> > 
> >         while (count --> 0) {
> >             *buffer++ = value;
> >         }
> >     }
> > 
> >                    inline void memset16(uint16_t buffer[], uint16_t value,
> > int count) {
> >         memsetT(buffer, value, count);
> >     }
> > ...
> > }
> > 
> > and in SkOpts_avx.cpp:
> > 
> > namespace SkOpts {
> >     void Init_avx() {
> >         memset16 = avx::memset16;
> > ...
> > 
> > and it expects everything to be fully optimized, eliminating all used
> > template instantiations.  But at the time we stream out for LTO that
> > does not have happened.
> > 
> > So I don't see how we can avoid this issue besides inventing some "DWIM"
> > mechanism here.
> > 
> > I would suggest to add -fno-lto to all TUs that are built with extra
> > machine specific flags as workaround.
> 
> Richard, would just adding -fno-lto to the flags for the SkOpts_foo.cpp
> files be sufficient to work around this on your end for GCC builds? I am
> willing to add this to Firefox's moz.builds to just make this easy. If GCC
> will let us selectively disable LTO on certain files like that, I am on
> board.

Yes, GCC allows that.  In the upstream GCC bug tracking this issue Jakub
also suggested to do the following instead of compiling the whole TU
with -mavx (also applies to the other CPU specific namespaces):

#include <skia.h>

#pragma GCC push_options
#pragma GCC target("avx")

namespace avx {

...

}

#pragma GCC pop_options

that makes sure all standalone instantiations of SKIA classes will not
have -mavx applied but only the inline copies in the functions inside
the avx namespace.

Note we've identified the issue causing the missed optimization which
is not applying the always-inline attribute (which SKIA uses almost
everywhere) to some C++ constructor clones.
Comment 84 Lee Salzman 2023-07-06 02:28:47 UTC
I landed a slightly different idea in Firefox that should hypothetically still work: https://phabricator.services.mozilla.com/D182855

I am just making sure I rename the skvx namespace every time we build with different arch options, so that it doesn't end up with a bunch of different ambiguous symbols in the skvx namespace. This is easier than doing anything in the source files themselves which would require vendored patches or upstream Skia changes.

It might take a day or so to actually get in nightly, but it's on autoland at the moment. You can also try applying the patch to your local builds, and see if it allows building with LTO again on GCC?
Comment 85 Richard Biener 2023-07-06 07:06:19 UTC
(In reply to Lee Salzman from comment #84)
> I landed a slightly different idea in Firefox that should hypothetically
> still work: https://phabricator.services.mozilla.com/D182855

Ah, yes - that's also a nice idea and also most portable I guess.

> I am just making sure I rename the skvx namespace every time we build with
> different arch options, so that it doesn't end up with a bunch of different
> ambiguous symbols in the skvx namespace. This is easier than doing anything
> in the source files themselves which would require vendored patches or
> upstream Skia changes.
> 
> It might take a day or so to actually get in nightly, but it's on autoland
> at the moment. You can also try applying the patch to your local builds, and
> see if it allows building with LTO again on GCC?

I'll leave that to our firefox package maintainers, meanwhile I have prepared an updated GCC 13 package to solve the missed inlines.
Comment 86 OBSbugzilla Bot 2023-07-06 07:45:03 UTC
This is an autogenerated message for OBS integration:
This bug (1212101) was mentioned in
https://build.opensuse.org/request/show/1097045 Factory / gcc13
Comment 90 Richard Biener 2023-07-10 08:47:46 UTC
I will have to revert the GCC 13 backport of the PR110334 frontend fix since
it breaks libreoffice build also with Skia - it looks like libreoffice is
possibly affected in similar ways here with its vendoring of Skia.
Comment 91 Danilo Spinella 2023-07-10 10:51:15 UTC
(In reply to Richard Biener from comment #90)
> I will have to revert the GCC 13 backport of the PR110334 frontend fix since
> it breaks libreoffice build also with Skia - it looks like libreoffice is
> possibly affected in similar ways here with its vendoring of Skia.

Yea, libreoffice is affected as well, there is a build log in bsc#1213177.
Comment 92 OBSbugzilla Bot 2023-07-10 11:55:04 UTC
This is an autogenerated message for OBS integration:
This bug (1212101) was mentioned in
https://build.opensuse.org/request/show/1097918 Factory / gcc13
Comment 95 Maintenance Automation 2023-07-17 09:36:51 UTC
SUSE-SU-2023:2850-1: An update that solves 13 vulnerabilities can now be installed.

Category: security (important)
Bug References: 1212101, 1212438
CVE References: CVE-2023-3482, CVE-2023-37201, CVE-2023-37202, CVE-2023-37203, CVE-2023-37204, CVE-2023-37205, CVE-2023-37206, CVE-2023-37207, CVE-2023-37208, CVE-2023-37209, CVE-2023-37210, CVE-2023-37211, CVE-2023-37212
Sources used:
SUSE OpenStack Cloud 9 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE OpenStack Cloud Crowbar 9 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE Linux Enterprise Server for SAP Applications 12 SP4 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE Linux Enterprise Software Development Kit 12 SP5 (src): MozillaFirefox-115.0-112.165.1
SUSE Linux Enterprise Server 12 SP2 BCL 12-SP2 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE Linux Enterprise Server 12 SP4 ESPOS 12-SP4 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE Linux Enterprise Server 12 SP4 LTSS 12-SP4 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE Linux Enterprise High Performance Computing 12 SP5 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE Linux Enterprise Server 12 SP5 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2
SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src): MozillaFirefox-115.0-112.165.1, MozillaFirefox-branding-SLE-115-35.12.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 96 Maintenance Automation 2023-07-17 09:36:55 UTC
SUSE-SU-2023:2849-1: An update that solves 13 vulnerabilities can now be installed.

Category: security (important)
Bug References: 1212101, 1212438
CVE References: CVE-2023-3482, CVE-2023-37201, CVE-2023-37202, CVE-2023-37203, CVE-2023-37204, CVE-2023-37205, CVE-2023-37206, CVE-2023-37207, CVE-2023-37208, CVE-2023-37209, CVE-2023-37210, CVE-2023-37211, CVE-2023-37212
Sources used:
SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1 (src): MozillaFirefox-115.0-150000.150.91.1, MozillaFirefox-branding-SLE-115-150000.4.25.1
SUSE Linux Enterprise Server 15 SP1 LTSS 15-SP1 (src): MozillaFirefox-115.0-150000.150.91.1, MozillaFirefox-branding-SLE-115-150000.4.25.1
SUSE Linux Enterprise Server for SAP Applications 15 SP1 (src): MozillaFirefox-115.0-150000.150.91.1, MozillaFirefox-branding-SLE-115-150000.4.25.1
SUSE CaaS Platform 4.0 (src): MozillaFirefox-115.0-150000.150.91.1, MozillaFirefox-branding-SLE-115-150000.4.25.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 99 Maintenance Automation 2023-07-19 16:30:08 UTC
SUSE-SU-2023:2886-1: An update that solves 13 vulnerabilities can now be installed.

Category: security (important)
Bug References: 1212101, 1212438
CVE References: CVE-2023-3482, CVE-2023-37201, CVE-2023-37202, CVE-2023-37203, CVE-2023-37204, CVE-2023-37205, CVE-2023-37206, CVE-2023-37207, CVE-2023-37208, CVE-2023-37209, CVE-2023-37210, CVE-2023-37211, CVE-2023-37212
Sources used:
openSUSE Leap 15.5 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
Desktop Applications Module 15-SP4 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
Desktop Applications Module 15-SP5 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP3 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise Real Time 15 SP3 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Enterprise Storage 7.1 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
SUSE Enterprise Storage 7 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1
openSUSE Leap 15.4 (src): MozillaFirefox-branding-SLE-115-150200.9.13.1, MozillaFirefox-115.0-150200.152.93.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 102 Maintenance Automation 2023-10-23 16:30:01 UTC
SUSE-SU-2023:4162-1: An update that solves one vulnerability, contains five features and has seven security fixes can now be installed.

Category: security (important)
Bug References: 1206480, 1206684, 1210557, 1211427, 1212101, 1213915, 1214052, 1214460
CVE References: CVE-2023-4039
Jira References: PED-153, PED-2005, PED-252, PED-253, PED-6584
Sources used:
openSUSE Leap Micro 5.3 (src): gcc13-13.2.1+git7813-150000.1.3.3
openSUSE Leap Micro 5.4 (src): gcc13-13.2.1+git7813-150000.1.3.3
openSUSE Leap 15.4 (src): gcc13-13.2.1+git7813-150000.1.3.3, cross-nvptx-gcc13-13.2.1+git7813-150000.1.3.2
openSUSE Leap 15.5 (src): gcc13-13.2.1+git7813-150000.1.3.3, cross-nvptx-gcc13-13.2.1+git7813-150000.1.3.2
SUSE Linux Enterprise Server 15 SP2 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Server 15 SP3 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise High Performance Computing 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Server 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Manager Server 4.3 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Desktop 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Manager Retail Branch Server 4.3 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Manager Proxy 4.3 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise High Performance Computing 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Server 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Server for SAP Applications 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Desktop 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro for Rancher 5.3 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro 5.3 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro for Rancher 5.4 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro 5.4 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro 5.5 (src): gcc13-13.2.1+git7813-150000.1.3.3
Basesystem Module 15-SP4 (src): gcc13-13.2.1+git7813-150000.1.3.3
Basesystem Module 15-SP5 (src): gcc13-13.2.1+git7813-150000.1.3.3
Development Tools Module 15-SP4 (src): gcc13-13.2.1+git7813-150000.1.3.3, cross-nvptx-gcc13-13.2.1+git7813-150000.1.3.2
Development Tools Module 15-SP5 (src): gcc13-13.2.1+git7813-150000.1.3.3, cross-nvptx-gcc13-13.2.1+git7813-150000.1.3.2
SUSE Package Hub 15 15-SP4 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Package Hub 15 15-SP5 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Manager Proxy 4.2 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Manager Retail Branch Server 4.2 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Manager Server 4.2 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro 5.1 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro 5.2 (src): gcc13-13.2.1+git7813-150000.1.3.3
SUSE Linux Enterprise Micro for Rancher 5.2 (src): gcc13-13.2.1+git7813-150000.1.3.3

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 103 Maintenance Automation 2023-10-31 12:30:54 UTC
SUSE-SU-2023:4287-1: An update that solves one vulnerability, contains five features and has seven security fixes can now be installed.

Category: security (important)
Bug References: 1206480, 1206684, 1210557, 1211427, 1212101, 1213915, 1214052, 1214460
CVE References: CVE-2023-4039
Jira References: PED-153, PED-2005, PED-252, PED-253, PED-6584
Sources used:
Toolchain Module 12 (src): gcc13-13.2.1+git7813-1.6.1, cross-nvptx-gcc13-13.2.1+git7813-1.6.1
SUSE Linux Enterprise High Performance Computing 12 SP5 (src): gcc13-13.2.1+git7813-1.6.1
SUSE Linux Enterprise Server 12 SP5 (src): gcc13-13.2.1+git7813-1.6.1
SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src): gcc13-13.2.1+git7813-1.6.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 104 Maintenance Automation 2023-11-02 12:30:03 UTC
SUSE-SU-2023:4287-2: An update that solves one vulnerability, contains five features and has seven security fixes can now be installed.

Category: security (important)
Bug References: 1206480, 1206684, 1210557, 1211427, 1212101, 1213915, 1214052, 1214460
CVE References: CVE-2023-4039
Jira References: PED-153, PED-2005, PED-252, PED-253, PED-6584
Sources used:
Toolchain Module 12 (src): cross-nvptx-gcc13-13.2.1+git7813-1.6.1, gcc13-13.2.1+git7813-1.6.1
SUSE Linux Enterprise High Performance Computing 12 SP5 (src): gcc13-13.2.1+git7813-1.6.1
SUSE Linux Enterprise Server 12 SP5 (src): gcc13-13.2.1+git7813-1.6.1
SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src): gcc13-13.2.1+git7813-1.6.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 106 Maintenance Automation 2023-11-16 16:30:13 UTC
SUSE-SU-2023:4458-1: An update that solves one vulnerability, contains five features and has nine security fixes can now be installed.

Category: security (important)
Bug References: 1206480, 1206684, 1210557, 1211427, 1212101, 1213915, 1214052, 1214460, 1215427, 1216664
CVE References: CVE-2023-4039
Jira References: PED-153, PED-2005, PED-252, PED-253, PED-6584
Sources used:
openSUSE Leap Micro 5.3 (src): gcc13-13.2.1+git7813-150000.1.6.1
openSUSE Leap Micro 5.4 (src): gcc13-13.2.1+git7813-150000.1.6.1
openSUSE Leap 15.4 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
openSUSE Leap 15.5 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP1 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP2 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP3 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise High Performance Computing 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Manager Server 4.3 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Desktop 15 SP4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Manager Retail Branch Server 4.3 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Manager Proxy 4.3 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise High Performance Computing 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server for SAP Applications 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Desktop 15 SP5 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro for Rancher 5.3 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.3 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro for Rancher 5.4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.5 (src): gcc13-13.2.1+git7813-150000.1.6.1
Basesystem Module 15-SP4 (src): gcc13-13.2.1+git7813-150000.1.6.1
Basesystem Module 15-SP5 (src): gcc13-13.2.1+git7813-150000.1.6.1
Development Tools Module 15-SP4 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
Development Tools Module 15-SP5 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Package Hub 15 15-SP4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Package Hub 15 15-SP5 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP3 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP1 LTSS 15-SP1 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server for SAP Applications 15 SP1 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Enterprise Storage 7.1 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE CaaS Platform 4.0 (src): cross-nvptx-gcc13-13.2.1+git7813-150000.1.6.1, gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.1 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.2 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro for Rancher 5.2 (src): gcc13-13.2.1+git7813-150000.1.6.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 107 Maintenance Automation 2023-11-20 12:30:03 UTC
SUSE-SU-2023:4480-1: An update that solves one vulnerability, contains five features and has nine security fixes can now be installed.

Category: security (important)
Bug References: 1206480, 1206684, 1210557, 1211427, 1212101, 1213915, 1214052, 1214460, 1215427, 1216664
CVE References: CVE-2023-4039
Jira References: PED-153, PED-2005, PED-252, PED-253, PED-6584
Sources used:
Toolchain Module 12 (src): gcc13-13.2.1+git7813-1.10.1, cross-nvptx-gcc13-13.2.1+git7813-1.10.1
SUSE Linux Enterprise High Performance Computing 12 SP5 (src): gcc13-13.2.1+git7813-1.10.1
SUSE Linux Enterprise Server 12 SP5 (src): gcc13-13.2.1+git7813-1.10.1
SUSE Linux Enterprise Server for SAP Applications 12 SP5 (src): gcc13-13.2.1+git7813-1.10.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 108 Maintenance Automation 2024-01-08 20:30:23 UTC
SUSE-SU-2024:0045-1: An update that solves one vulnerability, contains five features and has nine security fixes can now be installed.

Category: security (important)
Bug References: 1206480, 1206684, 1210557, 1211427, 1212101, 1213915, 1214052, 1214460, 1215427, 1216664
CVE References: CVE-2023-4039
Jira References: PED-153, PED-2005, PED-252, PED-253, PED-6584
Sources used:
SUSE Linux Enterprise Micro for Rancher 5.4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.4 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.5 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro for Rancher 5.3 (src): gcc13-13.2.1+git7813-150000.1.6.1
SUSE Linux Enterprise Micro 5.3 (src): gcc13-13.2.1+git7813-150000.1.6.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.