Bug 1220195

Summary: Embedded Perl Interpreter Messes up Locale Environment of Calling Process
Product: [openSUSE] openSUSE Tumbleweed Reporter: Stefan Hundhammer <shundhammer>
Component: OtherAssignee: Michael Schröder <mls>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: aschnell, fvogt, mls, mvidner, snwint
Version: Current   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: Test program
Test program with our fix / workaround

Description Stefan Hundhammer 2024-02-22 10:40:27 UTC
Since about the summer of 2023, the Perl version that comes with Tumbleweed changes the locale environment of a process initializing an embedded Perl interpreter to use only 7-bit ASCII (encoding "ANSI_X3.4-1968"), so special characters in languages like Japanese, German, Czech (and probably many more) are broken: They are replaced by a question mark '?' each.

This affects GNU gettext translations as well as file output.

We experienced this in YaST, where it was reported as bug #1216689, and it was only after months of investigation and about 100 comments in Bugzilla that we found the cause.

In our case, the symptom was that messages from libstorage-ng and from Perl modules had broken special characters, but those from our embedded Ruby, from the Qt UI and from the NCurses UI were correct.

We finally found a workaround that appears to work for us: After initializing the embedded Perl interpreter, we are now switching back to the global locale with

  uselocale( LC_GLOBAL_LOCALE );

For a summary and some screenshots, see the PR with this fix:

  https://github.com/yast/yast-perl-bindings/pull/31

More details at the original bug #1216689.


We found that there was a major difference in the Perl source code's locale.c between the versions we have in Leap 15.5 and recent Tumbleweed. We could roughly it down to sometime between perl-5.26.1 (Leap 15.5) not showing the problem and perl-38.2 (TW from 02/2024) where it definitely appears, probably perl-36 as well.

The perl change log says they improved the locale handling for multi-threading; that might be where this started.


We are not sure if this is an issue for just using Perl in the regular mode (not as an embedded interpreter); we had the symptoms in our Perl code using GNU gettext translations as well.

I will attach some minimalistic test programs for easier investigation.
Comment 1 Stefan Hundhammer 2024-02-22 10:47:42 UTC
Screenshots showing the problem:

https://bugzilla.suse.com/attachment.cgi?id=870506

YaST storage proposal in Japanese.

Notice the '????' in the white box in the center; those are messages from libstorage-ng which were affected.

Notice that the Qt UI elements everywhere else are not affected; probably because Qt is initialized first, and they are using much more sophisticated methods for locale handling, not just setlocale() and nl_langinfo().


https://bugzilla.suse.com/attachment.cgi?id=872716

YaST storage proposal in Czech; same problem.

This also shows an xterm opened from within YaST with Ctrl-Shift-Alt-X which inherits the environment variables from the YaST process.


https://bugzilla.suse.com/attachment.cgi?id=872798

Screenshot of a popup dialog in German showing messages that come partly from some Perl code ("Sie haben nur Kleinbuchstaben f?r das Passwort verwendet") and Ruby code ("Möchten Sie dieses Passwort wirklich verwenden"). The messages are confirmed to be correct UTF_8 in the translation file.
Comment 2 Stefan Hundhammer 2024-02-22 10:54:28 UTC
Created attachment 872927 [details]
Test program

Output:

initial locale:  de_DE.UTF-8
initial codeset: UTF-8

codeset after perl_construct(): ANSI_X3.4-1968

locale:  de_DE.UTF-8
codeset: ANSI_X3.4-1968
Comment 3 Stefan Hundhammer 2024-02-22 11:00:13 UTC
Created attachment 872929 [details]
Test program with our fix / workaround

This just adds a

  uselocale( LC_GLOBAL_LOCALE );

after intitializing the embedded Perl interpreter, and it has the desired effect. Output:


initial locale:  de_DE.UTF-8
initial codeset: UTF-8

codeset after perl_alloc: UTF-8
codeset after perl_construct(): ANSI_X3.4-1968

switching to the global locale

locale:  de_DE.UTF-8
codeset: UTF-8
Comment 4 Stefan Hundhammer 2024-02-22 11:04:21 UTC
I have no idea if any code other than YaST is affected by this.

It may affect other programs using an embedded Perl interpreter. It may also affect Perl code that just tries to use translations via GNU gettext; this remains to be investigated.

Maybe this is also worth to be reported upstream. Or maybe it's a WONTFIX on our side.


But in any case, I wanted this to be on public record, if only for others to find our workaround.
Comment 5 Stefan Hundhammer 2024-02-28 16:10:54 UTC
Our workaround https://github.com/yast/yast-perl-bindings/pull/31 appears to cause bug #1220375 (yast2 users crashing with an idle-looping 'Zypp-main' process with 100% CPU usage).

We'll have to revert it or at least find a less aggressive approach.
Comment 6 Stefan Hundhammer 2024-03-05 15:22:18 UTC
Possibly related: bug #1220972
Comment 7 Fabian Vogt 2024-04-05 09:16:31 UTC
This is already fixed in perl and just needs a backport:

https://github.com/Perl/perl5/commit/bf38d1cf744fcc49b715b9d633761aa67436c002

See https://github.com/Perl/perl5/issues/21366
Comment 8 Fabian Vogt 2024-04-10 09:15:46 UTC
(In reply to Fabian Vogt from comment #7)
> This is already fixed in perl and just needs a backport:
> 
> https://github.com/Perl/perl5/commit/bf38d1cf744fcc49b715b9d633761aa67436c002
> 
> See https://github.com/Perl/perl5/issues/21366

ping
Comment 9 Michael Schröder 2024-04-11 13:11:27 UTC
Hmm, it's unclear if this is really the correct fix for this. Let's give it a try, though.
Comment 10 Michael Schröder 2024-04-12 11:09:37 UTC
That won't work at all. I'll try reverting 7af2d2037375d58e700f9e1b217efb2c4db66133 as suggested by upstream
Comment 11 OBSbugzilla Bot 2024-04-12 12:15:03 UTC
This is an autogenerated message for OBS integration:
This bug (1220195) was mentioned in
https://build.opensuse.org/request/show/1167043 Factory / perl
Comment 12 Martin Vidner 2024-04-23 08:30:15 UTC
Thanks, I have tested that it fixes the problem in YaST!