Bugzilla – Bug 156045
fonts for "exotic" languages are often missing after initial reboot
Last modified: 2008-06-25 09:53:45 UTC
See also bug #155667. We often have the problem, that fonts for "exotic" languages are missing during the second phase of the installation because the packages containing these fonts are missing on CD1. During the first phase of the installation, everything works because the fonts are available in instsys (installation image). But after the installation of CD1 is finished, the initial reboot happens and the installation of the other CDs continues in the installed system. If the fonts for language X were missing on CD1, one will see boxes. In this bugreport I want to discuss the possible solutions for this problem.
*** Bug 155667 has been marked as a duplicate of this bug. ***
Until now the solution for this problem has been: For all languages supported during installation, make sure that the font-package which contains the font used in instsys for that language is on CD1. Problems with that solution: font-packages for some languages, especially CJK languages can be huge and space on CD1 is very "expensive" there is lots of other stuff besides fonts which absolutely needs to be on CD1 to make the installation work correctly. Therefore, it often happens that a font-package gets removed from CD1 and then the bug "only boxes are displayed for language X after the initial reboot" comes back. This happened again and again in the past, therefore it cannot be a good solution to require all these font packages on CD1.
One solution suggested was to split the font packages which contain collections of fonts into sub-packages so that for each language only the sub-package which contains the font needed for the installation needs to be on CD1. That saves some space on CD1. Problems with that solution: - still needs quite a lot of space on CD1, even the single fonts can be big Japanese: sazanami-gothic.ttf 7.4 M simplified Chinese: FZSongTi.ttf 17.0 M traditional Chinese: FZMingTiB.ttf 12.0 M ... - it does not solve the problem that the fonts may disappear without warning from CD1 if CD1 is full and something has to be removed because the people who create the CDs cannot know that these fonts are needed. - one always needs to add the fonts in 2 places: + in instsys + on CD1 this is very inconvenient when adding fonts for new languages because one has to talk both to Steffen Winterfeld for the instsys *and* to the people who create CD1. When splitting or renaming font-packages it is also always necessary to inform both Steffen and the people creating CD1.
Because requiring the font packages on CD1 is problematic, we thought of a different solution. As the fonts are already there in instsys, why not just copy them from instsys into the installed system?
Obvious advantage of copying the fonts from the instsys to the installed system: CD1 doesn't need to contain any fonts at all, which saves a lot of space on CD1 and removes to burden to track which fonts need to be on CD1 and to check that these are always there and don't get removed.
I think copying *all* fonts from instsys into the installed system is the way to go because it is easy. Of course one could try to copy only individual fonts depending on the language choosen for installation: "If language A copy only font B, of language=C, copy only font D, ...". But that makes it complicated and error prone. Therefore I think just copying *all* fonts is much better.
IPAGothic might be problematinc in this case...
We could delete these fonts when installation is finished.
Copying all fonts has some problems as well though (minor in my opinion): Let's first assume all fonts from instsys are copied to the same place in the installed system, e.g. everything from /usr/X11R6/lib/X11/fonts/truetype/, /usr/X11R6/lib/X11/fonts/misc, /usr/X11R6/lib/X11/fonts/uni is copied to the same place in the installed system. And let's assume that the installation is done in Chinese. Then the Chinese font packages will be installed at some point during the installation. Not from CD1, but at some point they will be installed. That will overwrite the Chinese fonts which have been copied from instsys. This is no problem at all, to the contrary, after rpm has overwritten /usr/X11R6/lib/X11/fonts/truetype/FZSongTi.ttf, "rpm -qf /usr/X11R6/lib/X11/fonts/truetype/FZSongTi.ttf" will nicely show that this file belongs to the package "ttf-founder-simplified". But fonts from languages which have not been selected as either the primary or a secondary language, may not be overwritten during the installation. For example, if Khmer has not been selected as a supported language, /usr/X11R6/lib/X11/fonts/truetype/KhmerOSsys.ttf will nevertheless exist in the installed system and "rpm -qf /usr/X11R6/lib/X11/fonts/truetype/KhmerOSsys.ttf" will say "this file does not belong to any package". In my opinion, this is not only bad, it also has a big advantage because it would make sure that a minimal set of fonts is always there which covers "almost everything", i.e. one will almost never see boxes when looking at web-pages in foreign languages, Unicode sample pages, etc. ... We already discussed a few times that it might be nice if such a minimal set of fonts to cover "almost everything" were installed by default. We also considered that having a single font covering almost all of Unicode (like e.g. CODE2000) might be nice to have and nice to be installed by default for that reason. A set of fonts which covers "almost everything" is even better than a single font because the fonts are designed specifially for certain languages and are thus usually much more beautiful compared to a big font to cover everything. Such a big font is always a compromise. (Adding Guy Lunardi <glunardi@novell.com> to CC: because he was also interested in having fonts (or a big font) to cover "almost everything" installed by default).
I think Mike's suggestion is the easiest way in short term. However I'm just wondering that if it's possible to eliminate the system rebooting after installing CD1, so that we could install all necessary packages before rebooting.
I'd rather have all these fonts in a separate directory, say .../fonts/yast or .../fonts/install to make it clear where the files are from. It would very much annoy me running rpm -qf on some font file discovering it doesn't belong to any package.
You cannot remove CD1 when you have little main memory (approx <256MB) because then yast runs directly from CD. Hence the reboot.
Like Steffen suggests, instead of copying the fonts to their "usual" place in the installed system, it is just as easy to copy them to some arbitrary other place in the search path of fontconfig. For example cp -r /usr/X11R6/lib/X11/fonts /mnt/usr/share/fonts/yast ^ | whereever the installed system is mounted and just delete the complete directory /usr/share/fonts/yast again when the installation is finished. On top of the advantage Steffen mentions (rpm -qf will not complain "does not belong to any package" on some font files), it also has the advantage that fonts which are not completely free (like the IPA fonts Takashi mentions in comment #7), will not remain on a system where OpenSuSE OSS is installed (free software only). Disadvantage is of course that one misses the opportunity to have a nice, minimal default set of fonts to cover almost all languages always available.
If we should choose to leave the fonts copied from instsys there after the installation, we need to remove all fonts from Instsys which might cause license problems if they remain after the installation. I.e. in that case, IPAPGothic would have to be removed from instsys and we would need to replace it with sazanami-gothic.
We should be careful at which time to remove */yast/* fonts (if we delete them after installation). YaST will continue to set up network and other stuff.
And, if we should choose to leave the fonts there after the installation but use a different directory like /usr/share/fonts/yast, another minor problem is that we will have duplicates then. For example we will have /usr/share/fonts/yast/FZSongTi.ttf and /usr/X11R6/lib/X11/fonts/truetype/FZSongTi.ttf apart from wasting disc space, this doesn't cause problems though.
I'd prefer to remove all */yast/* fonts after installation, instead of having duplicated fonts.
Takashi Iwai> We should be careful at which time to remove */yast/* Takashi Iwai> fonts (if we delete them after installation). YaST will Takashi Iwai> continue to set up network and other stuff. Yes, the directory must not be removed before *all* packages are installed *and* YaST exits. If YaST is started again after all packages are installed, it is OK because then the regular font packages are in place. But as long as (the Qt-)YaST is still running after the intial reboot, the fonts must not be deleted, YaST will not be able to use fonts which have been installed while it was already running.
Zhe Su> I'd prefer to remove all */yast/* fonts after installation, Zhe Su> instead of having duplicated fonts. Yes, I agree, if a new directory like /usr/share/fonts/yast is used it is probably better to remove it. I just remebered that duplicated fonts may cause problems if they have different versions. For example, if the package containing FZSongTi.ttf is updated and gets a newer version of FZSongTi.ttf in /usr/X11R6/lib/X11/fonts/truetype/FZSongTi.ttf, the old one in /usr/share/fonts/yast/FZSongTi.ttf will still be used because /usr/share/fonts/ is searched earlier by fontconfig than usr/X11R6/lib/X11/fonts/. I.e. if the fonts are not exact duplicates there is a problem.
If the fonts are copied to the "regular" directory where the files in the packages are as well, the problem with the duplicates will not occur either, only the "rpm -qf" "not owned by any package" problem with fonts for languages which were not selected.
What about an extra rpm - lets call it "installation-fonts", who contains the packages? This wouldn't break the rpm database and should be easier to maintain.
If the "installation-fonts" rpm has the fonts in the same place as the other font rpms, there will be file conflicts. If the "installation-fonts" rpm has the fonts in /usr/share/fonts/yast, we have the problem with duplicate fonts in the system (problem when updating font packages to new versions).
On top of that, an extra "installation-fonts.rpm" would still need to be on CD1 and would still waste a lot of space there, see comment #3, the sizes of some CJK fonts are quite big.
Jiri, please, continue with the discussion from the YaST installation point of view.
Here another thought to solve this: (one of these "you go to bed and your brain is still working" ideas :-) 1. copy the fonts from the installation system to a folder of the installed system which is not one of the normal font folders. 2. create links from the normal font folder(s) to the copied fonts. 3. run font configuration to make the fonts known 4. Let rpm overwrite the links during installation with real files. 5. clean up by removing the remaining links and the separate folder. 6. run font configuration again to remove the fonts that are gone now. I think this could solve all mentioned conflicts and would leave no traces. Of course this does only work in a filesystem that supports links. And it creates at least one additional font configuration run at the end.
Jens> 1. copy the fonts from the installation system to a folder of the Jens> installed system which is not one of the normal font folders. Jens> 2. create links from the normal font folder(s) to the copied fonts. This has no advantage at all over copying the fonts into say "/usr/share/fonts/yast" which is already in the search path of fontconfig. Jens> 3. run font configuration to make the fonts known Not necessary, when YaST2 starts and "fc-cache" has not yet been run, a cache is created in /root/.fonts.cache-2 which works just as well as the caches in /var/cache/fontconfig. Creating the caches in /var/cache/fontconfig is just useless overhead if the only program using the fonts is YaST2 and the fonts will be removed anyway when YaST2 exits. Jens> 4. Let rpm overwrite the links during installation with real files. Jens> Jens> 5. clean up by removing the remaining links and the separate folder. Instead of creating symlinks, you could just as well copy all the fonts into /usr/X11R6/lib/X11/fonts/truetype directly. If you want to clean up fonts afterwards which have not yet been overwritten by rpm, you could just remove all files in /usr/X11R6/lib/X11/fonts/truetype where rpm says that they do not belong to any package. Jens> 6. run font configuration again to remove the fonts that are gone now. Not necessary either. Jens> I think this could solve all mentioned conflicts and would leave Jens> no traces. Of course this does only work in a filesystem that Jens> supports links. And it creates at least one additional font Jens> configuration run at the end. There is no real difference to what I already suggested above. Copying the fonts to /usr/share/fonts/yast, then starting YaST2 and removing the directory /usr/share/fonts/yast just before YaST2 exits also doesn't create any conflicts and leaves no traces and no extra run of fc-cache is necessary either.
If we want to copy the inst-sys fonts into the installed system this can be done in "x11_finish.ycp" IMHO. There you can find the following code: if (x11_installed) { if ( Mode::update () ) { ... update case } else { //======================================== // Installation mode... //---------------------------------------- // make current X11 configuration available in installed system // copy /etc/X11/XF86Config from inst-sys to [/mnt]/etc/X11/xorg.conf // --- ... ADD FONT COPY CODE HERE } } I'm not sure what could be the right place to remove the fonts again. I assume this can be done in inst_finish but I think Jiri knows it better
Increase severity to "BLOCKER". Can we please fix this in time for SuSE 10.1?
At first, we have DVDs and network installation, there this problem does not exist. Second, changing it now has a huge impact on SL10.1 RC1 next week, so aj has to decide if he will take the risk. Else for this escalation it is far too late.
When implemting this (in my opinion rather easy method), we can save *huge* amounts of space on CD1 and this was always a big problem.
This should be done post 10.1 - so that it's in 10.2 alpha1.
Then we should make sure that at least the Chinese, Japanese, and Korean font packages containing the fonts used in the instsys are on CD1, otherwise we have a regression compared to SuSE 10.0 and 9.3.
Which fonts are used for Simplified Chinese and Traditional Chinese in instsys? As you know we don't have Founder fonts (FZSongTi etc.) in SUSE Linux OSS, those files are in AddOn disc. So for Simplified Chinese and Traditional Chinese we can only put Arphic fonts in CD1. So I think we can use Arphic fonts in instsys as well (if FZSongTi was used before). Arphic fonts are much smaller than Founder fonts. For Simplified Chinese, we can use gbsn00lp.ttf (5192076 bytes), for Traditional Chinese, we can use bsmi00lp.ttf (12643248). FZSongTi is much larger (17563828).
Currently we use FZSongTi and FZMingTiB for Chinese in instsys. We can change in instsys to the Arphic fonts, but I don't think this is absolutely necessary. But we should make sure that the Arphic fonts are available on CD1. If we continue to use FZSongTi and FZMingTiB in instsys, then the appearance of the fonts will change slightly after the 1st reboot, but I think this is no big problem.
Rudi, please, fix the CD layout. Then reassign back to jsrain@suse.de for tracking for the next release.
By using Arphic fonts in instsys, we can have more free space in CD1 (about 11M).
OK, I'll talk with Steffen Winterfeld to use the Arphic fonts in instsys. Then we should also switch from IPAGothic to "Sazanami Gothic" if we want to stick to completely free fonts in instsys.
Grumble, sazanami-gothic would require 5MB bigger space in instsys than IPA-gothic.
I know. The question is whether we can accept fonts in instsys which are not completely free or not.
Currently in instsys: /usr/X11R6/lib/X11/fonts/truetype/FZSongTi.ttf 17M (simp. Chinese) /usr/X11R6/lib/X11/fonts/truetype/FZMingTiB.ttf 12M (trad. Chinese) /usr/X11R6/lib/X11/fonts/truetype/ipagp.ttf 2.8M (Japanese) ==== 31.8M Please replace with: /usr/X11R6/lib/X11/fonts/truetype/gbsn00lp.ttf 5M (simp. Chinese) /usr/X11R6/lib/X11/fonts/truetype/bsmi00lp.ttf 13M (trad. Chinese) /usr/X11R6/lib/X11/fonts/truetype/sazanami-gothic.ttf 7.4M (Japanese) ===== 25.4M I.e. this change saves 6.4M and avoids all fonts with commercial licenses. In order to work around bugs in Qt3, we currently delete the Chinese fonts FZSongTi.ttf and FZMingTiB.ttf if Japanese has been selected in instsys. Please make sure to delete gbsn00lp.ttf and bsmi00lp.ttf instead after the change.
Cleared up the setup for CD1. On CD 1: - KhmerOS-fonts - dejavu - fonts-arabic - ghostscript-fonts-other - ghostscript-fonts-rus - ghostscript-fonts-std - indic-fonts - sazanami-fonts - ttf-arphic-bsmi00lp - ttf-arphic-gbsn00lp - unfonts - xorg-x11-fonts-cyrillic - xorg-x11-fonts-75dpi - xorg-x11-fonts-100dpi - xorg-x11-fonts-scalable Moved to other CDs: - bitstream-vera - cm-unicode - efont-unicode - ttf-arphic-bkai00mp - ttf-arphic-gkai00mp This setup should hopefully work so that nobody has to see boxes after the first reboot and the ISO size is not too big. Please check this with Beta9.
Will SLED/SLES 10 and SUSE 10.1 use the same instsys? As you know we use ttf-founder fonts in SLED and SLES, so if we use Arphic fonts in their instsys, then the font will be different between first stage and second stage installation. So if it's possible, I suggest to continue to use Founder fonts in SLED/SLES's instsys.
Yes, of course SLES/SLES and SuSE 10.1 use the same instsys. That's the reason why I wanted to keep the Founder fonts (and IPAGothic for Japanese) in the instsys.
Actually, that is not true. SLES/NLD/SL can have (to a certain degree) different inst-syses-
But I don't think it makes any sense to have different fonts in instsys for SLES and SuSE 10.1. We have enough problems with the font handling in Qt already and already need ugly workarounds like deleting the Chinese fonts when Japanese is selected. If the we have different fonts in instsys for SLES and SuSE 10.1, we will have different problems and need even more testing.
This is not a blocker.
comment 41 does not work at all
*** Bug 162064 has been marked as a duplicate of this bug. ***
as Beta 9 with "new" fonts on CD1 is released: please report if this setup is good enough for 10.1 Goldmaster.
So the fonts in instsys are not changed yet?
No, Steffen Winterfeldt tried to change to the Arphic fonts but he told me that this resulted in display of boxes only for Chinese (see his comment #48). Most likely this is caused by another font bug, probably in QT. I think we should keep the instsys as it is to avoid causing new problems which are difficult to solve that late in the beta phase. And we should avoid to have different fonts in instsys for SLES/SLED/SuSE Linux because we will probably not be able to debug problems in several different setups until the release.
Any news here for Beta9? => Does the solution with "old instsys" and "new fonts on CD1" work? I want to set this bug to "LATER" to remind us for the next release - but be shure that we have a working solution for 10.1.
still in needinfo ... Mike: any news ?
Until now I had only time to test that it does *not* yet work for Khmer, but this is due to a different problem, see bug #162064.
It doesn't work for Japanese either, the package "sazanami-fonts" is not installed after the initial reboot, although the package is available on CD1.
package ttf-arphic-bsmi00lp is missing on CD1 (RC1). Needed for traditional Chinese.
This is true for x86_64 - not for i386. We have not enough space on x86_64 - so some packages are moved to CD4.
OK, too bad. We should really implement the copy solution for SuSE 10.2, otherwise this problem with the fonts and the lack of space on CD1 will appear again and again.
OK. So as we have a solution for code10 now I set this to later...
According to https://bugzilla.novell.com/show_bug.cgi?id=304959#c14 created Fate #302548
*** Bug 399829 has been marked as a duplicate of this bug. ***
mass reopening all SuSE Linux bugs that are set to REMIND+LATER to change the resolution to WONTFIX (adapting to new policy)
Closing old LATER+REMIND bugs as WONTFIX - if you still plan to work on it, feel free to reopen and set to ASSIGNED. In case the report saw repeated reopen comments, it's due to bugzilla timing out on the huge request ;(