The statting goodness all comes from:

#0  0xb7386286 in _xstat () from /lib/libc.so.6
#1  0xb75b736c in FcFileIsDir (file=0x8139310 "/usr/share/fonts/wine/couree.fon") at stat.h:436
#2  0xb75b1664 in FcConfigAddFontDirSubdirs (config=0x8136998, d=<value optimized out>) at fccfg.c:453
#3  0xb75b169a in FcConfigAddFontDirSubdirs (config=0x8136998, d=<value optimized out>) at fccfg.c:458
#4  0xb75b1759 in FcConfigNormalizeFontDir (config=0x8136998, d=0x813bff0 "/usr/share/fonts/wine") at fccfg.c:482
#5  0xb75afaee in FcCacheReadDirs (config=0x8136998, cache=0x8137d00, list=0x8137da0, set=0x8137e40) at fccache.c:879
#6  0xb75afca5 in FcCacheReadDirs (config=0x8136998, cache=0x8137d00, list=0x8137d80, set=0x8137e40) at fccache.c:948
#7  0xb75afddf in FcCacheRead (config=0x8136998, cache=0x8137d00) at fccache.c:961
#8  0xb75b23df in FcConfigBuildFonts (config=0x8136998) at fccfg.c:281
#9  0xb75baeaf in FcInitLoadConfigAndFonts () at fcinit.c:85
#10 0xb75bafb5 in FcInit () at fcinit.c:103
#11 0xb75b1938 in FcConfigGetCurrent () at fccfg.c:362
#12 0xb75b3b8d in FcConfigSubstituteWithPat (config=0x0, p=0x8136a40, p_pat=0x0, kind=FcMatchPattern) at fccfg.c:1380
#13 0xb75b3bf3 in FcConfigSubstitute (config=0x0, p=0x8136a40, kind=FcMatchPattern) at fccfg.c:1592

It looks like this change from fccache-update-check.diff:

-	if (!FcDirCacheValid (dir) || !FcDirCacheRead (set, subdirs, dir, config))
+	if (FcDirCacheValid (dir) && FcDirCacheRead (set, subdirs, dir, config))

ie. doing the FcDirCacheRead() for every item even for a valid cache [ but not for an invalid one ? ] is the culprit.

So - the above is clearly junk ;-) it's not doing the stat on that path - but on this path instead: [ same patch ]:

-	base = file + strlen ((char *) file);
+	/* Skip this directory if already updated
+	 * to avoid the looped directories via symlinks
+	 */
+	name = FcConfigNormalizeFontDir (config, dir);

The FcConfigNormalizeFontDir does this big chunk of work - presumably the original path construction was designed to stop that being done => hence the problem.

No, you misread the patch.  It adds the disablement for the entry in ~/.fonts.cache-2 in the case /var/cache/fontconfig/* is found.

The whole stat things happen to obtain the _complete_ list of font directories.
Since you have only the list of top-level font directores like /usr/X11R6/lib/X11/fonts in your font config file, fontconfg needs to scan the dir tree and stores in the list.  Then this list is used to validate the given font path in FcConfigNormalizeFontDir().

I'll improve SuSEconfig.fonts to generate the complete
list of subdirectories already into /etc/fonts/fonts.conf,
this will avoid these stat() calls.

Thanks Mike :-)

"fontconfig" and "fonts-config" package improved according to
comment #6 submitted to STABLE/NOARCH.

Packages for testing are here:

10.0:        ftp://ftp.suse.com/pub/projects/m17n/10.0/RPMS

10.1 Beta:   ftp://ftp.suse.com/pub/projects/m17n/10.1/RPMS

Closing as FIXED.

fontconfig doesn't call stat () on font files anymore
(in the default configuration, you can still trigger stats
by adding extra directories in your ~/.fonts.conf. If you do that,
you have to add all subdirectories as well to avoid the stat () calls).

fontconfig still calls stat () on font *directories*.
I think this is no big problem though, compared the the number
of font files the number of font directories is small.

I tried running NLD10 with the updated package;

Running SuSEconfig I get:

Creating cache files for Xft .....FC: cache for /usr/X11R6/lib/X11/fonts is outdated,
    consider running "fc-cache -f" as root!
FC: cache for /usr/X11R6/lib/X11/fonts is outdated,
    consider running "fc-cache -f" as root!
... x 20 ...
but I believe that's a dup of n#148359#

OTOH - I'm still seeing the stats in my strace:

30620 open("/usr/X11R6/lib/X11/fonts/URW", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 27
30620 fstat64(27, {st_mode=S_IFDIR|0755, st_size=4944, ...}) = 0
30620 fcntl64(27, F_SETFD, FD_CLOEXEC)  = 0
30620 getdents64(27, /* 127 entries */, 4096) = 4080
30620 stat64("/usr/X11R6/lib/X11/fonts/URW/p052024l.afm", {st_mode=S_IFREG|0644, st_size=42730, ...}) = 0
30620 stat64("/usr/X11R6/lib/X11/fonts/URW/p052024l.pfb", {st_mode=S_IFREG|0644, st_size=131673, ...}) = 0
30620 stat64("/usr/X11R6/lib/X11/fonts/URW/n021023l.afm", {st_mode=S_IFREG|0644, st_size=43339, ...}) = 0
30620 stat64("/usr/X11R6/lib/X11/fonts/URW/n021023l.pfb", {st_mode=S_IFREG|0644, st_size=108217, ...}) = 0
30620 stat64("/usr/X11R6/lib/X11/fonts/URW/n021023l.pfm", {st_mode=S_IFREG|0644, st_size=2976, ...}) = 0
30620 stat64("/usr/X11R6/lib/X11/fonts/URW/hrgks.gsf", {st_mode=S_IFREG|0644, st_size=7036, ...}) = 0
30620 stat64("/usr/X11R6/lib/X11/fonts/URW/n019003l.afm", {st_mode=S_IFREG|0644, st_size=42882, ...}) = 0
...

[ unfortunately I also have 3 force installed versions of fontconfig with the same version in my rpmdb ;-( so - perhaps this is just me ].

Also - conceptually - why do we need to stat the font directories ? must confess, I'm fairly confused here - we used to have these fonts.cache-2 files all through the font directories, now we shove data into /var/cache/fontconfig - *presumably* to save on walking the whole directory hierarachy & reading from a ll over the disk at startup - instead just reading from 1 directory; and a few clustered files (?)

why then do we need to stat stuff outside ~ ? if someone installed fonts on the system without running fc-cache; that's a bug in their packaging (surely) and not one we need to try to detect on every app start - but perhaps I misunderstand again :-)

Michael Meeks> I tried running NLD10 with the updated package;
Michael Meeks> 
Michael Meeks> [...]
Michael Meeks> 
Michael Meeks> OTOH - I'm still seeing the stats in my strace:

Maybe you forgot to update fonts-config as well?

"SuSEconfig --module" fonts calls the perl script /usr/sbin/fonts-config.

This script has been updated to generate 

    /etc/fonts/suse-font-dirs.conf

Do you have that file? It should contain a *complete* list
of font directories.

Michael Meeks> OTOH - I'm still seeing the stats in my strace:
Michael Meeks> 
Michael Meeks> 30620 stat64("/usr/X11R6/lib/X11/fonts/URW/p052024l.afm",
                                                      ^^^

I think that the URW directory shows up here is a proof that you
didn't update "fonts-config". The perl script resolves the symbolic
link 

mfabian@magellan:~$ ll  /usr/X11R6/lib/X11/fonts/URW
lrwxrwxrwx 1 root root 28  2月  3 14:27 /usr/X11R6/lib/X11/fonts/URW -> /usr/share/ghostscript/fonts/
mfabian@magellan:~$

I.e. /usr/share/ghostscript/fonts/ will be searched by fontconfig directly
instead of following the symbolic link.

Michael Meeks> now we shove data into /var/cache/fontconfig -
Michael Meeks> *presumably* to save on walking the whole directory
Michael Meeks> hierarachy & reading from a ll over the disk at startup
Michael Meeks> - instead just reading from 1 directory; and a few
Michael Meeks> clustered files (?)

No, this is just for compliance with FHS. According to FHS, it should
be possible to mount /usr readonly, i.e. it might not be possible
to generate cache files there. Therefore cache files are generated
in /var/cache/fontconfig.

But fontconfig must check whether the cache files in /var/cache/fontconfig
are up-to-date, therefore it must compare with the timestamps of the
directories where the fonts are.

Michael Meeks> why then do we need to stat stuff outside ~ ? if
Michael Meeks> someone installed fonts on the system without running
Michael Meeks> fc-cache; that's a bug in their packaging (surely) and
Michael Meeks> not one we need to try to detect on every app start -
Michael Meeks> but perhaps I misunderstand again :-)

Fontconfig *does* try to detect this on every app start.

Sadly - no, I don't have /etc/fonts/suse-font-dirs.conf - so some function of force installing those packages must have got screwed up - urgh;

Is there a good way when building RPMs to test oneself of ensuring this won't happen ? ie. sub-versioning the existing strike version in some way ?

Either way - I'm still *highly* unconvinced that fontconfig must check whether the system cache files are up-to-date it's just a recepie for disaster IMHO.

* If you check the system font directories - -at-best- you discover that they have been updated without the cache being re-generated [ ie. mid-package transaction, or some-broken-system-package ].
   * Once you have spent some chunky I/O time discovering this fact what do you do ? complain about it ? - how valuable is that except in a debugging build ?
       * If you do something except complain - you then have to generate a large per-user cache instead that chews more I/O time, disk-space etc. & has to be maintained in parallel - causing myriad bug reports & problems etc.

In summary - I still don't get the rational here ;-) Of course, statting & updating ~/.fonts -R is a great plan; but trying to 2nd guess the system & repair any mis-configuration in the user's directory burning tons of time there doesn't seem to do anyone any favours (surely) ?

Michael Meeks> Sadly - no, I don't have /etc/fonts/suse-font-dirs.conf
Michael Meeks> - so some function of force installing those packages
Michael Meeks> must have got screwed up - urgh;

But you have installed the latest "fonts-config" package?

Then calling "SuSEconfig --module fonts" should create /etc/fonts/suse-font-dirs.conf.

What is the latest changelog entry of your "fonts-config" package.
It should be:

* Tue Feb 07 2006 - mfabian@suse.de
- Bugzilla #148361: generate /etc/fonts/suse-font-dirs.conf from
  a template containing glob patterns to a full list of all
  font directories to avoid that fontconfig calls stat () on all
  files in all sub-directories.

Michael Meeks> * Once you have spent some chunky I/O time discovering
Michael Meeks> this fact what do you do ? complain about it ?

No, fontconfig doesn't complain, it just builds a large ~/.fonts.cache-2 then.

Michael Meeks> * If you do something except complain - you then have
Michael Meeks> to generate a large per-user cache instead

That is indeed done.

Michael Meeks> that chews more I/O time, disk-space etc. & has to be
Michael Meeks> maintained in parallel - causing myriad bug reports &
Michael Meeks> problems etc.

This worked nicely with fontconfig before the binary caches were
introduced.

All these bugs only appeared after the introduction of the binary caches
because the cache code became so complicated.

It'd be nice to have a font.config option to specify whether to validate caches.

There are many many many rooms to optimize if you see the code.  But, stability is much more important right now :)

Sure - of course, the un-patched fontconfig behavior is to assume that the system is correctly setup (AFAIR) and not check that - since there's nothing very helpful we can do about it :-) Anyhow - perhaps 1 for another day - it is indeed a lot better now.

Thanks.

I did a quick hack right now, just for fun.  The packages are found at mbuild/bari-tiwai-103.

A new config option "validatecache" is added.
Add the following to ~/.fonts.conf
<config>
  <validatecache>
    <bool>false</bool>
  </validatecache>
</config>
then fontconfig trusts the existing cache fully, and never call excessive stat.  (stat is called once per eacn directory to compare the inodes, though.)

Created attachment 67086 [details]
A patch to add validatecache config option

Because fontconfig trusts the caches fully when this patch
and this option are used, crashes like this can happen:

    mfabian@magellan:~$ xfd -fa "SUSE Sans"
    mfabian@magellan:~$

works. Now move the fonts out of the way:

    mfabian@magellan:~$ sudo mv   /usr/X11R6/lib/X11/fonts/truetype/SUSE* /tmp
    mfabian@magellan:~$

The cache files still contain entries for these fonts, and as fontconfig
trusts the cache files, the following will crash:

    mfabian@magellan:~$ xfd -fa "SUSE Sans"
    Warning: Cannot convert string "SUSE Sans" to type XftFont
    セグメンテーション違反です (core dumped)
    mfabian@magellan:~$

This patch and option do indeed reduce the number of stats () calls:

    mfabian@magellan:~$ wc /tmp/old-stat
      3651  22214 333863 /tmp/old-stat
    mfabian@magellan:~$ wc /tmp/new-stat
      3338  20340 304959 /tmp/new-stat
    mfabian@magellan:~$

(old-stat and new-stat are files produced by "strace -estat -o
/tmp/old-stat fc-match sans" on my system).

But the gain is not very big.

I'll apply Takashi's patch nevertheless because it is useful
if one is able to make sure that all caches can be trusted.

Takashi's patch submitted upstream for discussion.

Changing Product to SUSE Linux 10.1 because it is not only NLD related.

Hi Mike:

>    mfabian@magellan:~$ xfd -fa "SUSE Sans"
> works. Now move the fonts out of the way:
>    mfabian@magellan:~$ sudo mv   /usr/X11R6/lib/X11/fonts/truetype/SUSE* /tmp
> The cache files still contain entries for these fonts, and as fontconfig
> trusts the cache files, the following will crash:

So - that is a bug in fontconfig - for sure; it should never crash. Are you certain that bug is present in the pristine fontconfig ? of coure - at that point IMHO it should warn like anything :-) "your system administrator is trying to kill your apps" or somesuch :-)

But that's really an orthogonal issue to this (deeply problematic) capability of trying to work around a borked system font setup by doing something yet more complicated, error prone & most likely extremely slow ;-)

Don't worry, this feature will be never turned on as default :)
It's for someone who is really sensible to speed, or a very uncommon situation like all directories are read-only-mounted.

This feature/bug is not in the upstream fontconfig, it is added by
Takashi's patch from comment #21.

But, as Takashi says, this feature is only to get another performance
improvement when you are *absolutely* sure that all caches are correct
and need not be checked. If the caches are not correct, it is "normal"
that crashes will happen.

But as we don't turn on that feature by default, it is OK in my opinion.

Move bug to openSUSE 11.2 to make it publicly visible.