Bug 147966

Created attachment 66324 [details]
Log of strace -t

This is probably the fontconfig problem (bug #147292). Updated packages for Beta 3 are available here:
http://download.opensuse.org/distribution/SL-10.1-OSS-beta3/fixed-rpms

Please reopen this bug if this does not fix the problem.

No, fontconfig is alredy the latest one.

Maby the problem isn't as fixed as it should be... I will mark this one as a duplicate because this is most likely the cause here (the original bug #143715 is still not marked as solved).

*** This bug has been marked as a duplicate of 143715 ***

Has anyone noticed that Acroread 7.0.5 is still slow to start?  In Beta 4 now, the fontconfig problem seems solved....all X aplications start normalle EXCEPT Acroread.  This is true for SUSE 10.0 AND 10.1 for Acroread 7.0.5.  Acroread 7.0.1 starts normally in BOTH versions of SUSE...something is different with 7.0.5.  Can someone please check this and confirm.  I still think that there is an issue with acroread.  For information, Acroread 7.0.5 seems to start up fine in CentOS 4.2 but NOT SUSE 10.0 or 10.1 beta.

Also, once acroread finally starts, it seems to work fine...it's just the start-up time and a LOT of hard drive activity....this has been confirmed with several computers for SUSE 10.0 and the test computer with SUSE 10.1 beta through 4 (with the fixed fongconfig issue).

It still happens on beta4, so it's definitely not that fontconfig problem.

The oprofile shows that it takes most time in libBIB.so (in Acrobat7/*/*/lib).

Is acroread still used as the default browser on NLD or any else?  If yes, we should increase the severity of this bug to CRITICAL or BLOCKER.

Acroread should definitely work... let's see if other people also experience problems here.

Takashi: Are you opening the bare program or are you trying to open a file? Does it always take so long to start?

Just starting "acroread".  Yes, it always take minutes to appear the window.

BTW, do you know any machine with 10.1 on which acroread is working without problems?  It'll be helpful to track down the problem...

This appears not to be an isolated problem. Raising severity.

Created attachment 69616 [details]
strace output

On my computer start of strace -o log acroread took aprox 56 seconds. I have Beta 4.

OK, I found a workaround.

Acroread creates a stupidly long list of all available fonts in ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst.  Making this an empty file reduces the start-up time significantly.  That is,

    % rm ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst
    % touch ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst

I'm not 100% sure whether any side-effects occur, but I bet it won't.  At least, browsing PDF files seems OK.

When does Acroread generate that list? At the (first) startup?

Created attachment 69648 [details]
nld10 beta4.2 -- strace -t 

No problem here:
acroread starts in less than 10 seconds on my NLD beta4.2, it is a poor PIII 800Mhz machine. 

As fast as my SL10.0 acroread on a more powerful machine.

#14: I think yes.  Possibly, a cache file generated by a version with a broken version/combination did something wrong.

#15: A possible difference is that home is NFS or not - or the number of fonts you installed.  Anyway, try to make the file mentioned in #13 empty.  It can be much faster :)

Did you check whether it depends on an existing old ~/.adobe/ directory?
I.e. does it work well after "mv ~/.adobe ~/.adobe.old"? 

No, it happens even after removing the whole ~/.adobe.

Acroread recreates ~/.adobe/* at the frist start.  Interestingly, the first start is really quick.
Then, start acroread again, and it takes minutes.

George,
as far as I know our special license for the Adobe Reader,
we should be allowed to adapt the start script
/usr/X11R6/bin/acroread
to workaround the problem, e.g. by adding something like
cat /dev/null 1>~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst 2>/dev/null || true

I believe we can add that work around as long as the consensus is that we won't be breaking any functionality.  Will it still be able to handle all the fonts it needs? If so I think its OK. 

The fonts included in acroread package are listed in AcroFnt07.lst.  It seems that UnixFnt07.lst contains only the list of external fonts.

Though, I don't know whether these external fonts are really used or not.

Also I'm not sure whether we should add this workaround _unconditionally_.  Acroread is working without hack for some (or most?) users...  Maybe more investigation is needed.

I can reproduce the problem with the slow startup as well.

Takashi> Acroread creates a stupidly long list of all available fonts in
Takashi> ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst.  Making this an empty file reduces
Takashi> the start-up time significantly.  That is,
Takashi> 
Takashi>     % rm ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst
Takashi>     % touch ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst
Takashi> 
Takashi> I'm not 100% sure whether any side-effects occur, but I bet it won't.  At
Takashi> least, browsing PDF files seems OK.

I can get the same speedup by only removing UnixFnt07.lst, the touch
is not necessary.

When using the touch, the file UnixFnt07.lst still has 0 bytes
when acroread has started.

After deleting the file but omitting the touch, Acroread starts
equally fast and when it has started the file is not empty.

Therefore I guess by omitting the touch one can reduce the probability
for side-effects.

Takashi> Acroread recreates ~/.adobe/* at the frist start.
Takashi> Interestingly, the first start is really quick.  Then, start
Takashi> acroread again, and it takes minutes.

I.e. acroread seem to be able to generate the file UnixFnt07.lst quite
fast. Only if the file is already there the slowness problem occurs.

So do I understand you guys correctly when assuming we should simply always
remove that file upon startup, so acroread regenerates it every time, instead
of using time evaluating Cache data?

Yes.

So I put in the acroread script:
rm ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst

I get these warnings...  
----------
(acroread:20359): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
FC: cache for /usr/X11R6/lib/Acrobat7/Resource/Font is outdated,
    consider running "fc-cache" as root!
FC: cache for /usr/X11R6/lib/Acrobat7/Resource/Font/PFM is outdated,
    consider running "fc-cache" as root!

The Gtk-WARNING is there regardless, if I run "fc-cache" as root then it doesnt complain after that...  Do you think I should check it in this way?  The speed is faster for loading, but on my fast box it goes from about 10 seconds to 3

I can confirm that adding that rm line to the acroread script inproves the startup time on my machine.  Running fc-cache as root also seems to improve it.
?
Perhaps fc-cache should be run at the end of SuSEconfig.fonts?

George Horlacher> FC: cache for /usr/X11R6/lib/Acrobat7/Resource/Font is outdated,
George Horlacher>     consider running "fc-cache" as root!
George Horlacher> FC: cache for /usr/X11R6/lib/Acrobat7/Resource/Font/PFM is outdated,
George Horlacher>     consider running "fc-cache" as root!

These warnings have been disabled in fontconfig for the final release
anyway.  We added a patch to fontconfig to produce these warnings only
for debugging purposes during the beta phase. You can ignore these
warnings.

George Horlacher> The Gtk-WARNING is there regardless, if I run
George Horlacher> "fc-cache" as root then it doesnt complain after
George Horlacher> that...  Do you think I should check it in this way?

Yes.

James Ogley> Perhaps fc-cache should be run at the end of SuSEconfig.fonts?

This is already done.

SuSEconfig.fonts does nothing but call the Perl-script /usr/sbin/fonts-config
which does the main work.

And fonts-config does run "fc-cache".

It is run *after* generating fonts.dir/fonts.scale, because running
"fc-cache" before that would be meaningless because changing
fonts.dir/fonts.scale changes the time-stamps again and fontconfig
will still think that fc-cache needs to be run.

But it is not run at the end of fonts-config because fonts-config does
a few things which already use fontconfig (generation of fontmap for
Ghostscript, generating Java fontsetup, ...) and this would be slow
if done before running fc-cache.

Suse Linux 10.1, beta8 (updated from 10.0):

a) Out of the box:
   Acroread starts in about 30s.
b) When removing and touching ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst
   Acroread starts in about 4s, that file stays at 0byte. 
c) When removing ~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst
   Acroread starts in about 4s, that file is created with 2MB (GOTO a))

I do not understand the "GOTO a" in the above comment
because each start of the Adobe Reader should run the
/usr/X11R6/bin/acroread start-script which removes the file again.

But removing the file could cause another problem:
What happens when the same user runs the Adobe Reader
several times concurrently (in particular what happens
when the browser plugin runs concurrently)?

The first Adobe Reader process would remove and re-create the file
and I assume Adobe Reader now depends on the content of the file
but a subsequent Adobe Reader process would remove and re-create it
with perhaps different content (when fonts have changed)
or the first process may try to read from the file just when it
is removed but not yet fully re-created.

Therefore it is perhaps more safe to do only a
  cat /dev/null >~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst
which only clears the content without any file removal.

As I wrote in comment #22, I guess that making the file empty
instead of removing it will be more likely to cause side effects.
This is just a guess though.

Nor do I (understand the GOTO). But it's the way it is. Update problems? Not sure what the script should look like, I get:

grep " rm " /usr/X11R6/bin/acroread
    rm -f $CERTNAME
    rm -f $CERTDATA
    rm -f $LOGFILE
(acroread-7.0.5-6)

We could also simply link this file to /dev/null...

I guess if acroread goes through all the trouble to generate such
a file, it probably has a purpose. Therefore I guess
it might cause side effects to make it empty, link it to /dev/null,
or similar things.

Johannes has a good point.  The acroread plugin shows the very same problem if the cache file exists, and there is no wrapper script for the plugin.  Setting a zero file is a good workaround, IMO (although if you use only plugin, this workaround will be never enabled).

BTW, looking at strace, acroread creates the file with O_EXCL, i.e. once after the file is created, it'll be never rewritten.  This implies that acroread even doesn't care about the changes of system fonts once if they are registered in that cache file.  A bad implementation :-<

I've submitted to autobuild an updated with the fix:
cat /dev/null >~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst

In testing it looks like the startup speed is really fast now.
I'll also work with acroread on this as I'm starting to get a look 
at the 7.0.7 pre-release canidate to see if it's a problem for it too.

RC3:
First acroread start after booting takes 90 (!) seconds on a IBM R50p which is rather new hardware. A second start takes 2 seconds.
If I do cat /dev/hda > /dev/null, so that the acroread files are thrown out of the cache, the next acroread start takes again 90 seconds.

I've looked in 10.1 RC3 acroread startup file and it has:
cat /dev/null >~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst

It seems to be working well for me speed wise.  If it's missing for 
you please verify your running the latest version and if you have 
that line in the /usr/X11R6/bin/acroread script (and that is the one your 
using to start up acroread). The package I got from 10.1 RC3 is: acroread-7.0.5-13.i586.rpm

This issue still exists in SUSE Linux 10.1 RC5. Though SUSE Linux 10.1 is GM, please fix it in SLE 10.

Have you verfied that the fix "cat /dev/null >~/.adobe/Acrobat/7.0/Cache/UnixFnt07.lst"  in the acroread startup script is not working?  Have you tried with and without that line in the startup script?  From my testing I saw a lot quicker results with that line added.  I wonder if an upgrade or some install issue is preventing the acroread script from getting the patch.  Can you confirm that the patched acroread script is there and how it runs with/without that line in it?

The 10.1 released version of /usr/X11R6/bin/acroread does contain a fix for this issue, and in fact changes the startup time from over a minute to only about 8 seconds on my system, which is a massive improvement.

However, in stracing the startup, I see that it is still using some wacky method to decide where to find libc.so.6, and that is bound to be slowing things down since it does this many many times during startup:

3264  access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000020>
3264  open("/opt/gnome/lib/tls/i686/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.00
0041>
3264  open("/opt/gnome/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.00001
7>
3264  open("/opt/gnome/lib/tls/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000016>
3264  open("/opt/gnome/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000016>
3264  open("/opt/gnome/lib/i686/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000023
>
3264  open("/opt/gnome/lib/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000016>
3264  open("/opt/gnome/lib/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000021>
3264  open("/opt/gnome/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000024>
3264  open("/lib/tls/i686/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000027>
3264  open("/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000014>
3264  open("/lib/tls/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000014>
3264  open("/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000014>
3264  open("/lib/i686/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000020>
3264  open("/lib/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000015>
3264  open("/lib/tm/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000021>
3264  open("/lib/libc.so.6", O_RDONLY)  = 3 <0.000020>

I don't have an LD_LIBRARY_PATH variable set in my environment, and those paths dont exist in /etc/ld.so.conf or in /etc/ld.so.conf.d/.  It seems to me that if the search algorithm was sane, it would START searching at /lib.  I don't understand where this search path is getting set.

I spoke too soon.  I ran /usr/X11R6/bin/acroread once and it started up
in 8 seconds.  On every subsequent attempt it takes over a minute to
start, so the fixes are NOT working on my system.  I am continuing to
analyze the trace file and finding LOTS of stuff that looks wrong to me,
but I don't really understand this stuff that well.

I hope someone who does can look at the tracelogs.  I've put new ones
out at wget http://home.comcast.net/~michaelnel/acroread_traces.tar.gz

Thanks
Michael

In analyzing the tracelog, I see thousands instances of opens trying to open things that are clearly NOT directories while using the O_DIRECTORY flag on the open:

open("/usr/X11R6/lib/X11/fonts/CID/encodings.dir", (O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = -1 ENOTDIR (Not a directory)
<0.000021>

I am over my head here, but I did read in the open(2) man page the
following:

O_DIRECTORY   If pathname is not a directory, cause the open to fail.  This
              flag is Linux-specific, and was added in kernel version
              2.1.126, to avoid denial-of-service problems if opendir(3) is
              called on a FIFO or tape device, but should not be used
              outside of the implementation of opendir.

...so, it looks to this non programmer that they shouldn't be using the
O_DIRECTORY flag on an open() call, but what do I know?  I'm just trying
to figure out why it takes this thing so long to open.

I have also tried the following things:

rpm -e the program, followed by using locate to make sure all remnants of it are gone, then reinstalling.

Tried installing Adobe's version (AdobeReader) from their website (same problem, if not perhaps worse).

Created a new test user with a clean environment, same problem.

I found acroread-7.0.1-4.i586.rpm and installed that.  It comes up in 4 seconds.  I ran traces on it and put them along with the 7.05 traces in a new archive:

http://home.comcast.net/~michaelnel/acro_traces_701_705.tar.gz

One major thing I note in the summary is that the 7.05 version makes about 32,000 open() calls while starting up, but the 7.01 version only does 3459 open() calls.

Mike, any comments on #44?

I don't think I can figure out what's wrong here, but if I can provide any more info or test anything else please let me know.

Nothing wrong in our side.  It's a bug in acroread, more exactly, a bug in the binary-only library included in acroread 7.05.  So, we cannot fix it.

At _every_ start, acroread tries to parse down the X11 font path _recusrively_ by unknown reason.  At the first time after boot (or resume), the kernel directory cache is cold.  That's why it takes long time.  At the second start, it's faster just because of hot kernel cache.

This happens no matter what fonts.conf you set and no matter what fonts cache exist.  No way to prevent it, AFAIK.

The "fix" in our package prevents a different problem.  Acroread tries to parse its own cache file containing all the system fonts, which eventually results in reading all font files on the system at every time.

Will someone at SuSE report this to Adobe?

I also don't understand why some 10.1 users are not having any problem with acroread 7.05, but others have these long delays in startup.

Another data point to ponder:  If I grab AdobeReader_enu-7.0.5-1.i386.rpm from the Adobe website and then rpm -e the SuSE-supplied 7.05 acrobat rpm and install the Adobe one in its place, I get the same slow startup.

But I can install that very same AdobeReader_enu-7.0.5-1.i386.rpm on my test machine which is a far less powerful hardware platform (Celeron 700MHz, 768MB RAM, single 9GB scsi drive) running CentOS 4, and it starts up in 3 seconds.

Sounds to me like there is something specific to SuSE that it doesn't like.

Andreas> Mike, any comments on #44?

Takashi already gave the information in his comment #49.
→ removing NEEDINFO.

I've given the current details to an Adobe engineer working on acroread. Hopefully they can help...

Thanks for doing that, George.  I realize this is proprietary software and that SuSE can't directly fix it.

Any further developments and / or progress on this bug?

this should be tracked at Adobe, acroread is closed source, so there is nothing what we can do.

The workaround used here causes problems with bug #275088.

I’m currently updating acroread to version 8.1.1 and if I continue
using that workaround, acroread will only find external fonts only
if the font directories are specified in PSRESOURCEPATH.

If I remove that workaround, acroread finds all fonts even if
PSRESOURCEPATH is not changed from the default.

Therefore, I’ll remove that workaround from our acroread 8.1.1
package.

The first startup is slow then because acroread creates the list
/home/mfabian/.adobe/Acrobat/8.0/Cache/UnixFnt08.lst but subsequent
starts are fast.

When trying to view the test document from bug #275088

    acroread Recovery_Manual.pdf

it will still look wrong when trying this immediately when first
starting acroread. Apparently acroread generates this font list
but uses it only during the next start. After starting acroread
again, the document looks correct.