Bug 156927

Created attachment 72163 [details]
top output showing beagle daemon's CPU usage

Created attachment 72500 [details]
top output showing beagle daemon's CPU usage after three days running

After the week-end the beagle daemons are still running full-steam.

Can you please attach a tarball of ~/.beagle/Log ?

Do you use evolution?  Possibly a dup of bug 156927.

(In reply to comment #4)
> Do you use evolution?  Possibly a dup of bug 156927.

To understand recursion, you have to understand recursion first ;-)

Yes, this bug is a duplicate of 156927 - because it is exactly _this_ bugreport ;-))

Maybe you wanted to add another bug number here?

I'm not using Evolution. I'm using KMail.

I'm a bit reluctant to attach .beagle/Log/ as it seems to contain a lot of private information. Is there anything specific I can extract from the logs or something I should check for?

Bug 156916 is what I meant, but if it's not Evolution, then it might not apply.

Would you mind sending me your logs in an email rather than attaching them to the bug?  The main things I am looking for are exceptions in the logs, whether the indexer is caught on one particular message, whether the backend is recrawling (but not reindexing) your mailboxes.

One thing to try is to run beagled with certain backends enabled and disabled.  If you run "beagled --allow-backend kmail" do you see the CPU being pegged?  What about if you run "beagled --deny-backend kmail"?  That'll help me track it down to a specific backend.

Looking at the top output, why do you have two beagled processes?  Are they broken into threads, or is one beagled and the other beagled-helper?

Another thing that would be helpful is when you notice it using this much CPU, send it SIGQUIT and attach the various stack traces to the bug.

I'm not that familiar with the KMail backend; that was written by a contributor rather than me, so I need a decent starting point.

There is a decent chance this is a dup of 156916.  Cornelius, can you try the --deny-backend thing I suggested in comment 7?  If disabling it fixes the issue, can you then try moving the ~/.beagle/Indexes/KMailIndex/offset-* files out of the way and restarting with that backend enabled?

Ok, I'm now running beagled with "--deny-backend kmail". Let's see, if the problem persists.

I submitted an updated gmime (2.2.1) for SLED 10 Beta 9.  Try that and see if it's fixed, assuming it's the KMail backend that's giving you troubles.

With "--deny-backend kmail" CPU usage seems to be better. It's a bit hard to tell, because after the weekend the daemon had crashed. The last lines of debug output were:

** ERROR **: file ../../mono/io-layer/handles-private.h: line 339 (_wapi_handle_share_release): assertion failed: (info->handle_refs > 0)
aborting...
Aborted
Debug: The daemon appears to have gone away.
Debug: Shutting down helper.
Debug: (1) Waiting for 1 worker...
Debug: waiting for server '/tmp/beagle-cschum-f7b463d7-6e9b-4093-a394-f6452a8c330c/socket-helper'
Debug: Server '/tmp/beagle-cschum-f7b463d7-6e9b-4093-a394-f6452a8c330c/socket-helper' shut down
Debug: Exiting


I will will try the new gmime version and see if it fixes the problem.

I have now upgrade gmime to 2.2.1-3. This also required to upgrade beagle to 0.2.3-1. Is this enough or are there any other packages which have to be updated?

The only message which is in the logs is "Warn: Extended attributes are not supported on this filesystem.  Performance will suffer as a result.", but nothing more, so I'm not sure that the daemon is actually doing something.

That's fine.  The debugging level was turned down in the 0.2.3 packages.  If you run beagled with --debug you will see the verbose messages again.  It's probably running, though.

What's the state of things?  Is the CPU usage improved?  Can we close this bug?

The CPU usage is improved. The daemon doesn't run constantly with full CPU usage anymore. But it still consumes a lot of CPU and it also seems to use lots of memory. I don't know if this is due to a bug or if it meant to be that way.

In addition to that it doesn't seem to give any useful search results for my mails. It only gives results for two mail folders regardless of the search terms. Is there a way to see, what parts of my mail folders beagle has indexed?

Created attachment 74680 [details]
top output showing beagle daemon's CPU usage after two days running

CPU usage after running for two days with the new gmime and beagle packages.

What should we do with this bug? The main bug seems to be fixed. Is the CPU usage as it's now as it is supposed to be, or is this still caused by bugs?

Should I open separate bugs for the lack of useful search results in mails and the missing option to find out the status of the index?

Can I do anything about the crash mentioned in comment #12? Is this a separate problem or has it been fixed by the new beagle version?

I believe the CPU usage is working as it should.  It will use the CPU for indexing, and it may take a few seconds at a time, but it should never peg it for an extended period.

As for the lack of useful results, do you know if this is caused by the fact that all of your data is not yet indexed?  There is a tool called beagle-index-info which will tell you the status of each index.  There isn't any indication in the UI, though.  That is filed upstream at bugzilla.gnome.org #160333:

http://bugzilla.gnome.org/show_bug.cgi?id=160333

If you think that your index is reasonably complete, but you're not getting useful results, I'd suggest opening a separate bug about that.  Certainly if your Beagle daemon has been running for a few days without incident I would expect that you'd be fully indexed.  We can debug whether or not the document is indexed at all, if it was indexed properly, etc.

As for the crash in comment #12, that's an internal Mono assertion and not directly a Beagle bug, so I'd suggest opening a separate bug about that against Mono.

I think this one is ok to close?

I'm closing this as FIXED.  Please reopen if you see the issue again.