Bug 148311

Summary:	Beagle indexing leaves stray processes
Product:	[openSUSE] SUSE LINUX 10.0	Reporter:	Cengiz Gunay <cgunay>
Component:	GNOME	Assignee:	Joe Shaw <joeshaw>
Status:	RESOLVED FIXED	QA Contact:	E-mail List <qa-bugs>
Severity:	Major
Priority:	P5 - None
Version:	Final
Target Milestone:	---
Hardware:	x86-64
OS:	SuSE Linux 10.0
Whiteboard:
Found By:	Other	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---
Attachments:	Email from cron after killing beagle-crawl-system output of "beagled --fg --debug --replace" The patch

Description Cengiz Gunay 2006-02-06 04:12:08 UTC

Beagle indexing process that is spawned from cron.daily apparently never quits. These processes, although idle, stay in memory day after day. I found 60 of them the other day. ps ax looks like this:
---
10314 ?        Ss     0:00 /bin/sh /usr/lib/beagle/beagle-crawl-system
10345 ?        SN     0:00 su beagleindex -c /usr/sbin/beagle-build-index --target /var/cache/beagle/indexes/applications --recursive --enable-text-cache /usr/share/applications /usr/local/shared/applications /opt/gnome/share/applications /opt/kde3/share/applications
10346 ?        SNl    0:00 mono --debug /usr/lib64/beagle/BuildIndex.exe --target /var/cache/beagle/indexes/applications --recursive --enable-text-cache /usr/share/applications /usr/local/shared/applications /opt/gnome/share/applications /opt/kde3/share/applications
---

These three are repeated for each day's cron jobs. I'm not sure why they are stuck. Here's a snippet from the daily debug mail that I get from Beagle indexer:
...
DEBUG: +file:///usr/share/doc/glibc/libc_46.html
DEBUG: +file:///usr/share/doc/glibc/libc_47.html
DEBUG: Size: VmRSS=96.2 MB, size=8.60, 189.9%
DEBUG: Process too big, shutting down!
DEBUG: Scanned 5436 files in 643 directories
DEBUG: CrawlWorker Done
DEBUG: IndexWorker Done
DEBUG: Restarting helper
WARN: Unable to set IO-priority for process to idle
DEBUG: Loaded 0 records from /var/cache/beagle/indexes/windows/FileAttributesStore.db in
0.003s
DEBUG: Starting CrawlWorker
DEBUG: Scanned 0 files in 0 directories
DEBUG: CrawlWorker Done
DEBUG: Starting IndexWorker
DEBUG: Size: VmRSS=11.4 MB, size=1.00, 0.0%
DEBUG: IndexWorker Done
--- END OF REPORT

It complains that "Process too big, shutting down!" at some point, but there are more messages after that, and it concludes with "IndexWorker Done", but it stays resident.

Comment 1 Joe Shaw 2006-02-06 15:06:44 UTC

Hmm, I can't duplicate the issue myself, unfortunately.

Can you open the /usr/lib/beagle/beagle-crawl-system script and add a --disable-restart option to the beagle-build-index call there (before --target)?

It'll be a little tougher on your system, but I think it'll give us a little bit better debugging output.

Comment 2 Cengiz Gunay 2006-02-08 17:15:55 UTC

Created attachment 67052 [details]
Email from cron after killing beagle-crawl-system

Before I send more information after I change the option in the crawl script, here's an exception I found in one of the cron emails, that escaped my attention last time.

I will send another message if I can get the debug outputs from the modified script.

Comment 3 Cengiz Gunay 2006-02-08 17:20:34 UTC

Comment on attachment 67052 [details]
Email from cron after killing beagle-crawl-system



WARN: Unable to set IO-priority for process to idle
DEBUG: Loaded 762 records from /var/cache/beagle/indexes/applications/FileAttributesStore
.db in 0.011s
DEBUG: Starting CrawlWorker
DEBUG: Starting IndexWorker
DEBUG: Size: VmRSS=11.3 MB, size=1.00, 0.0%
WARN: Exception caught while executing :Void IndexWorker()
WARN: System.NullReferenceException: Object reference not set to an instance of an object
in <0x00000> <unknown method>
in (wrapper managed-to-native) System.Runtime.InteropServices.Marshal:PtrToStringAnsi (in
tptr)
in <0x0016d> Mono.Data.SqliteClient.SqliteDataReader:ReadNextColumn ()
in (wrapper remoting-invoke-with-check) Mono.Data.SqliteClient.SqliteDataReader:ReadNextC
olumn ()
in <0x0001b> Mono.Data.SqliteClient.SqliteDataReader:Read ()
in (wrapper remoting-invoke-with-check) Mono.Data.SqliteClient.SqliteDataReader:Read ()
in <0x00027> Beagle.Daemon.FileAttributesStore_Sqlite:ReadOrWait (Mono.Data.SqliteClient.
SqliteDataReader reader)
in <0x0013e> Beagle.Daemon.FileAttributesStore_Sqlite:Read (System.String path)
in <0x0004a> Beagle.Daemon.FileAttributesStore:Read (System.String path)
in <0x0002b> Beagle.Daemon.FileAttributesStore:IsUpToDate (System.String path, Beagle.Dae
mon.Filter filter)
in <0x00026> Beagle.Daemon.FileAttributesStore:IsUpToDate (System.String path)
in <0x001a7> Beagle.Daemon.BuildIndex:IndexWorker ()
in (wrapper delegate-invoke) System.MulticastDelegate:invoke_void ()
in <0x0003e> Beagle.Util.ExceptionHandlingThread:ThreadStarted ()
DEBUG: Scanned 761 files in 7 directories
DEBUG: CrawlWorker Done
DEBUG: Size: VmRSS=12.1 MB, size=1.08, 1.9%
DEBUG: Size: VmRSS=12.1 MB, size=1.07, 1.9%
DEBUG: Size: VmRSS=12.0 MB, size=1.07, 1.7%
DEBUG: Size: VmRSS=12.0 MB, size=1.06, 1.6%
DEBUG: Size: VmRSS=12.0 MB, size=1.06, 1.6%
DEBUG: Size: VmRSS=12.0 MB, size=1.06, 1.6%
DEBUG: Size: VmRSS=12.0 MB, size=1.06, 1.6%
DEBUG: Size: VmRSS=11.9 MB, size=1.06, 1.4%
DEBUG: Size: VmRSS=11.9 MB, size=1.06, 1.5%
DEBUG: Size: VmRSS=11.9 MB, size=1.06, 1.4%
DEBUG: Size: VmRSS=11.9 MB, size=1.05, 1.3%
DEBUG: Size: VmRSS=11.8 MB, size=1.05, 1.3%
DEBUG: Size: VmRSS=11.8 MB, size=1.05, 1.2%
DEBUG: Size: VmRSS=11.8 MB, size=1.05, 1.2%
DEBUG: Size: VmRSS=11.3 MB, size=1.01, 0.2%
DEBUG: Size: VmRSS=10.7 MB, size=0.95, -1.3%
DEBUG: Size: VmRSS=10.8 MB, size=0.96, -1.1%
DEBUG: Size: VmRSS=10.8 MB, size=0.96, -1.1%
DEBUG: Size: VmRSS=10.8 MB, size=0.95, -1.2%
DEBUG: Size: VmRSS=10.7 MB, size=0.95, -1.2%
DEBUG: Size: VmRSS=10.7 MB, size=0.95, -1.2%
DEBUG: Size: VmRSS=10.7 MB, size=0.95, -1.2%
DEBUG: Size: VmRSS=10.7 MB, size=0.95, -1.3%
DEBUG: Size: VmRSS=10.7 MB, size=0.95, -1.3%
DEBUG: Size: VmRSS=10.7 MB, size=0.95, -1.3%
DEBUG: Shutdown Requested
WARN: Unable to set IO-priority for process to idle
DEBUG: Loaded 4202 records from /var/cache/beagle/indexes/documentation/FileAttributesSto
re.db in 0.053s
DEBUG: Starting CrawlWorker
DEBUG: Starting IndexWorker
DEBUG: Size: VmRSS=11.4 MB, size=1.00, 0.0%
DEBUG: Size: VmRSS=11.5 MB, size=1.01, 0.2%
DEBUG: Size: VmRSS=11.7 MB, size=1.03, 0.7%
DEBUG: Size: VmRSS=11.7 MB, size=1.03, 0.7%
DEBUG: Size: VmRSS=11.7 MB, size=1.03, 0.7%
DEBUG: Size: VmRSS=11.9 MB, size=1.05, 1.1%
DEBUG: Size: VmRSS=13.8 MB, size=1.21, 5.3%
DEBUG: Size: VmRSS=17.0 MB, size=1.50, 12.5%
WARN: Exception caught while executing :Void IndexWorker()
WARN: System.NullReferenceException: Object reference not set to an instance of an object
in <0x00000> <unknown method>
in (wrapper managed-to-native) System.Runtime.InteropServices.Marshal:PtrToStringAnsi (in
tptr)
in <0x0025a> Mono.Data.SqliteClient.SqliteDataReader:ReadNextColumn ()
in (wrapper remoting-invoke-with-check) Mono.Data.SqliteClient.SqliteDataReader:ReadNextC
olumn ()
in <0x0001b> Mono.Data.SqliteClient.SqliteDataReader:Read ()
in (wrapper remoting-invoke-with-check) Mono.Data.SqliteClient.SqliteDataReader:Read ()
in <0x00027> Beagle.Daemon.FileAttributesStore_Sqlite:ReadOrWait (Mono.Data.SqliteClient.
SqliteDataReader reader)
in <0x0013e> Beagle.Daemon.FileAttributesStore_Sqlite:Read (System.String path)
in <0x0004a> Beagle.Daemon.FileAttributesStore:Read (System.String path)
in <0x0002b> Beagle.Daemon.FileAttributesStore:IsUpToDate (System.String path, Beagle.Dae
mon.Filter filter)
in <0x00026> Beagle.Daemon.FileAttributesStore:IsUpToDate (System.String path)
in <0x001a7> Beagle.Daemon.BuildIndex:IndexWorker ()
in (wrapper delegate-invoke) System.MulticastDelegate:invoke_void ()
in <0x0003e> Beagle.Util.ExceptionHandlingThread:ThreadStarted ()
DEBUG: Size: VmRSS=17.5 MB, size=1.54, 13.4%
DEBUG: Size: VmRSS=18.9 MB, size=1.66, 16.6%
DEBUG: Size: VmRSS=21.1 MB, size=1.86, 21.5%
DEBUG: Size: VmRSS=24.0 MB, size=2.11, 27.9%
DEBUG: Size: VmRSS=24.1 MB, size=2.12, 28.0%
DEBUG: Size: VmRSS=24.3 MB, size=2.14, 28.6%
DEBUG: Size: VmRSS=27.8 MB, size=2.44, 36.1%

Comment 4 Joe Shaw 2006-02-08 17:53:53 UTC

Are you running on a 64-bit machine?

Comment 5 Joe Shaw 2006-02-08 17:58:06 UTC

err, I missed the hardware part and see that you are.  Never mind. :)

Probably a dup of http://mail.gnome.org/archives/dashboard-hackers/2006-January/msg00096.html

I think it might be 64-bit uncleanliness in the Mono.Data.SqliteClient we have imported into Beagle.  CVS Beagle contains an updated version, but it also appears to have other (non-64-bit-related) issues.

Comment 6 Joe Shaw 2006-02-08 18:20:29 UTC

I've just checked in a fix to Beagle CVS so that the beagle-build-index tool will no longer hang if it encounters an error.  It doesn't, however, fix the particular error you're seeing.

Comment 7 Cengiz Gunay 2006-02-08 18:22:10 UTC

Created attachment 67072 [details]
output of "beagled --fg --debug --replace"

Here's the output from "beagled --fg --debug --replace". So probably it's not a duplicate of that bug. You can see in the attachment that I'm not getting a segfault, but a System.NullReferenceException.

Comment 8 Joe Shaw 2006-02-08 18:28:00 UTC

A segfault and a NullRefException are essentially the same thing (especially when using something which copies between managed and unmanaged code like Marshal.PtrToStringAnsi()).

Comment 9 Joe Shaw 2006-02-09 00:04:48 UTC

I just fixed the problem in CVS.  I will whip up a patch for the package in 10.1 tomorrow; I'll try to build it against 10.0, but I'm not sure it'll build fully.

Comment 10 Joe Shaw 2006-02-09 18:43:23 UTC

Submitted this to STABLE and PLUS, so this will be fixed for 10.1.  I can build 10.0 packages, but they'll be of the latest version in our build system -- 0.2.1 -- but chances are you won't have the dependencies for that.  I'm not aware of a way to build against 10.0 as it was released; I am definitely not an autobuild expert.

Comment 11 Joe Shaw 2006-02-09 18:55:38 UTC

Created attachment 67406 [details]
The patch

Oh, by the way, here's the patch if you want to try it on a local build.  It's against the 0.2.1 release, but will probably apply to older versions.

Comment 12 Cengiz Gunay 2006-02-27 16:19:37 UTC

Adding --disable-restart to /usr/lib/beagle/beagle-crawl-system script actually stopped beagle from leaving stray processes. So my problem is solved AFAIK. Thanks..