Bug 476081

Summary: Too many mod_mono instances running and mod-mono server stops responding
Product: [Mono] Mono: Tools Reporter: Anil Kumar <kuanil>
Component: mod_monoAssignee: Marek Habersack <mhabersack>
Status: RESOLVED DUPLICATE QA Contact: Marek Habersack <mhabersack>
Severity: Critical    
Priority: P5 - None CC: forgotten_xR7TQ7T0Yf, juraj, veerapuram.varadhan
Version: 2.2.x   
Target Milestone: ---   
Hardware: Other   
OS: SLES 10   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Anil Kumar 2009-02-16 09:05:40 UTC
After discussion with Mono team for bug id 440486, iFolder code was modified so that it is supported to run with Mono2.x runtime. Currently we are running/testing iFolder with Mono2.2 runtime tree (This tree is a directory structure with all mono runtime libraries inside iFolder) as it was decided alongwith Mono team. When iFolder server is accessed through mod-mono with very requests coming in one by one, everything works fine. But once many clients connect and server also starts using mod-mono frequently, suddenly it is seen that there are many instances of mod-mono running and server stops responding. Once this happens, then every minute many instances of mod-mono comes up and get added. Server becomes non-responsive. This can be reproduced within15-30 minutes. Our ST person, when tries to connect around 15 clients, this happens in first time itself. 
I read http://www.mono-project.com/Mod_mono#Advanced_options 
and tried putting "MonoAutoRestartTime simias10 00:00:120 " into our conf file for a restart at every 2 minutes, but this is not working at all. No restart of mod-mono happens. 
I also used : 
<Location /mono>
  SetHandler mono-ctrl
  Order deny,allow
  Deny from all
  Allow from 127.0.0.1
</Location>
to check the mod-mono server, but no of requests currently being processed does not cross 3 (or may be when i am checking this no has reduced ...not sure).

i am writing here a part of the result of "pe -ef | grep mono"  : 


wwwrun   32538     1  0 13:11 ?        00:00:00 /opt/novell/ifolder3/bin/../lib/Mono/usr/bin/mono /opt/novell/ifolder3/bin/../lib/Mono/usr/lib/mono/2.0/mod-mono-server2.exe --filename /tmp/mod_mono_server_simias10 --applications /simias10:/opt/novell/ifolder3/lib/simias/web --nonstop
wwwrun   32554     1  0 13:11 ?        00:00:00 /opt/novell/ifolder3/bin/../lib/Mono/usr/bin/mono /opt/novell/ifolder3/bin/../lib/Mono/usr/lib/mono/2.0/mod-mono-server2.exe --filename /tmp/mod_mono_server_simias10 --applications /simias10:/opt/novell/ifolder3/lib/simias/web --nonstop
wwwrun   32563     1  0 13:12 ?        00:00:00 /opt/novell/ifolder3/bin/../lib/Mono/usr/bin/mono /opt/novell/ifolder3/bin/../lib/Mono/usr/lib/mono/2.0/mod-mono-server2.exe --filename /tmp/mod_mono_server_simias10 --applications /simias10:/opt/novell/ifolder3/lib/simias/web --nonstop

and so-on...........


For you reference, i am writing the conf file used by iFolder and which reproduces this issue : 
Include /opt/novell/ifolder3/lib/Mono/bin/mod_mono.conf

Alias /simias10 "/opt/novell/ifolder3/lib/simias/web"
AddMonoApplications simias10 "/simias10:/opt/novell/ifolder3/lib/simias/web"
MonoSetEnv simias10 "SimiasRunAsServer=true;SimiasDataDir=/var/simias/data/simias"
MonoServerPath simias10 /opt/novell/ifolder3/lib/Mono/bin/mod-mono-server2
<Location /simias10 >
        MonoSetServerAlias simias10
        Order allow,deny
        Allow from all
        SetHandler mono
</Location>

even if i do apache2 restart/reload or pkill mono, it does not kill mono.. after sending SIGKILL to mono process, it kills mono. But these manual workarounds won't be acceptable by customers.Also iFolder will have big deployments where restarting server every now and then won't be acceptable scenario. So we would like to have a fix.
Comment 1 Anil Kumar 2009-02-16 09:11:33 UTC
[Mon Feb 16 12:21:12 2009] [notice] Apache/2.2.3 (Linux/SUSE) configured -- resuming normal operations
This is the message inside /var/log/apache2/error_log


** ERROR **: file ../../mono/io-layer/handles-private.h: line 160 (_wapi_handle_set_signal_state): assertion failed: (thr_ret == 0)
aborting...
Stacktrace:

[Mon Feb 16 13:11:28 2009] [error] (70014)End of file found: read_data failed
[Mon Feb 16 13:11:28 2009] [error] Command stream corrupted, last command was 7
[Mon Feb 16 13:11:37 2009] [error] Failed to connect to mod-mono-server after several attempts to spawn the process.
[Mon Feb 16 13:11:43 2009] [error] Failed to connect to mod-mono-server after several attempts to spawn the process.
[Mon Feb 16 13:11:53 2009] [error] Failed to connect to mod-mono-server after several attempts to spawn the process.
[Mon Feb 16 13:11:57 2009] [error] Failed to connect to mod-mono-server after several attempts to spawn the process.
.................................................
..............................................
..................................................
** (/opt/novell/ifolder3/bin/../lib/Mono/usr/lib/mono/2.0/mod-mono-server2.exe:7542): WARNING **: process_set_current: error creating process handle
[Mon Feb 16 13:21:58 2009] [error] Failed to connect to mod-mono-server after several attempts to spawn the process.
Comment 2 Marek Habersack 2009-02-16 09:56:07 UTC
This bug is a duplicate of Bug 472732

*** This bug has been marked as a duplicate of bug 472732 ***