Bugzilla – Bug 157474
parse-metadata uses enormous amount of memory
Last modified: 2006-11-09 08:18:45 UTC
Adding the factory tree with rug sa --type=yum http://dist.suse.de/factory/suse factory the parse-metadata script is called by zmd and uses 780 MB of memory: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 23530 69.2 72.6 782324 746512 ? D 12:17 1:22 /usr/lib/zmd/parse-metadata /var/li During this time my system becomes unusable. We need to reduce the memory usage, otherwise this will not work on systems with 256 MB of main memory since they'll swap to death (and PM likes to support those!) This could be a memory leak.
Created attachment 72440 [details] memgrind output I run valgrind's memory checker on a smaller data set as: valgrind --leak-check=full --log-file-exactly=/tmp/valgrind /usr/lib/zmd/parse-metadata /var/lib/zmd/zmd.db yum /var/cache/zmd/web/files/dist.suse.de/factory-e/suse/ http://dist.suse.de/factory-e/suse
Klaus, valgrind output has lots of references to zypp::parser, so it might be the problem is in zypp.
Re-assigning to Klaus based on #2, and adding Tambet, James and Nat on the cc to also look into this further.
*** Bug 157901 has been marked as a duplicate of this bug. ***
FYI, On my system it's taking over 1.1GB of memory, and it runs on each reboot too. (very glad I've got 2GB of RAM on this box.
*** Bug 155354 has been marked as a duplicate of this bug. ***
*** Bug 151515 has been marked as a duplicate of this bug. ***
*** Bug 156452 has been marked as a duplicate of this bug. ***
Jiri, any status on this?
On PPC with 256M of RAM and installing factory (beta8), my machine still runs out of RAM when it gets to the installation overview page in YaST. When I turn on swap, there is about 80MB of swap used at this stage. At the end of installation it's up to 150MB, but that's still a lot better than 1.9G swap, which I reported earlier...
On my machine it is also hogging processor (constantly 90%+ at log on), makes it very slow or totally unresponsive. Several times I've reset the X server after 4-5 minutes of sitting waiting to log in, only to leave it for 10 minutes or so and it eventually logs on. Other times the system logs on normally before parse-metadata starts to hog the system (issuing "top" can take up to 10 seconds before it starts working for example).
- Fixed memory leak in XMLNodeIterator XMLNodeIterator never released memory allocated for parsed data. - disabled storing filelist (YUMFileListParser) and changelog (YUMOtherParser) Even without the leak filelist and changelog(!) data are too big to store them plain. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 19044 pts/1 S+ 0:34 6 3301 164334 123072 24.1 ./Parse DURATION: 35 sec. parseSource: dir:///Local/FACTORY ----------------------------------------------------------
I think the XMLNodeIterator leak will not help here. Factory is not a YUM repository.
factory is *also* a YUM repository, see the first command...
It was mentioned in the thread of a duplicate bug that a potential problem may be that the xml is being read in as a DOM object. What's the status or update on this?
Does that mean zypp backend does not parse 'filelists' and 'other' files? If so, I'd take them out from zmd as well, no need to download huge files if noone will look at them.
On my system (512 MB RAM) zmd/parse-metadata also takes all available memory until the system freezes. Tested with 10.1Beta8. from top: --------- top - 23:09:46 up 2 min, 5 users, load average: 3.38, 1.65, 0.62 Tasks: 89 total, 3 running, 86 sleeping, 0 stopped, 0 zombie Cpu(s): 33.1% us, 8.3% sy, 0.0% ni, 4.8% id, 53.4% wa, 0.5% hi, 0.0% si Mem: 516112k total, 475280k used, 40832k free, 22588k buffers Swap: 265032k total, 0k used, 265032k free, 218144k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4822 root 25 0 176m 141m 5248 R 99.4 28.1 0:12.42 parse-metadata from pstree: ------------ `-zmd /usr/lib/zmd/zmd.exe |-parse-metadata /var/lib/zmd/zmd.db yum/var/cache/zmd/web/files/ftp4.gwdg .de/pub/opensuse/distribution/SL-OSS-factory/i
For the first time ever, my PPC machine with 256MB of ram did not freeze at the installation overview page when installed factory. It just became extremely slow with a high load. After waiting for about an hour I did a swapon /dev/hda2, and the overview screen was ready in a few seconds, after swapping out 80MB. Earlier it took just about 2-3 minutes for a complete freeze, when I forgot swapon, so there is some progress...
We now skip incompatible architectures during XML parsing and only read 'interesting' files (containing sbin, bin, lib, lib64) from the filelist. On a x86_64 host this brings the number of files down to ~12000 packages (down from ~23000) and approx 180MB virt memory.
Tested factory as of April 3rd, 2006. On PPC with 256MB of RAM this is the first time ever when installing factory, that I can get to the "Installation settings" screen without turning on swap first. I must mention, that it's still almost unusable slow. When swap is turned on, it's still used heavily, about 85MBs during configuration, but at least it's not slow.
I have a customer that is seeing this on SLED 10, the performance/slowness side of the problem has been bad enough to prompt him to contact Novell. While I have been able to duplicate the 100% utilization while the zen-updater is refreshing, I have not been able to duplicate the slowness. It seems, however that at least 1 other person commenting on this defect has seen a performance hit. Is there an update for this defect?
Jack, the original reported issue was fixed for SLED10. IF you still have a issue, please open a new bugreport.