Memory Leak!

[Chalnoth]

This is most definitely a problem.

First of all, the problem that I noticed was that UT occasionally stutters after playing for a little bit.

But, the problem doesn't appear to be related to UT.

For example, right now, at the beginning of writing this post, I'm sitting at about 245MB of memory in use. The only programs I'm currently running are XMMS, KPPP, and Mozilla. There's no way that these three (in combination with whatever Linux is running in the background...drivers and whatnot) should take up that much memory.

As an example, after booting up completely, I was sitting at around 90MB of memory in use. I also looked at top for a few moments, and I could actually see the memory usage climb, by about 8kb/sec.

Finally, at the end of writing this message, I'm sitting at about 251MB of memory in use.

Has anybody seen a problem like this, or have an idea of how to fix it? I suppose I may have quite a lot of debugging to do...but I was hoping somebody else out there might have some hints...

[Chalnoth]

Oh, yeah, and I guess it could help to give my system specs as well.

I'm running on an nForce 415-D chipset motherboard, with the latest drivers installed (246, I believe).

For the sound, I'm no longer using the nForce drivers, but I'm instead using the alsa i8x0 drivers. nVidia's nForce drivers don't work with everything, though the Alsa drivers have problems with sound quality sometimes...damn I hope that all gets worked out soon.

I'm also running a GeForce4 Ti 4200 with the 3123 drivers.

I downloaded and am running Fluxbox as the display manager. I also had to turn off apmd at startup so the machine wouldn't crash.

Everything else is from the default installation of RedHat 7.3.

[Chalnoth]

I guess I'll just talk to myself some more...

Well, it looks like there might not be a memory leak after all. After looking around a bit on this forum, I noticed that it could just be the cache. In fact, just after posting, the amount of used memory started to approach my installed RAM amount (256MB). It didn't exceed it. Now I'm sitting at about 105MB of used memory minus cache and buffer.

[The Baron]

Guy I know had a similar problem... and this is what it was:

http://www.redhat.com/mailing-lists/redhat-devel-list/msg04789.html

http://www.redhat.com/mailing-lists/redhat-devel-list/msg04816.html

Not using swap. :p

Now, with you being you, I should PROBABLY assume that swap is fine... but oh well.

[Chalnoth]

Yeah, swap appears to be working just fine.

But, it's really looking like the problem is simply that Linux is using up too much system RAM with cache, resulting in unecessary swap usage.

That is, I think what's happening is it's getting within about 5MB of my total RAM (which is probably far too low, considering the RAM is almost certainly highly fragmented...), and when UT's RAM requirements change a bit mid-game, I get massive stalls.

After reexamining the memory usage, though, I should have plenty. It would be nice if I could find a variable that I could modify to reduce the amount of RAM that the cache will attempt to use, and see if that fixes the stuttering problems.

[de><ta]

what distribution of linux are you using and what version?

I also doubt that flux and the rest of the applications you have running in the background chew up that much RAM.

What exactly did you taylor or change???

9o mb at the start is fairly high, I get about that much using KDE with all the widgets and background application running.

How much of swap have you allocated by the way?

Have you tried looking up what the memory is actually being allocated to?

[Chalnoth]

Please read my second post.

[kappax]

LOL, THIS IS NOT A MEM LEAK! this is good that you ram is used up!. THIS IS NOT WINDOWS.

Linux uses a cache,

type top and you will see it!!

Mem: 515164K av, 347116K used, 168048K free, 0K shrd, 9184K buff
Swap: 1028152K av, 0K used, 1028152K free 230180K cached


look at all that cached, and look at my used and my av.

this is normal to have verry little "FREE" ram.

if someting needs the ram it will get it, but till then it sits in the cache, and so apps that have benopen befor will open again but faster , insted of pullin from the HD it will pull from the cache (ram)


this is noraml not a problem.

[kappax]

Originally posted by Chalnoth
Yeah, swap appears to be working just fine.

But, it's really looking like the problem is simply that Linux is using up too much system RAM with cache, resulting in unecessary swap usage.

That is, I think what's happening is it's getting within about 5MB of my total RAM (which is probably far too low, considering the RAM is almost certainly highly fragmented...), and when UT's RAM requirements change a bit mid-game, I get massive stalls.

After reexamining the memory usage, though, I should have plenty. It would be nice if I could find a variable that I could modify to reduce the amount of RAM that the cache will attempt to use, and see if that fixes the stuttering problems.

the cache does not make you swap. LINUX is smart. if it needs ram it will wipe something from the cache. if you have less than 12 megs in swap then there is no problem ( or if you do you just have verry little ram), Linux can be run on 4 megs of ram.

Linux has great memery managment.

If you dont want your computer using swap, then turn swap off! via editing the fstab. I know many ppl that have 512+ megs ram that dont use swap, i use swap because i am often working with files 1 gig +

[Chalnoth]

Originally posted by kappax
LOL, THIS IS NOT A MEM LEAK! this is good that you ram is used up!. THIS IS NOT WINDOWS.

I take it you didn't read my fourth post down. And yes, swap is being used. I would never have thought this was a problem if I hadn't seen symptoms that were reminiscent of a memory leak.

Typically what I notice is that after such a stall in UT (generally while the system still is using ~100MB of cache), the amount of swap used increases. This means, to me, that the cache isn't being cleared out fast enough to eliminate the need for the swap. In other words, I'd really like to see an option to force Linux to leave some more free RAM, and see if that fixes the problem. I'm sure it's possible to make Linux do this, but I was hoping that there was an option for it.

Regardless, I'm going to be moving back to Windows for at least a little bit. Sound doesn't seem to be working with my nForce2 (although it is no longer crashing upon loading Gnome/KDE). I'll return to Linux when some new sound drivers are released.

[Chalnoth]

Update:

One other thing, I'm not actually completely convinced there isn't a memory leak going on. Even taking the cache/buffer into account, memory usage is way above what it seems it should be.

For example, you're showing about 110MB of memory in use. Running rather few programs, I was showing about 140MB (cache taken into account). Anyway, I guess I'll have to do a more in-depth analysis to be certain. I suppose I'll do that later.

[lunix]

Chanloth it is not a memory leak. What is happening is that X is mmaping your AGP aparature and adding your graphics card memory as allocated memory. When top asks how much memory each application is using X reports way more memory than it is actually using skewing the results.

[Chalnoth]

Originally posted by lunix
Chanloth it is not a memory leak. What is happening is that X is mmaping your AGP aparature and adding your graphics card memory as allocated memory. When top asks how much memory each application is using X reports way more memory than it is actually using skewing the results.

I'll look into this, but I wasn't entirely referring to the absolute memory usage being used to detect whether or not there was a memory leak. Anyway, I think I may have thought of a way to get my nForce2 sound to work, but it's going to take some time...so I'm not going to work on it just yet...once I do, though, I'll deal with this issue.

[lunix]

Run top and press "M" to sort by memory to help diagnose the leaking app. Beyond that its arcane arts reserved for developers. Anyhow looking into it you will find that the mmaped buffers in X with 3D acceleration seriously screw your memory readings.
To bad about the pauses in ut must be anoying. Perhaps you could try changing agp drivers from nvagp to agpgart but Im just guessing here. It would be strange for linux to swap out an app that is in use, I know the vm sucks but not that bad. Good luck and have fun Chalnoth.

[Chalnoth]

I don't think it would need to swap out an app in use to slow down the program. Swapping anything could slow down the system. Anyway, I'm now trying to get the R9700 to work in Linux. It's interesting...agpgart won't recognize my nForce2, so x won't start using ATI's drivers.

Update:
Hrm, that brings up a possibly different problem. The stuttering would definitely be more pronounced if DMA was disabled. I checked /proc/ide/hda/settings
Under "Using dma" there was a value of 0.

Do you know how to enable DMA?

[Phyre]

Chalnoth, check out the hdparm command. It's in /sbin, so you might have to invoke it with /sbin/hdparm. As root (su), run this command to see what kind of rates you're getting:

/sbin/hdparm -Tt /dev/hdx (where x is your linux drive).

To enable DMA, su and type /sbin/hdparm -d1 /dev/hdx. There are other things you can to do with hdparm to boost the performance of your drive too. This might help ya out in that respect.

http://www.linuxnetmag.com/en/issue7/m7hdparm1.html

Once you have the settings that you need for your drive, you can put them in your rc.d file so that they will load every time. Can't remember exactly which rc.d file. I have to do a search on it everytime I reinstall. LOL

Also, what motherboard/chipset are you using. I'm not sure if it's been fixed in RedHat 8, but VIA used to have problems with not beeing able to get to ATA100 modes. The source for it's module had to be modified a bit, hence forcing a rebuild of the kernel.

Hope this gets you a little further into this caper.

Phyre

[lunix]

You can make sure your drive are configured by adding the autotune option for you kernel. e.g. my /boot/grub/menu.lst has the following options for my kernel:

title=Gentoo Linux
root (hd0,1)
kernel /boot/bzImage vga=791 root=/dev/hda2 ide0=autotune ide1=autotune ide3=autotune acpi=no-idle hdc=ide-scsi

[Chalnoth]

Thanks for the replies. Investigating now...

I'm using an nForce2 motherboard, and here are the hdparm -Tt /dev/hda results:

buffer-cache: 304MB/sec
buffered-disk: 3.29MB/sec

(Disk is IBM 75GXP...reads should max out at somewhere close to 30MB/sec sustained...)

Update:
Oh, yeah, the hdparm -d1 /dev/hda definitely worked.

buffer-cache: 298MB/sec
buffered-disk: 31.22MB/sec

Update #2:
I went ahead and inserted the commands (for HD and DVD/CD-RW drives into the rc.local file, and now it all works great. Thanks again :)

[Chalnoth]

Oh, and one last little thing.

With UT, the major suttering I got previously has dropped back to very minor pauses. They're now bearable, but shouldn't be there at all. Anybody know where I can configure the cache in Linux?

[kappax]

Originally posted by Chalnoth
Update:

One other thing, I'm not actually completely convinced there isn't a memory leak going on. Even taking the cache/buffer into account, memory usage is way above what it seems it should be.

For example, you're showing about 110MB of memory in use. Running rather few programs, I was showing about 140MB (cache taken into account). Anyway, I guess I'll have to do a more in-depth analysis to be certain. I suppose I'll do that later.

I would like to know whaty you mean by this

" I was showing about 140MB (cache taken into account)."

plez dont start adding numbers.

used is used
460308k used
free is free
54812k free
cached is cached
181568k cached

each one means someting.


post us what you see when you type "top" at a console.

[kappax]

Originally posted by Chalnoth
Oh, and one last little thing.

With UT, the major suttering I got previously has dropped back to very minor pauses. They're now bearable, but shouldn't be there at all. Anybody know where I can configure the cache in Linux?

the cache does not cause you to swap!

And second, I still dont even know how much ram you have !!


effectiv free meme
free + cached = free

when someitng needs ram it will clear out the cache (part of ot not all of it ) and make roome for the new program needing to be run.

Second Ut2003 is a real real real hard game on computers.
I dont know your computer specks but pausing now and this can be related to

misconfigured vid card.
bad IDE setings (seems you fixed thoes)
misconfigured or bad sound card drivers (try turning it to HW accel insted of default sound settings )

what sort of FPS are you getting ?


what system do you have ? vid card amount of ram ? ddr ? sound card ?


we know nothing about your system.



oh to configure the cache you would have to rewrite part of your kernel. ( as far as i know)

you rellay should not need to mess with the cache.


Sounds like you are running a older distro, or a distro that has not optmized there kernel

you need to look in to getting a kernel with
eather
preimpt+low late
or one with preimpt
or one with low late

if you use other patches on your kernel it will be hard to incoperate both preimpt and lowlate, but low late i think is the best otheres think preimpt is.

I have had a kernel with lowlate and preimpt but my drivers for my ide card have made it to where i can only use low late.

[Chalnoth]

Originally posted by kappax
the cache does not cause you to swap!

Um. I'm getting swapping with cache still in use (~100MB or so of cache in use). I have 256MB of RAM.

when someitng needs ram it will clear out the cache (part of ot not all of it ) and make roome for the new program needing to be run.

Right. What I'm apparently seeing is that the cache isn't being cleared out quickly enough, and a little bit is being written to swap to clear out the necessary memory (Generally 200k-500k at a time).

While this could be an optimization issue (whatever is being written to swap is most probably not something that's been used recently), I'd rather that the cache is cleared instead of writing things to swap.

Second Ut2003 is a real real real hard game on computers.
I dont know your computer specks but pausing now and this can be related to

misconfigured vid card.
bad IDE setings (seems you fixed thoes)
misconfigured or bad sound card drivers (try turning it to HW accel insted of default sound settings )

Athlon XP 2000+
nForce2
GeForce4 Ti 4200

Anyway, even if the IDE settings were bad, they should not have ever affected in-game performance. There shouldn't be anything writing to the HD while I'm playing.

I haven't checked the FPS, but I'll have to get back to you on that (in windows atm).

It is conceivable that it's the sound drivers. I'll have to look into that some more.

Sounds like you are running a older distro, or a distro that has not optmized there kernel

Redhat 7.3

[kappax]

Originally posted by Chalnoth
Um. I'm getting swapping with cache still in use (~100MB or so of cache in use). I have 256MB of RAM.



Right. What I'm apparently seeing is that the cache isn't being cleared out quickly enough, and a little bit is being written to swap to clear out the necessary memory (Generally 200k-500k at a time).

While this could be an optimization issue (whatever is being written to swap is most probably not something that's been used recently), I'd rather that the cache is cleared instead of writing things to swap.



Athlon XP 2000+
nForce2
GeForce4 Ti 4200

Anyway, even if the IDE settings were bad, they should not have ever affected in-game performance. There shouldn't be anything writing to the HD while I'm playing.

I haven't checked the FPS, but I'll have to get back to you on that (in windows atm).

It is conceivable that it's the sound drivers. I'll have to look into that some more.



Redhat 7.3

ok yes it could be IDE, UT2003 is a large game and has many textures, and can need to get data from the HD at any time.

2

you can keep the 7.3, but upgrade to the 2.4.19 kernel with preimpt and lolate

as far as i know, there is only 2 cases someting goes to swap.

1. you are out of ram and need to use vertual ram.

2. someting sits in the ram for a long time without being accessed, then is put on swap because it is not needed.

[Chalnoth]

Yes, I just recently upgraded the kernel, and that seems to have helped. I'm back to using the Radeon 9700 for now, though, so it may be a little while before I get back to looking at this stuff (R9700 doesn't want to work in Linux due to lack of agpgart for nForce2).

[psych0]

I seem to be having a very similar ssortt of a problem... Except I can't say that Liinux is doing a very good job caching, sinnce my X is hellishly slow!!!!

I'm running

a p4 - 1,8 G Toshiba Sattelite laptop...
with 512 Ram
and a GeForce4

it is mad annoying that even under Windows the GUI part of the system runs as fast as a rabbit, but once I switch to Linux it is actually slower than my old Acer lappy with 128ram and a bad bad bad videocard by neomagic.
It is slow to thee point where I caan actually see it redrawing the screen 10 times when I switch virtual workplaces... or ask it to shade a window (windowmaker style doubleclick on a titlebar)!!!

this is all the same whetthher I run the new drivers which come with the X distro or the NVIDIA provided drivers...

here is the info I have showing up in top:

Mem: 515620k total, 350308k used, 165312k free, 15660k buffers
Swap: 385452k total, 0k used, 385452k free, 194624k cached

22851 root 9 0 176m 47m 3480 S 7.3 9.5 1:34.95 X

here is the more alarming info I get from the Gnome thing
X -> using 176 MB memory and from
Total: 176 MB
Rss: 47,9 MB
Shared: 3.4 MB


this all seems to be verry alarming... anyhow, whenever I start a new app onnce X is running it seems that it has to go through swap slowing everrything down...
I do not have clue about the agp factor of it all...
is there any way to fix it???
or is this a commmon problem? if u can call it such?