My original problem is described in this thread: http://www.nvnews.net/vbulletin/showthread.php?t=74171
So I've been goofing around with settings for 2 solid days to get 3d apps to work, and I think I've finally found the problem. It seems that the nVidia driver overclocks the FX5200. I enabled CoolBits and this is what nvclock reports when the card starts:
ppc@akula:~$ nvclock -s
Card: nVidia GeforceFX 5200
Card number: 1
Memory clock: 265.500 MHz
GPU clock: 249.750 MHz
(nvidia-settings doesn't even see the CoolBits turned on, oh well)
So, I lowered both the memory clock and GPU clock to 220MHz and, lo and behold, 3D works good and solid!
Now, I've played around with several frequencies, and the problem reoccurs more and more frequently as the frequency goes up. At 230MHz, Tuxracer halts after a little while, but not always, so I may lower it even further as I test some more, don't take 220MHz as a magic number. Also, setting the GPU clock higher than the memory clock makes the monitor scream for mercy and crashes the computer, so I'd advise to always set them both together at the same value if you don't want to brick your monitor, at least with the same card as mine.
After rebooting several times, I noticed that the default clock speeds (at least the memory clock speed) seems to vary somewhat at each restart. My guess is nVidia added some not-so-clever speed test that determines how hard the card can be driven. What would be good is if someone at nVidia is listening and tells us what the safe, bottom values are for these frequencies for each card (fat chance...). Also, as a side note, I don't know if this applies to video cards, but having severely underclocked (and smoked) PCs in the past, I'd advise against lowering the frequencies too much. The lowest I've tried for the FX5200 is 200MHz so far and it seems happy enough. But officially-sanctioned safe values from nVidia would be nice.
To make this report complete, I'm not certain this single solution is what got the 3D finally going (although I have a strong feeling it is), so I'll also detail what else I did since I ranted yesterday about this problem:
- Switched to linux-126.96.36.199: no result
- Enabled high-memory (since I have 1G of ram and an AGP aperture of 128M, I figured the total is over 1G so Linux might have some page table issues, who knows...): no result
- Enabled/installed the i2c temp sensors: no result, the video card doesn't overheat
- Tried to drop a 2 year-old nvidia binary core into the 8762 driver distribution and hacking files around to get it to compile and insmod without dependency problem: crashed the box
- Backed to version 7182-legacy of the nVidia driver: no result
- Backed to version 8756 of the nVidia driver: some results: oolite seemed to lock up less fast, a minute or two instead of seconds. In retrospect, I guess this driver uses more relaxed frequency settings.
- Underclocked the driver manually with nvclock: problem seemingly solved (I haven't tested a lot yet, but I'll sure play games all night tonight to check it out :-) )
If you have the frozen-3D problem, I hope this might help you too.
EDIT: oolite running for over 2 hours now in fullscreen and windowed mode, with all sorts of complex 3d things happening, and no trouble at all. It's rock solid.