Just another data point. I'm seeing similar behavior.
8800 512MB (G92)
Core 2 quad @ 2.80GHz
8GB system RAM
4xRE2 in a RAID10 config
Fedora 10 x86_64 (KDE desktop)
NVidia 180.51 drivers
MPlayer svn (pulled on 26 Apr 2009)
Test files are various H264 720p videos played from the local hard disk (max read STR = ~120MB/sec). For example, this freely-downloadable video (select the 1.4GB HD version):
With VDPAU enabled I don't see explicit MPlayer errors or warnings, just jerky playback, with very brief pauses every couple of seconds. VDPAU does indeed reduce CPU utilization but at the cost of performance.
In my experience, on this hardware:
VDPAU <-- lowest absolute CPU use; sporatically stuttering playback
default <-- middling performance and CPU use
S/W decode in 4 threads <-- highest CPU use; best playback performance