Hello all, and big thanks for NVIDIA for VDPAU.
Just thought to summarize few things that helped me get VPDAU mplayer to level that it is now (so that I can link to this post instead of explaining the same thing again and again in irc

:
- 180.18 beta drivers
- latest mplayer-vdpau patch
To get the thing running in first place:
- removed -ass subtitle option from command line as it uses filters which do not work in current implementation (not very informative error code/behaviour in this one..)
To get most h264 files to play:
- change line 705 of lib_vo/vo_vdpau.c to max_references = ((12 * 1024 * 1024) / surf_size) + 11;
To remove tearing:
- disabled x compositing
- closed other overlay apps (vdr in my case)
To remove slight jerkiness in playback:
- added -cache 8192 to command line
+11 thingy didn't cause me any side effects, but then again I have 512 megs on my 8500GS and I do not run any other vram heavy things like compiz..