Regarding the mplayer issues, I believe that at least some of those are caused by mplayer2. I've been experimenting with mplayer2 on and off for quite some time and I've always reverted back to vanilla mplayer because of all sorts of weird issues including the green flashing.
Possible freezes when toggling between FS<->window are caused by KWin unredirecting. You can either disable unredirecting for fullscreen apps (System settings->Desktop effects->Advanced) or create a rule in KWin to disable compositing when SMPlayer is running. I can't tell if it's KWin or VDPAU at fault here.
Originally Posted by RealNC
Btw, if you do this, then you aren't really using VDPAU. You're decoding on the CPU.
Even though the decoding is done on CPU, the video is still being drawn using VDPAU. Since your problems go away when you switch compositing off, it's probably a drawing problems.
I'm not having any problems with Flash on 9800M GTS, you can also disable the HW decoding altogether by adding "EnableLinuxHWVideoDecode=0" to /etc/adobe/mms.cfg. My over 3 years old laptop can handle 1080p Flash videos without VDPAU with ease.