I just want to tell you I've got the same issue.
I also have a Dell XPS M1530 with a 8600M GT. I'm running Archlinux and using Gnome 3. I use mplayer + vdpau too. Driver version : 270.41.06.
AFAIK, I have this issue since Arch switched to Gnome 3, but I cannot tell for sure.
I tried to fix this, but couldn't. The only thing I managed to do in order to watch H264 videos was switching back to vo=xv int my mplayer config file, but that's not really a solution.
If anybody has fixed it, I'd be glad to know how.