I did that with my old gfx, GF9500 256MB (I had too low memory errors with VDPAU) but this config is long gone.
Here is the only thing that left - maybe you will figure it out.
~/.gnome2/nautilus-scripts/Play\ with\ VDPAU...
/usr/bin/nvidia-settings --config=/home/dk75/.nvidia-settings-rc --ctrl-display=:1 --load-config-only
/usr/local/bin/mplayer -noconfig all -vo vdpau -vc ffh264vdpau,ffmpeg12vdpau,ffvc1vdpau,ffwmv3vdpau -demuxer lavf -alang eng,en,jpn,jp,pol,pl -slang eng,en,pol,pl -mc 2 -softvol -softvol-max 200 -ass -fontconfig -fs -tskeepbroken -tsprobe 100000 "$1"
gksudo xhost +localhost; sudo xinit /home/dk75/.xinitrc.vdpau "$1" -- :1 -config "/etc/X11/xorg.conf-vdpau" -depth 24 -audit 0 -auth /etc -nolisten tcp -a 2 &
is basically this same as you primary one but with added at the end
Option "Composite" "Disable"
you could prepare different (with new name, like .nvidia-settings-rc-vdpau
so it will be not replaced during normal desktop settings testing) or not.
With that you run any VDPAU accelerated file with RightMouseButton on it and by choseing Play with VDPAU...
from pulldown menu.
And it should close this new separate screen after closing mplayer.
OK. I've tested it right now and it works. The only thing is that it needs user to type its password and user must be sudoers