I am having a similar issue also. I've had a myth box set up for about a month now, and have been unable to run it in 1080i because the previous drivers always locked up the X-server when I tried to play any type of video. With the newest driver all video plays fine, except 1080i transport streams.
I've tried running with XVMC and with only XV, and XV seems to work the best (i get about 30-40% CPU utilization on my Athlon 64 3200+), but I get a stuttering that comes and goes (maybe 10 seconds good, 10 seconds bad). I also see a refresh rate of 60.05 in nvidia-settings, no matter the modeline I use.
I was originally using the onboard 6150 on my Asus A8N-VM CSM, but I figured the stuttering might be a problem caused by not enough bandwidth to the 6150 or something, so I bought a PCI-E 6600 LE. Same exact behavior, so I'm figuring it's a driver issue. If you guys can get this one last bug worked out, I'll be incredibly happy!