VDPAU hangs with high-resolution H.264 on GTX 460
I have a weird problem with VDPAU on my GTX 460 1024 MB card:
Hardware accelerated playback usually works fine for resolutions up to 720p (including hiqh quality scaling and deinterlacing), but acts up on 1080i/p files. I only use MPlayer.
When playing these streams, playback is fine until some random point where X completely hangs (i.e. mouse cursor not moving, audio playback from MPlayer stopping) for a short duration of about 2-5 seconds. After that playback continues as if nothing happened. It seems that the more complex the source is, the more frequent and longer the hangs are. Interlaced sources seem to affected worse than non-interlaced ones, even if deinterlacing is disabled.
I've uploaded a sample ("VDPAU_hangs.ts") that should exhibit the problem. Attached to this post is debug output from a typical problematic MPlayer run.
If playback doesn't hang on its own, try moving or resizing the window which should reproduce it reliably.
Playback on Windows is fine, just as it is on Linux when using other NVidia GPUs.
Software decoding with VDPAU display also works fine in all cases. Only hardware decoding seems to be affected.