[Stephen Warren]

stecklum, H.264 streams may require a different number of reference frames. Each reference frame requires memory. I don't know how many reference frames are required by qvdpautest or the stream you tested with, but this is the most likely answer. If you need more investigation, please take a look at the VDPAU problem reporting sticky.