Does the RAM usage keep growing forever? I suspect that RAM usage quickly grows to a very large value, but is then fairly stable.
I think your problem is simply the same as in http://www.nvnews.net/vbulletin/showthread.php?t=155849
. The symptom is simply that VDPAU will now use an extremely large amount of system memory. However, I don't think there's a leak per se; none of the memory is ever lost, it's just that some structure is now far larger than it should be, and hence consumes more memory before. This is probably more noticeable when playing streams with subtitles, since more VDPAU objects are allocated, which triggers this condition.