Is this limitation due to lack of support in the driver? Or is vdpau simply not usable in linux without x regardless?
Well to begin with, the nvidia binary driver is X only, so obviously nothing is going to be supported on the framebuffer side of things by the binary driver. Perhaps vdpau support could be made to work with KMS (under the nouveau driver) but I'm not sure that is possible, and it would be of very low priority if it is.
In addition, X is perfectly fine for your purposes, provided you use a minimal window manager (or no window manager at all). Unless you are working with a less than 32mb system, I don't think RAM will be an issue, and that is the only area where a non-X system may be better.