It seems to me that what you're having is the old, known "out of vram" problem.
You can try this little trick here: http://smspillaz.wordpress.com/2007/...ur-video-card/
It should decrease video memory consumption with mplayer's XV vo driver.
It has few other advantages as well :-)
I saw in your bug report that you're using Gentoo there, so here you go:
An ebuild for MPlayer svn, shich includes the patch mentioned above and also sports the coolest vo driver you'll ever see - matrixview!
Everything goes to media-video category.