It must be either your software or driver is faulty at some specified hardware.
I have no memory rush.
MPlayer with VDPAU and with OSD subtitles engine used 89M all the time (I've tested 30min. clip)
MPlayer with VDPAU and ASS subtitles engine started at 98M without sub, used 865M at start ASS engine and it stoped at 871M - it didn't repeated again.
Second run started ASS engine at 483M and ended at 634M after 5min to the near end. Then there was something strange - new chapter or something and it went to 763M. Still... it left 1.5G memory free for me so no problem. ;P
Thrid run started at 485M and growed to 634M in 5min. Then suddenly 763M near the end (28min of the clip).
It seems repeatable.
MPlayer with XV started at 48M with no sub and ended at 56M with ASS engine and 6 fonts loaded.
ION/9400M hardware with 260.19.06 driver
mplayer -nosub -noautosub -ass -vo vdpau movie_file.mkv
and I've turned subtitles ON by typing "J
" 2 min after clip started.