I've been having exactly the same problem for ages, but nobody seems to have been able to help. I've submitted a bug report to NVidia, but to no avail.
I seem to get an X server crash about 50% of the time when starting any app which uses the Xv (Xvideo) extensions - e.g. xine, mplayer, xawtv, etc.
It's gratifying to know that I'm not going totally insane...