no, you aren't insane... : )
I also use the xv output.
My xserver doesn't crash the 50% of times, but quite rarely.
It never crashed after I disabled kdm.
I will post my log if it happens again.
Maybe this problem depends by videocard or agp chipset?
I have a gf2 mx400 (asus v7100 pro) with tv-out (I enable it by nvtv when needed, not by XF86Config).
My mobo is an Asus via694X. I use agpgart with agp_try_unsupported option. This is the only way I can get agp working on my mobo.
Is your configuration similar?