Well, I found a workaround to my problem. I discovered that the crash happens when you close X, not when you start it. So, now, instead of having the second X server "on demand", I have it running at the time. Switching between VTs works like a charm. Just don't try to logout or everything will go down the drain (at least with my driver/card combination).