I'm giving up for the moment... It seems that when I change the XFree config in runlevel 3 (no X), and then try to come back to runlevel 5, the lib are not changed. (Before, I was doing everything with sax2, X running).
But then I get a nice black screen. With an error that I've already seen... Don't remember exactly, something about the display 0:0. I'll try to get the exact message and post it here.