What distro is it that you are using ???
I use Gentoo and and periodicallly I have to rebuild my font cache after updating certain things (kdelibs, Xfree etc etc) otherwise X takes along time to start.
Perhaps you should try doing this ??
The command i use is fc-cache (could be different in your distro) as root.
Give it a go, it cant hurt