No, it's not normal. From the driver README:
Q: X takes a long time to start. What can I do?
A: Most of the startx delay problems we have found are caused by incorrect
data in video BIOSes about what display devices are possibly connected
or what i2c port should be used for detection. You can work around
these problems with the XF86Config option "IgnoreDisplayDevices"
(please see the description in (app-d) APPENDIX D: XF86CONFIG OPTIONS).