I finally solved my problem.
After trying the same version on my winXP partition, it showed the same problem, took much longer to boot winxp with that version. So I installed 31.23 and now everything is running smooth, 31.23 does not slow X by much, a few second is all.
On more thing was the 2d performance with 41.91. I couldn't even drag a window without the entire system slowing down.
Don't use 41.91 go for the older version.