After hours of trying, it is fixed.
I tried to compile a 2.16.20 kernel, only thing in the config I changed was "Support for video mode selection". Now I cant specify vga= on the bootline but I didnt use that anyways
Then I unmerged the portage nvidia driver and installed the one from the site. Dont ask me what of the 3 changes fixed it, but I don't care to be honest: glad this misery is over