Well I seem to have solved my problem. I was messing around trying different driver version and decided to try the newest beta (177). It gave the exact same behavior I was receiving with the 173 version and while searching around a bit more I came across someone who had a Xeon processor with 4 GB of RAM, like me. He said he dropped down the RAM to 2GB and it seemed to have cured his problems. At this point I thought to myself, self, wouldn't it be stupid if that solved your problem too?
Lo and behold, it did. I don't like having to use 2GB of RAM when I have 2x2GB sticks so I can't even salvage a mere 3GB, but I'll do what I have to I guess. Please, oh please can we have AMD64 NVIDIA drivers soon?