Is this your first attempt at getting nVidia X drivers going? Or did a previous version work for you? The 5328 drivers don't work on this motherboard (a Via one) but I can make do with some older ones. Still waiting for AMD64 / kernel 2.6 compatibility too.
If it is your first attempt don't forget to modprobe the nvidia module.
Don't know what could be screwing up your XFConfig file. That is darn weird.