I'm not sure what that rmmod error means, but you should always remove Mandrake's drivers before trying to install a newer version.
And you do have to also be in text mode, with X not running, because the Makefile removes the old modules from memory before installing the new ones. If X is running, then the modules won't remove, which means the new ones won't install.
Once you have the nVidia drivers installed (and once you configured your XF86Config-4 file), you'll be able to get into X. Once you can get into X, then you can change to runlevel 5 if you want. But I'd install from runlevel 3, personally...