Linux drivers may be compiled in the kernel or as modules. Modules don't need a full kernel recompilation.
Nvidia drivers work as modules (because they are closed-sourced). So, as a matter of fact, there's no need to recompile your kernel from scratch. All you have to do is 'make' the module against your current sources. But the installer does all the job for you. All you need are the kernel-sources and the 'make' libraries (as well as the gcc libraries).
(I answered in English just in case someone else needs this answer).
Use the source, Tux.