Yep, big difference
The kernel source package is the linux kernel sources for your distribution, AND the precompiled binaries for your kernel
What this does is that when you go to compile a linux hardware driver, it uses those binaries to correctly build your driver for your specific kernel.
The headers on the other hand are just a collection of *.h files that are also needed to compile drivers. Those header files give information to the compiler when compiling your binaries.
So yeah, make sure you have both installed on your system, and that the package version of both the kernel sources and headers are the same as your kernel version.