a kernel just installs
- a kernel image (usually vmlinuz in / or /boot)
- System.map ( usually vmlinuz in / or /boot )
- kernel modules (usually in /lib/modules/<kernel-version>/ )
if you can boot your custom kernel leave vmlinuz and System.map in place, because they are the right ones.
you can leave also the kernel modules in place... they only occupy some space.
the most is in the kernel-source-tree. that one can be deleted safely. if you are unsure and your systems runs: leave it as it is, until you are sure what you are doing