I'm not positive how to find out. If the system is set up one way, then doing an /sbin/lsmod
before starting X will list "rivafb" if you are using it. If it's not set up as a module, though, then it won't.
If you can find a config-xxxx file somewhere (perhaps in /boot) that matches your kernel, read through it and look for something like "CONFIG_FRAMEBUFFER_RIVAFB" or something. I don't remember what it's called exactly. Make sure it says "=n" after it; this turns it off. If it says =y, then you'll have to recompile a kernel to turn it off (when you're configuring the new kernel, select no for all framebuffer devices except "VESA VGA").
If it already says "=n", though, then rivafb isn't your problem.
Wait a minute -- are you hitting Ctrl-Alt-1, or Ctrl-Alt-F1? You need to use F1...