Re: Java3D anti-aliasing
OK, but the way those properties are used still makes no sense to me, oh well. This doesn't mean the problem is not Java or Java3D. What versions of Java and Java3D are you using? Also, run the command glxinfo | grep multisample, if you find GLX_ARB_multisample and GL_ARB_multisample in the output then the problem is likely in Java or Java3D.
The reason forcing AA on in the nVidia Display Settings makes it run so slow is because not only is the window being forced to AA, but so is the rest of your desktop (since you're running Compiz-Fusion). The forced AA might run better if you turn off Compiz while running the program. Also, just a random crazy idea, try turning off Compiz and running the program (no forced AA) and see what happens.
Might also see if Mesa is interfering with nVidia's OpenGL libraries, though since the properties are reporting it as nVidia this probably isn't the case.