I was playing around some more with this. It appears for me at least with mplayer, using the video output drivers x11 or gl2, I do not get the jaggies. Also in xine if i use Xshm (which i believe is the same as x11 in mplayer) I do not get the jaggies.
So the output drivers that cause the jaggies for me are xv and sdl. The output drivers that do not are x11 and gl2.
These are the commands I used:
$ mplayer -vo gl2 -dvd 1 -fs
$ mplayer -vo x11 -dvd 1 -zoom
Now, i am not sure if the -fs or -zoom features are using software or hardware fullscreen which could just be masking or working around the jaggie problem. Anyone else have some input?