Using that metamode 1024x768@640x480, 640x480 apps in X think that the current resolution is 640x480 (atleast some apps like movie players think that). So when you use that metamode and launch a movie on the TV and you put it in fullscreen mode it will work correctly. (will verify that now ..)
The thing you want with Xserver switching is not possible. It is possible when you use more than 1 videocard but that will cause some other trouble.