I have the same problem with the EDID based 1920x1080 interlaced mode that my TV reports. Basically, my guess is the driver doesn't calculate the 1080 when the TV responds with 1920x540 interlace. So, the resolution that the driver sets up is 540interlaced with a virtual desktop size of 1920x1080, which ends up showing the top portions to the TV at 270lines of resolution.
Using this modeline for my TV:
ModeLine "1080i3" 148.4 1920 2008 2052 2200 1080 1084 1094 1125 +hsync +vsync interlace
Gives me 1920x1080 interlaced output. Basically I doubled all the values for the vertical info from the EDID and also doubled the clock rate (from 74.18 rounded up). From the logs, this effectively gave me:
1920x1080 interlace at 60hz, which I think is correct for 1080i.
Beta driver 9742 is the first release that has enabled me to play videos via XV/XVMC over DVI without lockups on PCI-E or onboard 6150! So I'm super happy about that, although the jitter problems still persists for interlaced videos. The quick and dirty solution for me on the jitter/stuttering was just to deinterlace, which wasn't necessary with 8183. Searching and playing around, my guess is something got changed with vertical syncs after 8183... the jitters doesn't happen at the same spots everytime, sometimes it's smooth and othertimes stuttery, so perhaps a timing issue. With mythv, turning on opengl v-sync smoothed things out, but causes mythtv to crash after each video, so for now I'll stick with deinterlacing 1080i content for 1080i output (upgraded to FC6 so 8183 isn't a viable fallback anymore).
I figured perhaps it was a speed problem, but this happens with 6150 onboard, 6200 TC, 6600 and 7900GT. The jitters were there with VGA/Component output too, so not limited to just DVI.