After downloading older drivers and trying several of them I found that version 1514 has practically no delay at all.
After a quick testing OpenGL programs seem to be working beautifully as well as mplayer and awtv. 1541 version doesn't even seem to be visibly slower on TNT than newer drivers.
Versions 2313 and 2880 cause a delay of about 10 seconds and 2960 nearly a minute.
For me this is a solution enough, though it would ofcourse be interesting to know what causes this delay in newer drivers.