I also experience the same problem from time to time, and find it to be very annoying. I mostly use mplayer, and it seems to occur after I watch a few videos - and the behavior isn't very predictable. But I have found that I can solve the situation by simply restarting X.
I'm running 4496 on a GeForce4 Ti4200, on an athlon 1.33 Ghz.
Also, a while back it seemed to occur fairly often and so I disabled agp support in my XF86Config. This seemed to decrease the frequency of the problem, but it still occurs every so often.