I would like to mention that I am experiencing the exact same issues and they are really getting on my nerves. I have tried a lot of searching to find the cause of this problem.
To really see just how much this is bothering me take a look at this thread that I made to the XOrg guys
in an attempt to get them to help me.
I have no idea why this is happening but I would really like to get to the bottom of the problem. I have attached a bug report to this issue but I would like to note that the bug report was generated when the issue in question was NOT happening.
I will come back to see updated posts and to submit a better bug report when the actual issue happens again.