Okay, I think I found it. Running Firefox under the logged in user seems fine, but running Firefox as a different user (what I usually do) shows the massive performance problem. Must be X11 pixmap caching or X11 quirk or something. $DISPLAY is :0 for both accounts.
It also seems to be a long standing problem with Firefox vs nVidia drivers:
Does Nouveau support triple-head + rotation?