Uhm, the point of XRender is to offload rendering tasks to the GPU. Of course software fallbacks get faster if you have a fast CPU, but that's not the idea.
Many drivers only seem "fast" at XRender because they don't accelerate much and software fallbacks are used all the time.
NVidia's drivers and (starting with ~2.11) Intel's drivers accelerate most operations that are feasible, given the hardware. The downside is that operations in the XRender pipe that cannot be accelerated and cause software fallbacks can be really slow, as pixmaps need to be transferred from gpu to host and back again. Sometimes even multiple times; this is referred to as migration ping-pong. Drivers try to detect and workaround such situations, but that's not always possible.
TL;DR if XRender hardware acceleration is supported, works and used correctly, it is ****ing fast. If not, it can cause pathologically slow software fallbacks.