I had the same problem (with kernel 2.4.20-18.7), but I couldn't even switch back to console (go ssh!). Apparently something badly messes with the hardware.
My solution: Install the driver again, using the -n switch (do not use precompiled kernel interfaces). That fixed the problem for me.
I've had this problem with both the 4496 and 4363 driver. The solution is only tested for 4496.