F16/290.06: Unstable FPS on Xeon/470, working just fine on Athlon 635/210.

[gilboa]

Hello all,

As the title suggest, since the upgrade to Fedora 16 (coupled w/ 3.1 kernel and v290.06 drivers) I'm having extremely unstable fps on my Xeon workstation [1], while my reference machine [2] is showing no signs of having any type of performance issues.
Using the 2.6.39 kernel and v285 drivers, my Fedora 15 machine showed no signs of this issue.

Description:
While playing spring RTS or UT2004 the fps will be well-over 200fps, and every couple of seconds (no matter of the game is idle or active) the fps will drop down 30-40fps for a second or two (or even momentarily hang), and then jump back to the previous level.
In order to remove any type of external noise, I've disabled all the services and shut down all the VM's, and limited both games to cpuset 0,1 (to reduce the core hopping).
Memory usage was ~300MB out of 12GB.

Looking at mpstat, I could see that both games were eating their usual 95-100% CPU time until both suddenly drop to 50-70% CPU with systime being jumping to 30-50%.
As far as I could see, (using top and iotop) there was no sign of any external process jumping in.
I've tried to use oprofile to locate the reason for the sudden drop, but as far as I could see, there was none.

Machine specs:
[1] Workstation:
2 x Xeon 5680 (HT and VT are enabled; IO-VT is disabled.)
6 x 2GB RAM (6GB in each bank).
1 x GTX470 w/ v290.06 drivers.
5 x 320GB SATA in SoftRAID5.
Fedora 16, Linux 3.1.1.

[2] Reference:
1 x Athlon 635 (Quad core)
4 x 2GB RAM.
1 x GT210 w/ v290.06 drivers.
3 x 500GB RAM SoftRAID5.
Fedora 16, Linux 3.1.1.

Bug report attached.
Please let me know how I diagnose this issue.
(Rerun oprofile looking for something in particular?)

- Gilboa

[gilboa]

Bug report attached.

[kwizart]

FYI I will push the current nvidia release to the new kernel in updates-testing (3.1.2-1.fc16) later today.
You should be able to reproduce with 290.10

[gilboa]

I'll update the kernel + driver to the latest version in -testing and report back.

[ars]

I have the same, problem, But i have Debian Testing. Using the 285 and 290 drivers with the kernel 3.1 appear the problem, but with kernel 3.0 and the 285 there are not problems.
I will try to prove the kernel 3.0 and 290 drivers.

[kwizart]

The nvidia driver 290.10 is pushed to the testing repository, you can get it along with the new fedora kernel using:
yum update --enablerepo=updates-testing,rpmfusion-nonfree-updates-testing xorg-x11-drv-nvidia

[gilboa]

3.1.2-1.fc16.x86_64 + v290.10 showing the same problem - though as I much as I can see, the occurrence is somewhat less frequent.

- Gilboa

[ars]

I tested the 290.10 drivers with the Kernel 3.0 and the performance is good. The problem is with the Kernel3.1.

[gilboa]

Originally Posted by ars

I tested the 290.10 drivers with the Kernel 3.0 and the performance is good. The problem is with the Kernel3.1.

Umm... Sadly enough F16 doesn't have 3.0.x kernels for me to test.

- Gilboa

[gilboa]

Bump?

[gilboa]

FWIW F17 / 295.53 / 3.4 exhibit the same problem. I've got a bad feeling this issue is scheduler related, but I have no idea how to test it. (Plus, given the use of binary drivers, I doubt that a kernel bugzilla entry will gain much attraction....)

- Gilboa

[gilboa]

I believe I just got nominated for the "Weird solution of 2012" award.
While trying to debug this issue, I ran games with oprofile running - and sadly enough nothing sprang into view.
*However* (drums!), due to the late hour / lack of sleep / far-too-many-hours-at-work, I mistake-ably ran sysprof instead of oprofile (no idea how or why, don't ask) and Shazam! stable FPS.
as in rock-solid.

Stop sysprof, and the FPS shutters like a drunk on far too many bottles of whiskey. Start it again, and it's smooth sailing.

For the life of me I cannot begin to think where I should report this issue. Weird!

- Gilboa