X lockup, 4 head OpenGL

[Stephen Webb]

(console error is printed when the lockup happens: "Disabling IRQ #66" )

I'm running two 6600 GT PCIe graphics cards in an ABIT NI8-SLI motherboard. The motherboard is set in "sli mode" (8 pcie lanes per card), but the cards are *not* connected with the SLI connector.

X is running with 4 independent monitors, not with xinerama or twinview or anything.

Kernel is:
Linux renderbox 2.6.13.4 #3 SMP Tue Oct 25 23:17:27 EDT 2005 i686 GNU/Linux
(using dual core P4 processor)

OpenGL works on all displays, but when I simulatenously run intensive OGL apps (glxgears in non-fullscreen mode doesn't seem to cause problems) on two displays of the same card the machine will lock up or at least get into a very unresponsive state. If I can successfully kill the apps, X doesn't work. Killing and restarting X doesn't fix things either (not fully, anyway).

1. If I run the openGL apps on outputs of separate cards, no problem.
2. If I *remove* one of the cards, and run on the two outputs of the singe card, no problems.


I tried attaching the bug report...problems with the interface..not sure.

get it here:
http://web.qx.net/lizjones/nvidia-bug-report.log.gz

Any help appreciated. I'm about to go crazy.

-Steve

[netllama]

Both of your nvidia cards are sharing IRQ 66. That is not good, and would need to be corrected in the BIOS. If you're not using the latest BIOS, I'd suggest doing so.

Thanks,
Lonni

[Stephen Webb]

Thanks for the response.

I flashed the BIOS, but no luck.

Both cards are still using the same interrupt, and as far as I can tell there isn't an option in the BIOS to change this. Is there a kernel boot parm, or module loading parm that controls this?

Can you suggest a different motherboard that is known to work under Linux? I'm using an Abit NI8-SLI motherboard -- but would be happy to get some other SLI board if it will fix this problem.

Thanks again

-Steve

[netllama]

In your bug report I see:
irq 66: nobody cared (try booting with the "irqpoll" option)

Have you tried booting with the irqpoll option?

I can't recommend specific motherboards, however if you search through the threads here, I'm sure you can find those which are more problematic than others.

Thanks,
Lonni

[JaXXoN]

Originally Posted by Stephen Webb

Can you suggest a different motherboard that is known to work under Linux?

I'm having a triple-head setup with two 7800GTX (xinerama, first card in
twinview mode) in an Asus A8N-SLI Premium: i can play UT2004 for
hours (at 3840x1200), but with three glxgears running on the twinview
screens, it may happen that one or more glxgears windows may
"hickup" (hanging for a seconds or two, then proceeding) - this might
be related to your problem.

Other than that, the system runs rock stable as long as you don't try
to switch to text console, do CTRL+ALT+BACKSPACE or try to run
"nvcone-crash".

regards

Bernhard

[Stephen Webb]

"Did you try booting with the "irqpoll" option?"

Yes, I did. It didn't really help. It _may_ have made the system not fully lock up, but I can't say for sure. (I had a login shell open to the machine, and after the "lockup" I could still occassionally gain control of the machine (well enough to reboot, but it was very very unresponsive)

Regarding the shared interrupt being a problem, I found this on the net -- I don't know the credentials of the source, but it seemed interesting in light of what is going on with me:

Anybody who tells you "our device needs its own IRQ" is probably reading from a script written in the ISA days. If not, they're telling you that their device is not compliant with the PCI specs, and your best bet is to give it no IRQ at all. In other words, get it out of your machine completely.

I also started poking in the nvidia driver code and found this:

...
status = request_irq(nv->interrupt_line, nv_kern_isr,
SA_INTERRUPT | SA_SHIRQ, "nvidia",
(void *) nvl);
...

Two things are interesting to me. First is the nv->interrupt_line parameter. I wonder if I could hack this part of the code to request different interrupt lines for the two cards. I'm going to try to do something along those lines.

The second thing that is interesting to me is the SA_SHIRQ flag, which tells the kernel that the interrupt is shared. It seems that if the driver is informing the kernel that it is requesting a shared interrupt, then shared interrupts must be ok -- or at least that's how I'm reading it.

Could someone out there with a *working* dual card setup tell me what their /proc/interrupts file looks like. I'd like to see how your system is set up.

Thanks
-Steve
(I'm ordering a Gigabyte GA-8N SLI royal motherboard...If that doesn't fix it I'm going to have to talk to the devil...ATI

[Stephen Webb]

Update:

I'm still having trouble with X locking up and "IRQ XXX disabled" messages.

I am on motherboard #3 now -- this one does actually assign two different IRQs to the different cards, but my problem persists.

To recap:

Linux kernel 2.6.13-4
Two FX 6600-GT graphics cards, not connected with the SLI bridge connector.
4 independent X screens

Basic openGL programs don't seem to cause problems (glxgears, ssystem...) but my program does.

I am using "advanced" features of the drivers, such as fragment shaders, glXGetVideoSyncSGI(), glXAllocateMemoryNV(), etc.

The lockups (not always a total system freeze) are often preceeded by a "Disabling IRQ xyz" error.

It only seems to happen when all of the below conditions are true:
a) Both CPU cores are active
b) I am using two video cards
c) I am using both outputs on at least one card.

Is there any way the "new driver" will address my problem?

My only other option (that I know of) is the quad PCIe motherboard from Gigabyte, but I don't think I can buy that yet (anyone know?)

Thanks again

-Steve

[Stephen Webb]

Correction:

ssystem does lock the system up as well. It just takes quite a bit longer.
(and I get the familiar "Disabling IRQ #106" error message)

-Steve

[neversleep]

Stephen,

Did you find a solution to this issue? I have a very similar problem.

-Ron

[netllama]

Does this problem exist with 1.0-8174 as well? If so, is the failure identical to what occurs with 1.0-7676?

-Lonni

[Stephen Webb]

I tried the new drivers, and unfortunately I get the same (or more or less the same) behavior.

The application freezes, I get a "disabling IRQ.." message (sometimes), and the machine (X server?) locks up -- sometimes I can kill the app / x server in time to prevent a lockup.

So, unfortunately, the driver didn't fix it.

-Steve

[neversleep]

Stephen,

I think I have found the solution to this. It appears to be a PCI/PCIE initialization issue.

See thread: (61616) http://www.nvnews.net/vbulletin/showthread.php?t=61616

Regards,
Ron