Stumped!! Can't run more then 3 NVIDIA Cards!?!

[alvinist]

Can anyone out there offer any advice? For some reason I can't get more then 2 nvidia cards working!! I keep getting this in my X.log:

(**) NVIDIA(2): ConnectedMonitor string: "dfp,dfp"
(EE) NVIDIA(2): Failed to initialize the NVIDIA graphics device PCI:1:0:0.
(EE) NVIDIA(2): Please see the COMMON PROBLEMS section in the README for
(EE) NVIDIA(2): additional information.
(EE) NVIDIA(2): Failed to initialize the NVIDIA graphics device!

I've looked in the COMMON PROBLEMS section but none of that helped.

I'm running an Asus P5W64 board with 4 PCI-Slots & 4x GeForce 7300 GT.

Promise to buy whoever a very decent lunch (cash by paypal) if you can help me get this going. I'm getting desperate!!!

Thanks for any advice!!

Regards,

-Al



cat /proc/driver/nvidia/cards/*
Model: GeForce 7300 GT
IRQ: 16
Video BIOS: 05.73.22.50.53
Card Type: PCI-E
DMA Size: 39 bits
DMA Mask: 0x7fffffffff
Bus Location: 08.00.0

Model: GeForce 7300 GT
IRQ: 17
Video BIOS: 05.73.22.50.53
Card Type: PCI-E
DMA Size: 39 bits
DMA Mask: 0x7fffffffff
Bus Location: 07.00.0

Model: GeForce 6200
IRQ: 23
Video BIOS: ??.??.??.??.??
Card Type: PCI
DMA Size: 32 bits
DMA Mask: 0xffffffff
Bus Location: 01.00.0

dmesg | grep -i nvi
nvidia: module license 'NVIDIA' taints kernel.
NVRM: loading NVIDIA UNIX x86 Kernel Module 100.14.19 Wed Sep 12 14:12:24 PDT 2007

[Revs]

Didn't think this was possible yet. No quad SLI drivers.

[Thunderbird]

This should work though I have never seen someone connecting 8 displays but it should work. The config file looks ok from a quick glance. It sounds like some device initialisation issue which can be related to acpi / irq stuff.

I'll paste the full description from the docs:
Q. My X server fails to start, and my X log file contains the error:

(EE) NVIDIA(0): The NVIDIA kernel module does not appear to
(EE) NVIDIA(0): be receiving interrupts generated by the NVIDIA
graphics
(EE) NVIDIA(0): device PCI:x:x:x. Please see the COMMON PROBLEMS
(EE) NVIDIA(0): section in the README for additional information.


A. This can be caused by a variety of problems, such as PCI IRQ routing
errors, I/O APIC problems or conflicts with other devices sharing the IRQ
(or their drivers).

If possible, configure your system such that your graphics card does not
share its IRQ with other devices (try moving the graphics card to another
slot if applicable, unload/disable the driver(s) for the device(s) sharing
the card's IRQ, or remove/disable the device(s)).

Depending on the nature of the problem, one of (or a combination of) these
kernel parameters might also help:

Parameter Behavior
-------------- ---------------------------------------------------
pci=noacpi don't use ACPI for PCI IRQ routing
pci=biosirq use PCI BIOS calls to retrieve the IRQ routing
table
noapic don't use I/O APICs present in the system
acpi=off disable ACPI

[netllama]

Please see the forum sticky posts.

[etherfish]

I realize netllama already answered you, but I wanted to chime in.

I can verify that Linux can run more than 3 video cards, specifically, I have a machine running 6 concurrently. The first is AGP, the next 5 are PCI. Fortunately, the 5 PCI cards are distributed across 3 different PCI busses. The machine itself is a Dual Xeon 2.66GHz. We're using xinerama to stitch together six twinview setups of 1024x768 LCDs for a total screen size of 4096x2304. We can do HW accelerated OpenGL over nearly the entire screen - some part of the NVIDIA drivers or libraries choke when we create a visual too close to 4096 pixels wide.

Here's an excerpt from the Xorg.0.log:
(--) PCI:*(1:0:0) nVidia Corporation NV34 [GeForce FX 5200] rev 161, Mem @ 0x71000000/24, 0x90000000/28
(--) PCI: (3:1:0) nVidia Corporation NV34 [GeForce FX 5200] rev 161, Mem @ 0x73000000/24, 0xa0000000/28
(--) PCI: (3:2:0) nVidia Corporation NV34 [GeForce FX 5200] rev 161, Mem @ 0x74000000/24, 0xb0000000/28
(--) PCI: (4:1:0) nVidia Corporation NV34 [GeForce FX 5200] rev 161, Mem @ 0x76000000/24, 0xc0000000/28
(--) PCI: (5:0:0) nVidia Corporation NV34 [GeForce FX 5200] rev 161, Mem @ 0x77000000/24, 0xd0000000/28
(--) PCI: (5:1:0) nVidia Corporation NV34 [GeForce FX 5200] rev 161, Mem @ 0x78000000/24, 0xe0000000/28


The computer is named, of course, "Mr. Universe."

[alvinist]

WOW!!! Can you tell me what motherboard you have?

Thanks!

[alvinist]

Thanks much for the reply! I've tried all of that but no luck....

[etherfish]

The machine with 6 cards is an Intel Corporation SE7505VB2 server motherboard. Also, fwiw, my workstation has three Quadro 3500's working correctly. lspci posted below.


In your post, you indicate: "I'm running an Asus P5W64 board with 4 PCI-Slots & 4x GeForce 7300 GT." However, the output below that you posted indicates two PCI-E 7300GT's and only 1 PCI 6200GT. It doesn't help that you're attempting to run two similar cards and an extra card from an older generation. But, I guess you must have been testing things.

Also, just an observation, your motherboard's PCI-E slots can be set to one of two configurations:
configuration one: x8, x8, x4, and x8 mode
configuration two: x16, x0, x4, and x0 mode

I don't know much else about the motherboard, maybe you should check it to confirm your PCI-E slots are working in the right configuration? Use lspci to verify. Perhaps there is a BIOS setting?

Can you post the lines from the Xorg log that have NVIDIA(2) in them?
grep NVIDIA\(2\) /var/log/Xorg.0.log

Can you also post the output from the following command:
grep width /var/log/Xorg.0.log


my lspci excerpt:
02:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500 (rev a1)
07:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500 (rev a1)
86:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500 (rev a1)

[TonyT]

from what I know, you can really only run 2 cards as video cards, using the 3rd card as a Physics card...for Quad SLI you need 2 7950 Gx2 which each have 2 cores....

(2 cores x 2 cards in SLI = )

Tony T.

P.S.

I heard some rumors about ATI comming out with "trifire" using all 3 GPU slots on some mobos (pci-e or pci-x)

but then again, who wants ATI? the nVidia guy clearly PWN3D the ATI guy and the ATI guy got crushed first.

[zander]

@TonyT: that is not the case; the NVIDIA Linux graphics driver supports up to 8 GPUs at this point, provided the host system can support this number of GPUs (i.e. enumerates them correctly, assigns resources and gets interrupt routing, etc., right). Note that the desired configuration in this case does not involve SLI FrameRendering.

[alvinist]

Very nice! Yes, I did have 4 GeForce 7300 GTs but since that setup didn't work I decided to do some testing. For whatever reason I can't seem to get more then 2 Nvidia cards to initialize but under Windows everything works just fine so I'm a bit disappointed. Even if I have 12 screens working under Windows it will do me no good ! I'm sure most of us agree in this forum.

Have any pics of your system you can post? I would really like to see it!

Thanks!

Regards,
-Al

Originally Posted by etherfish

The machine with 6 cards is an Intel Corporation SE7505VB2 server motherboard. Also, fwiw, my workstation has three Quadro 3500's working correctly. lspci posted below.


In your post, you indicate: "I'm running an Asus P5W64 board with 4 PCI-Slots & 4x GeForce 7300 GT." However, the output below that you posted indicates two PCI-E 7300GT's and only 1 PCI 6200GT. It doesn't help that you're attempting to run two similar cards and an extra card from an older generation. But, I guess you must have been testing things.

Also, just an observation, your motherboard's PCI-E slots can be set to one of two configurations:
configuration one: x8, x8, x4, and x8 mode
configuration two: x16, x0, x4, and x0 mode

I don't know much else about the motherboard, maybe you should check it to confirm your PCI-E slots are working in the right configuration? Use lspci to verify. Perhaps there is a BIOS setting?

Can you post the lines from the Xorg log that have NVIDIA(2) in them?
grep NVIDIA\(2\) /var/log/Xorg.0.log

Can you also post the output from the following command:
grep width /var/log/Xorg.0.log


my lspci excerpt:
02:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500 (rev a1)
07:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500 (rev a1)
86:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500 (rev a1)

[alvinist]

Zander,

Does each card needs its own dedicated IRQ? Under linux my setup won't work but under Windows XP every works fine...

Thanks

Originally Posted by zander

@TonyT: that is not the case; the NVIDIA Linux graphics driver supports up to 8 GPUs at this point, provided the host system can support this number of GPUs (i.e. enumerates them correctly, assigns resources and gets interrupt routing, etc., right). Note that the desired configuration in this case does not involve SLI FrameRendering.