Only 2 out of 5 GPU's starting

[jbarsellotti]

We are trying to drive 9 monitors (a 3x3 array) using two Quadro (NVS 440) and one GeForce 7300GS.
Motherboard - Asus Striker Extreme
CPU - Intel 6700 quad core
OS - Fedora 7 (and briefly Ubuntu 7 - same result)

The gpu's for the first Quadro are PCI:3:0:0 and PCI:4:0:0
For the second Quadro PCI:10:0:0 and PCI:11:0:0
And for the GeForce PCI:6:0:0

It appears that only the first two gpu's found in the "ServerLayout" section are started. We have changed the layout and a different two are started.

We also tried each of pci=noacpi, pci=biosirq, noapic and acpi=off - none seem to have any effect.

For gpu's that fail to start Xorg.0.log simply states -

"Failed to initialize the NVIDIA graphics device PCI:4:0:0"
"Failed to initialize the NVIDIA graphics device PCI:10:0:0"
"Failed to initialize the NVIDIA graphics device PCI:11:0:0"

In this case the gpu's at PCI:3:0:0 and PCI:6:0:0 are starting

Please see the attachments -

xorg.conf
Xorg.0.log
cards (the result of "cat /proc/driver/nvidia/cards/*")

[netllama]

I'd guess that you're hitting the kernel's vmalloc limitation, however without a bug report, I have insufficient information to determine what is wrong. I'd suggest reading the driver README as well as the forum sticky posts.

[zander]

Please generate and attach a complete nvidia-bug-report.log file after the problem has reproduced (see http://www.nvnews.net/vbulletin/showthread.php?t=46678); please also see the section on kernel virtual memory exhaustion in the README (http://us.download.nvidia.com/XFree8...hapter-09.html).

[jbarsellotti]

4 out of 5 gpu's working !!!

I first tried vmalloc=256MB alone which caused the nvidia kernel module not to load. Adding the uppermem line fixed it and 4 gpu's started. I added another 128MB (which should have been ample) but still only 4. I know gpu 5 is good. It seems there must be some other problem.

grub now looks like this

title Fedora (2.6.21-1.3228.fc7)
uppermem 524288
root (hd0,4)
kernel /boot/vmlinuz-2.6.21-1.3228.fc7 ro root=/dev/md5 video=vesaff vga=normal vmalloc=384MB
initrd /boot/initrd-2.6.21-1.3228.fc7.img

I have posted the bug report

john

[netllama]

The only thing that I see in the bug report is:
(EE) NVIDIA(8): Failed to initialize the NVIDIA graphics device PCI:11:0:0.
(EE) NVIDIA(8): Please see the COMMON PROBLEMS section in the README for
(EE) NVIDIA(8): additional information.
(EE) NVIDIA(8): Failed to initialize the NVIDIA graphics device!

Was this bug report generated while the problem was present?

[jbarsellotti]

The problem has changed from the original. Since adding "vmalloc=256MB" 4 out of 5 gpu's are starting. Increasing to 384MB did not help.

When I did "startx -- -logverbose 6" 4 out of 5 gpu's started. I then created the log.

john

[netllama]

This doesn't look like a vmalloc exhaustion issue any longer (as you noted). The only error in your last bug report was what I noted above.

Its possible that your motherboard isn't capable of initializing 5 GPUs. Have you verified that you're using the latest BIOS?

[jbarsellotti]

The bios is the latest (as is everything else). One thing I didn't mention before because I thought it would be easy to straighten once all gpu's are working - but now I'm not so sure.

If we label the intended layout as (a 3x3 array)

1 2 3
4 5 6
7 8 9

With 4 out of 5 gpu's we get

1 2 7
6 6 7
x x 3

It seems the "ServerLayout" section is being ignored. I changed from relative to absolute layout as follows (and changed the default layout to layout1)

Section "ServerLayout"
Identifier "Layout1"
Screen 0 "Screen0" 0 0
Screen 1 "Screen1" 1280 0
Screen 0 "Screen2" 2560 0
Screen 1 "Screen3" 0 1024
Screen 0 "Screen4" 1280 1024
Screen 1 "Screen5" 2560 1024
Screen 0 "Screen6" 0 2048
Screen 1 "Screen7" 1280 2048
Screen 0 "Screen8" 2560 2048
InputDevice "Keyboard0" "CoreKeyboard"
InputDevice "Mouse0" "CorePointer"
EndSection

and got

1 2 5
6 4 7
x x 3

Rather odd - but at least it can be fixed by moving cables


jb

[jbarsellotti]

I've decided to abandon this - at least for now.

This morning I installed a 64 bit OS and all gpu's are starting.

In the past, every attempt to run 64 bit failed because there were just too many things I couldn't get to work properly. One of them was xinerama! It was just too easy to switch back to 32 bit where everything worked perfectly (that's when I was using only four monitors). Xinerama is now working and I'll be more persistent with the rest.

Thanks for the help

jb