Can't see 2nd Quadro FX 4600 on CentOS 4.4

[solofx]

We have 2 Quadro FX 4600's in an HP xw9400 Workstation, but the nvidia driver will not recognize the second board. We get the following error from the nvidia-xconfig program:

# nvidia-xconfig -a

Using X configuration file: "/etc/X11/xorg.conf".
NVIDIA: could not open the device file /dev/nvidia1 (Input/output error).

WARNING: Unable to use the nvidia-cfg library to query NVIDIA hardware.

ERROR: Unable to determine number of GPUs in system; cannot honor
'--enable-all-gpus' option.

Backed up file '/etc/X11/xorg.conf' as '/etc/X11/xorg.conf.backup'
New X configuration file written to '/etc/X11/xorg.conf'

We have 2 identical systems and they are both having the same problem. We've swapped boards and power supplies and still only one board is working.

I'm attaching an nvidia-bug-report.log.

Thanks for any help you can give us!

[netllama]

I don't see an nvidia-bug-report.log. What is the output from lspci on this system?

[squeen]

I'm working with solofx on this...here's the machine's PCI map output:

/sbin/lspci
00:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
00:01.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
00:01.2 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a3)
00:02.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1)
00:02.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2)
00:04.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1)
00:05.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
00:05.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
00:05.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
00:06.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2)
00:06.1 Audio device: nVidia Corporation MCP55 High Definition Audio (rev a2)
00:08.0 Ethernet controller: nVidia Corporation MCP55 Ethernet (rev a3)
00:09.0 Ethernet controller: nVidia Corporation MCP55 Ethernet (rev a3)
00:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:05.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
18:00.0 VGA compatible controller: nVidia Corporation: Unknown device 019e (rev a2)
2b:00.0 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X Bridge (rev 06)
2b:00.1 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X Bridge (rev 06)
40:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
40:01.0 RAM memory: nVidia Corporation MCP55 LPC Bridge (rev a3)
40:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
40:01.2 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a3)
40:0a.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
40:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
40:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
45:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 21)
58:00.0 VGA compatible controller: nVidia Corporation: Unknown device 019e (rev a2)
6b:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 02)

[netllama]

The problem is here:
allocation failed: out of vmalloc space - use vmalloc=<size> to increase size.
NVRM: RmInitAdapter failed! (0x25:0xffffffff:1011)
NVRM: rm_init_adapter(1) failed

I'd suggest reading the driver README's discussion of this kernel bug.

[squeen]

Sorry to be so lost on this, but I searched through the HTML README
http://us.download.nvidia.com/XFree8...hapter-05.html
without finding the rm_init reference. Would you mind providing me more specific directions?

[squeen]

Oh. I think I found it:

Kernel virtual address space exhaustion on the X86 platform

On X86 systems and AMD64/EM64T systems using X86 kernels, only 4GB of virtual address space are available, which the Linux kernel typically partitions such that user processes are allocated 3GB, the kernel itself 1GB. Part of the kernel's share is used to create a direct mapping of system memory (RAM). Depending on how much system memory is installed, the kernel virtual address space remaining for other uses varies in size and may be as small as 128MB, if 1GB of system memory (or more) are installed. By default, the kernel reserves a minimum of 128MB.

The kernel virtual address space still available after the creation of the direct system memory mapping is used by both the kernel and by drivers to map I/O resources, and for some memory allocations. Depending on the number of consumers and their respective requirements, the Linux kernel's virtual address space may be exhausted. Newer Linux kernels print an error message of the form below when this happens:

allocation failed: out of vmalloc space - use vmalloc=<size> to increase size.

The NVIDIA kernel module requires portions of the kernel's virtual address space for each GPU and for certain memory allocations. If no more than 128MB are available to the kernel and device drivers at boot time, the NVIDIA kernel module may be unable to initialize all GPUs, or fail memory allocations. This is not usually a problem with only 1 or 2 GPUs, however depending on the number of other drivers and their usage patterns, it can be; it is likely to be a problem with 3 or more GPUs.

Possible solutions for this problem include:

[...]

If your system is equipped with an X86-64 (AMD64/EM64T) processor, it is recommended that you switch to a 64-bit Linux kernel/distribution. Due to the significantly larger address space provided by the X86-64 processors' addressing capabilities, X86-64 kernels will not run out of kernel virtual address space in the foreseeable future.

I was planning on going 64 bit some time soon for these machines. Might as well make it sooner rather than later.

[__entropy__]

http://us.download.nvidia.com/XFree8...ppendix-l.html

>> Kernel virtual address space exhaustion on the X86 platform

[squeen]

I reinstalled CentOS 5 x86_64 and now both GPUs are recognized. Thank you all very much for helping me solve this.

As a side note, on CentOS 5 (64bit), after the normal install process (from a clean filesystem), the boot up process hangs (not just the usual X server failure) when it should switch to the visual login screen. This happened every time I tried a fresh (re)install---three times in total with various options selected. Eventually, I booted to single user mode and installed the NVIDIA driver and then restarted. That cleared up the problem.

The only other oddies during the NV driver (script) install were:
1) the kernel-dev package was not installed by default, so I had to track it down
2) there was an error message (1 time) about one of the GL libraries not being a symbolic link. I ran it again, but the message never reappeared.

[squeen]

Update:

X desktop and apps are sluggish. I'm trying to pin point the problem.