Multi-GPU Linux console unusable - Wrong display mode & wrong GPU

BinarySplit · 03-26-10, 10:44 PM

Some time during the boot process, after fsck but before X loads, my Linux shell console becomes unusable, though X runs without any problems. When I run in single-user mode or I press Ctrl+Alt+F1, my primary GPU's monitors turn off and my secondary GPU's monitor displays: "Mode Not Supported. H:16kHz V: 14.7Hz". If I reboot and swap this monitor with a newer monitor, I just get "Mode Not Supported" on the new monitor. All monitors are connected using DVI and I have not had problems with any OSs detecting the wrong display modes for any of them in the past. I have also confirmed that a virtual terminal exists at C-M-F1 as I can log in and run commands such as reboot, even though I can't see anything.

In addition to displaying the console using an unsupported mode, even if I physically disconnect all displays but GPU0's primary display I can't get the console or even the Mode Not Supported message to appear, indicating that the console ONLY displays on my secondary GPU, even if it has no monitors attached.

I'm running a 260 as GPU0 with 2 monitors attached and a 7600 as GPU1 with 1 monitor attached. I run Ubuntu 10.04 beta 1 with the 195.36.15 drivers.

This bug does not occur when I physically remove GPU1, however I would prefer not to do this as I use 3 screens and I frequently boot to windows. This bug also doesn't appear if I'm boot the Ubuntu LiveCD, which uses the Nouveau drivers.

How do I make the console appear using a working display mode on my first gpu/display? Being unable to access the console has made it very difficult to debug problems with X as I cannot restart X or edit xorg.conf without either removing a GPU or booting the LiveCD.

**AaronP** · 03-26-10, 11:49 PM

Something must be touching your GPUs during the boot process. The only suspicious thing I see in your log is a message from the fb console driver:

Quote:

[ 2.419725] vga16fb: initializing
[ 2.419728] vga16fb: mapped to 0xffff8800000a0000
[ 2.419867] fb0: VGA16 VGA frame buffer device
[...]
[ 3.286675] Console: switching to colour frame buffer device 80x30

Your log file does show the fbcon and vga16fb modules loaded. Does the problem go away if you blacklist those drivers?

BinarySplit · 03-27-10, 06:41 AM

Thanks for your quick response, AaronP.

I've (finally) blacklisted those drivers and am still having the same problems. I've attached an updated bug report.

For the record, blacklisting fbcon is a tedious process on the latest Ubuntu, you may want to link people to a guide in the future.
Here's what I had to do to get it to stop loading:

Add fbcon and vga16fb to the blacklist at /etc/modules/blacklist.conf (I also added dependent modules font, bitblit, tileblit, softcursor and vgastate just to be sure)
Ensure fbcon isn't listed for inclusion in /etc/initramfs-tools/modules
Change /etc/initramfs-tools/initramfs.conf to say MODULES=dep instead of MODULES=most
Run update-initramfs -u
Run update-lilo (I use lilo. I'm not sure if grub users need to update anything after updating their initrds though)
Finally confirm fbcon isn't running with lsmod | grep fbcon

**AaronP** · 03-27-10, 11:36 AM

Hmm, that's interesting. The fbcon driver is definitely gone, so I wonder if this is an interaction problem with the VGA arbiter. Could you please try this experiment? Move the nvidia.ko kernel module somewhere else and configure your system to boot to a console. If the problem doesn't reproduce then, please try running "insmod <whatever you renamed nvidia.ko to>" and see if the problem occurs then.

BinarySplit · 03-27-10, 07:17 PM

The system boots to console and is usable when the nvidiafb.ko and nvidia-current.ko files are missing. Replacing them and depmodding 'nvidia' froze the console and I had to reboot, however if I replaced them and then started X, X works properly but the C-M-F1 console is broken again.

I've attached the syslog for my second session, where I removed the .ko's, started as normal, killed GDM, replaced the .ko's in the console then restarted GDM. The syslog was up to this line right before I started GDM.

Code:

line 1173: Mar 28 12:48:30 KLOCKWORK kernel: [   20.996732] eth1: no IPv6 routers present

Ignore the pages of Gnome errors - my home dir was inaccessible at the time.
Also attached is the bug report from before the driver was loaded

**AaronP** · 03-28-10, 12:51 AM

Wait, are you saying that simply running depmod caused the problem? That shouldn't load the nvidia driver at all. Or did you mean modprobe or insmod?

I suspect that the kernel's built-in VGA console is not properly requesting VGA access from the arbiter, so when the nvidia driver loads and tells the arbiter that it doesn't need access, the kernel turns it off. That's speculation, though, so I'll need to talk to the kernel guys to figure out for sure what's going on.

You could try disabling the VGA arbiter, but you might need to recompile the kernel for that.

BinarySplit · 03-28-10, 05:02 AM

Ack, sorry. I meant modprobe.

Where is the VGA arbiter and how would I go about disabling it?

**AaronP** · 03-28-10, 07:28 PM

It's the VGA_ARB option in the kernel configuration, labeled "VGA Arbitration".

03-27-10, 11:36 AM	#4
AaronP NVIDIA Corporation Join Date: Mar 2005 Posts: 2,487	Re: Multi-GPU Linux console unusable - Wrong display mode & wrong GPU Hmm, that's interesting. The fbcon driver is definitely gone, so I wonder if this is an interaction problem with the VGA arbiter. Could you please try this experiment? Move the nvidia.ko kernel module somewhere else and configure your system to boot to a console. If the problem doesn't reproduce then, please try running "insmod <whatever you renamed nvidia.ko to>" and see if the problem occurs then.

03-28-10, 12:51 AM	#6
AaronP NVIDIA Corporation Join Date: Mar 2005 Posts: 2,487	Re: Multi-GPU Linux console unusable - Wrong display mode & wrong GPU Wait, are you saying that simply running depmod caused the problem? That shouldn't load the nvidia driver at all. Or did you mean modprobe or insmod? I suspect that the kernel's built-in VGA console is not properly requesting VGA access from the arbiter, so when the nvidia driver loads and tells the arbiter that it doesn't need access, the kernel turns it off. That's speculation, though, so I'll need to talk to the kernel guys to figure out for sure what's going on. You could try disabling the VGA arbiter, but you might need to recompile the kernel for that.

03-28-10, 05:02 AM	#7
BinarySplit Registered User Join Date: Mar 2010 Posts: 4	Re: Multi-GPU Linux console unusable - Wrong display mode & wrong GPU Ack, sorry. I meant modprobe. Where is the VGA arbiter and how would I go about disabling it?

03-28-10, 07:28 PM	#8
AaronP NVIDIA Corporation Join Date: Mar 2005 Posts: 2,487	Re: Multi-GPU Linux console unusable - Wrong display mode & wrong GPU It's the VGA_ARB option in the kernel configuration, labeled "VGA Arbitration".