Nvidia support for Xen

[Armedblade]

you could try what I posted:

"I load the uvesafb module ahead of doing a startx as this works around loss of video when switching to virtual terminal or after exiting Xorg."

[moenchmeyer]

Originally Posted by Armedblade

you could try what I posted:

"I load the uvesafb module ahead of doing a startx as this works around loss of video when switching to virtual terminal or after exiting Xorg."

Hi Armedblade,

thank you for your answer.
I had tried this already by starting Linux into runlevel 3 + modprobe uvesafb + startx (or init 5)

Unfortunately this did not help. Any other suggestions ?

Ralph

[uOpt]

I am running Xen-4.0 with a 2.6.32.25-debian (squeeze) Dom0 and I get this kernel panic when starting the NVidia drivers.

X.Org X Server 1.7.7
nVidia Corporation G71 [Quadro FX 2500M]
NVIDIA GLX Module 270.41.06

Message from syslogd@xenlap at Sat May 7 19:44:58 2011 ...
xenlap kernel: [ 63.850826] Bad pagetable: 000f [#1] SMP

Message from syslogd@xenlap at Sat May 7 19:44:58 2011 ...
xenlap kernel: [ 63.850836] last sysfs file: /sys/devices/platform/dock.0/docked
giving up.

Is this a new one?

The NVidia drivers compiled without problems against the kernel in question. At the time of the driver installation I had booted directly into the Dom0 kernel (no Xen hypervisor around it).

[uOpt]

Is anybody able to run with a 2.6.32 variant Dom0 kernel?

Would I maybe have more success splitting this off and run the X11 server with NVidia drivers in a DomU and PCI passthrough?

[moenchmeyer]

Originally Posted by Armedblade

you could try what I posted:

"I load the uvesafb module ahead of doing a startx as this works around loss of video when switching to virtual terminal or after exiting Xorg."

I checked the /var/log/messages to see whether there were any messages loading uvesafb. Actually, there were error messages

May 19 17:24:31 xen kernel: [ 33.953191] uvesafb: failed to execute /sbin/v86d
May 19 17:24:31 xen kernel: [ 33.953193] uvesafb: make sure that the v86d helper is installed and executable
May 19 17:24:31 xen kernel: [ 33.953196] uvesafb: Getting VBE info block failed (eax=0x4f00, err=-2)
May 19 17:24:31 xen kernel: [ 33.953198] uvesafb: vbe_init() failed with -22
May 19 17:24:31 xen kernel: [ 33.953203] uvesafb: probe of uvesafb.0 failed with error -22

Obviously, v86d is not available. And maybe this is the reason why loading the uvesafb module does not help to make the ttys 1 to 6 visible after starting X11 with the nvidia module and the Xen kernel.

Unfortunately, there is no v86d RPM for Opensuse 11.4, which I could install. Did anybody try to compile v86d for Opensuse 11.4 and the xen-kernel?

[moenchmeyer]

In my last two contributions to this thread I described problems with the nvidia driver and Xen in an Opensuse 11.4 environment. To get the nvidia driver (version 270.41.06) running with kernel 2.6.37.6-0.5-xen (x86_64) was no problem. However, the terminals tty1 to tty6 were no longer usable after the start of the X server as all the ttys besides tty7 appeared completely black.

The good news is, that, eventually, I got Opensuse 11.4 and it's XEN kernel running together with the latest nvidia driver in Dom0. I now have a graphical KDE desktop with 3D acceleration on tty7 - without running into the previous trouble with getting blank, black ttys (tty1 to tty6, tty10) after the start of the X server. A switch between the graphical desktop on tty7 and the other ttys by using CTRL ALT Fn or "chvt n", is possible, now, without any problems - and I can configure and use any physically available uvesafb mode on the ttys.

The hint of Armedblade to use the "uvesafb" module before starting X was quite right and the key to the solution. Many thanks to Armedblade !

But one has to get the "v86d" helper daemon running, too. And for me, who had never used "uvesafb" before, it was a bit of experimenting. As a complete overview over the steps necessary may be helpful for others, I'll describe briefly what steps are to be performed to set up an Opensuse 11.4 XEN system with the nvidia driver in the Dom0:

Step 1 - Installation of Opensuse with the Xen packages:
Install Opensuse 11.4 regularly, but check the option "hostserver for Xen Virtual Machine" when configuring the SW packages just before starting the installation. (Otherwise you must install the virtualization packages and the Xen kernel packages afterwards.) In addition install the kernel sources and compilers to be able to compile kernel modules later on.

Step 2 - after a boot of the default kernel (not the Xen kernel):
Get the latest nvidia driver (in my case 270.41.06). Try to install it in runlevel 3 by using the installation routine (sh NVIDIA-Linux-x86_64-270.41.06.run). The installation probably complains about the already running Nouveau driver. Answer positively to the suggestion of the nvidia setup program to install a file for the deactivation of the Nouveau driver.

Step 3 - after a boot of the default kernel:
Deactivate KMS, e.g. by using Yast and the editor for /etc/sysconfig: Set the option NO_KMS_INITRD under "system > kernel" to "yes". (Alternatively, you can edit the file "/etc/sysconfig/kernel", accordingly.)
Reboot into level 3 - but use the kernel option "nomodeset" in the grub dialog before booting - just to be on the safe side regarding KMS deactivation. Install the nvidia driver again and check that it is working correctly for the normal Opensuse 11.4 kernel. You can deinstall the Nouveau driver packages, now, if you want to.

Step 4 - boot the XEN kernel and install the nvidia driver in the XEN Dom0:
Reboot the system. In the Grub menu, switch to the option for starting XEN. Then boot the XEN kernel into runlevel 3. Login and install (and thereby recompile) the Nvidia driver module for the XEN kernel by performing

cd Your_Directory_With_THE_Nvidia_Driver
export IGNORE_XEN_PRESENCE=1 SYSSRC=/usr/src/linux-2.6.37.6-0.5 SYSOUT=/usr/src/linux-2.6.37.6-0.5-obj/x86_64/xen

sh ./NVIDIA-Linux-x86_64-270.41.06.run

See previous contributions in this thread for the meaning of the environment variables IGNORE_XEN_PRESENCE, SYSSRC and SYSOUT. The kernel versions in the above statements of course depend on what you actually have installed on your system. (The above is valid after a regular Opensuse 11.4 installation without any kernel updates.)

The compilation and installation of the nvidia driver module should work without any problems.

Step 5 - in the XEN environment (Dom0): Start the X server
Now, change to runlevel 5 (init 5). The X server should start successfully on tty7. Unfortunately, a CTRL ALT Fx or a "chvt x" afterwards leads to a black terminal ttyx. I.e., the tty is invisible - although, actually, it still is active and processes commands from the keyboard. Therefore, you can get back to your graphical desktop environment at any time by pressing CTRL ALT F7.

Step 6 - in the XEN Dom0: Verify that you have an uvesafb.ko module
Back again in in your graphical desktop on tty7, verify that the uvesafb.ko module is present in
/lib/modules/2.6.37.6-0.5-xen/kernel/drivers/video.

Opensuse's XEN kernel was compiled and installed together with this module. So, you should have "uvesafb" present on your Opensuse 11.4 system. If not, you must recompile the Xen kernel with this module and install it.

Step 7 - in the XEN Dom0: Install v86d
Now, get the sources for "v86d" (v86d0.1.10.tar.bz2) from here:
http://dev.gentoo.org/~spock/projects/uvesafb/

Unfortunately, there is no RPM for the helper daemon "v86d", which you could use for Opensuse 11.4 - at least I did not find any. However, you need a working "v86d", otherwise "uvesafb" will not work !

Therefore, extract the downloaded source archive as root and go through the standard steps "./configure" and "make". With "./configure" use the option "--with-x86emu". The compilation and linking should work without problems. Copy the resulting executable file "v86d" into the directory "/sbin".

Step 8 - in the XEN Dom0: uvesafb modes
Find out about the codes for available uvesafb (framebuffer) modes by entering the command:

cat /sys/bus/platform/drivers/uvesafb/uvesafb.0/vbe_modes

Step 9 - in the XEN Dom0: Create an uvesafb.conf file
As root create a configuration file "uvesafb.conf" in the directory "/etc/modprobe.d/". Set your preferred uvesafb mode (e.g. 1280x800-32), which you later on want to use for the ttys, by inserting the following line into the file :

options uvesafb mode_option=1280x800-32 scroll=ywrap


Step 10 - in the XEN Dom0: Modify your grub configuration and eliminate vesa vga parameters
Now, change your Grub boot loader configuration: Remove any "vga" parameters as
"vga=mode-0xnnn" or "vga=0xnnn" for the boot entries of the Xen kernel. "nnn" here represents your chosen standard vesa mode for your ttys. (You can use Yast for modifying the boot loader configuration or edit grub's boot loader files directly).

This step is an important and necessary step ! The standard vesa modes would later interfere with the "uvesafb" framebuffer modes. The standard vesa setup is not compatible with "uvesafb"!

Step 11 - Reboot Xen and load the uvesafb module:
Reboot the Xen kernel into runlevel 3. Of course, now, your boot messages on tty1 appear in a coarse 80x25 resolution as you have removed the vesa vga option in your boot configuration. Do not worry about that. You will use your uvesafb mode instead in a minute! At runlevel 3 login as root.

Now, try "modprobe uvesafb" and watch the changes. Check the last lines in "/var/log/messages" to verify that the uvesafb module was loaded without any errors. "uvesafb" should have used "v86d", recognized the screen and set your preferred terminal mode from the "uvesafb.conf" file. I.e., the terminal tty1 and all other ttys (2-6 and 10) should now appear in your chosen uvesafb mode - e.g. the mode 1280x800-32, if you chose that one.

Step 12 - enjoy your graphical desktop with 3D acceleration and switch to visible ttys (tty1 to tty6, tty10) whenever you like :
Use startx or go to runlevel 5 to start your X server. Verify, that a CTRL ALT Fx now leads to a visible tty in your chosen uvesafb mode. And be happy with the 3D acceleration of your nvidia card in your graphical desktop environment - whatever you need it for in a XEN Dom0!

Remaining steps - automize your boot up procedure
Modify your boot up scripts such that "uvesafb" is loaded before the X server is started.


Useful links:
https://wiki.archlinux.org/index.php/Uvesafb
http://dev.gentoo.org/~spock/projects/uvesafb/

[uOpt]

So that means the SMP panic I see in 2.6.32 might simply be fixed in 2.6.37?

I dread the thought of having to go through the ordeal of going to a 2.6.37 or 2.6.38 Xen enabled Debian Not that they have that in the first place.

Might have to go down to 2.6.26? Meh.

[moenchmeyer]

Originally Posted by uOpt

So that means the SMP panic I see in 2.6.32 might simply be fixed in 2.6.37?

I dread the thought of having to go through the ordeal of going to a 2.6.37 or 2.6.38 Xen enabled Debian Not that they have that in the first place.

Might have to go down to 2.6.26? Meh.

I simply do not know. I just tried everything with Opensuse 11.4 on two systems with different hardware. Both systems have quad core processors Q9550 and an i7 950 and as said I use kernel version 2.6.37.6 (x86_64). I tried with 2 different Nvidia cards - a 9800 GTX+ and a GTX 460 - and it works with both of them. With and without virtual machines running.

What happens if you do not use the Xen kernel but a standard 2.6.32 kernel ?

[uOpt]

Originally Posted by moenchmeyer

I simply do not know. I just tried everything with Opensuse 11.4 on two systems with different hardware. Both systems have quad core processors Q9550 and an i7 950 and as said I use kernel version 2.6.37.6 (x86_64). I tried with 2 different Nvidia cards - a 9800 GTX+ and a GTX 460 - and it works with both of them. With and without virtual machines running.

What happens if you do not use the Xen kernel but a standard 2.6.32 kernel ?

I can boot the Dom0 capable 2.6.32 kernel inside and outside Xen. It runs the current NVidia drivers just fine if Xen isn't around it. Boot the same kernel inside Xen, get the SMP panic I posted when starting X11.

Somebody wrote somewhere that 2.6.32 is probably a lost cause for the NVidia drivers because of VM changes. I wonder whether I might be further with the reversed engineered drivers. But the point about this laptop is that it runs decent 3D *and* Xen. Kinda disappointing.

[uOpt]

Xen made it into Linux-3.0.

Based on the above it seems that there is a chance that kernels > 2.6.32 run the NVidia drivers without the lock panic I had.

I might see light at the end of the tunnel here.

[gilboa]

Originally Posted by uOpt

I might see light at the end of the tunnel here.

Which hopefully in this case, isn't the train... *

- Gilboa
* Read having nVidia break badly on 3.0 or above...

[uOpt]

Originally Posted by gilboa

Which hopefully in this case, isn't the train... *

- Gilboa
* Read having nVidia break badly on 3.0 or above...

LA LA LA I can't hear you

BTW, is there any problem running the NVidia drivers in an OS that has (not is) KVM guests?