View Single Post
Old 04-28-10, 01:15 AM   #97
Armedblade
Registered User
 
Join Date: Oct 2007
Posts: 23
Default Re: Nvidia support for Xen

These are my latest notes on how to use the NVIDIA binary driver (latest version recently released has not been tested) with a Xenlinux (non-pvops) kernel. It is merely a work-around for a "NVIDIA: unknown symbol init_mm" error which occurs as soon as the module tries to load. Ssee NOTE [1] for my specific kernel build details.

Note: This hack is experimental and I have been using it for just over 1 week now. Hopefully you will provide feedback on your own results or perhaps you know of a better solution.

Parts of these instructions are borrowed from the many guides that have come before mine. I have chosen to detail some of the areas where I ran into trouble. Please note, I do not run a desktop manager like [k|g]dm and I suggest that you disable it for the time being

Boot a non-xen kernel and install the NVIDIA driver. Ensure xorg is working properly. This step is to get common userland libraries populated that would be used regardless of the kernel version.

Boot the xen kernel and assuming /lib/modules/$(uname -r)/source does indeed point to your xen kernels sources:

$ sh NVIDIA-Linux-x86_64-195.36.15-pkg2.run --extract-only
Creating directory NVIDIA-Linux-x86_64-195.36.15-pkg2
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 195.36.15......................................... .................................................. .................................................. .......................

$ cd NVIDIA-Linux-x86_64-195.36.15-pkg2/usr/src/nv/
$ patch -p0 < ~/nv-hack.txt
patching file nv-linux.h

$ export IGNORE_XEN_PRESENCE=1
$ export SYSSRC=/lib/modules/$(uname -r)/source
$ export SYSOUT=/lib/modules/$(uname -r)/build
$ make clean module
<a lot of noise here>
NVIDIA: left KBUILD

$ sudo cp nvidia.ko /lib/modules/$(uname -r)/kernel/drivers/video/
$ sudo depmod -a
$ sudo modprobe nvidia

nv-hack.txt contents:
Code:
--- nv-linux.h.orig	2010-04-27 19:52:12.000000000 -0400
+++ nv-linux.h	2010-04-27 19:52:21.000000000 -0400
@@ -1013,7 +1013,7 @@
         if (!kernel)                                    \
             __pgd = pgd_offset(__mm, address);          \
         else                                            \
-            __pgd = pgd_offset_k(address);              \
+            __pgd = NULL;                               \
         __pgd;                                          \
     })
Results: I have tested mythfrontend for several hours which has ran fine while connected to a pv domU mythbackend on the same box. I have also ran games known to work in wine. I load the uvesafb module ahead of doing a startx as this works around loss of video when switching to virtual terminal or after exiting Xorg. Something that can happen regardless if xen is involved. As far as the kernel, there are a few udev burps in dmesg that I need to investigate but it has been plenty stable.



NOTE [1]
I am using 2.6.31.12 kernel sources + xen patches (http://gentoo-xen-kernel.googlecode.....31-14.tar.bz2)

$ cd && wget http://www.kernel.org/pub/linux/kern...6.31.12.tar.gz
$ wget http://gentoo-xen-kernel.googlecode.....31-14.tar.bz2
$ mkdir xen-patches && cd xen-patches && tar xjf ~/xen-patches-2.6.31-14.tar.bz2
$ cd /usr/src && tar xzf linux-2.6.31.12.tar.gz

Example patch method (this is borrowed from a guy named Boris aka dbaxps):
Code:
#! /bin/bash

cd /usr/src/linux-2.6.31.12 || exit

for P in `ls ~/xen-patches/*.patch1 | sort`
do
patch -p1 -s -i $P
if [ $? = 0 ]; then
echo $P applied
else
echo "Error processing "$P
exit 1
fi
done
Note: I couldn't get debian's make-kpkg to build this one properly so use the old school solution for now. UPDATE see NOTE [2]

$ cd /usr/src/linux-2.6.31.12
$ cp /boot/config-<some old version> ./
$ make oldconfig
$ make menuconfig #<insert link to guide for options to enable for xen here>
$ make -j4
$ sudo make modules_install install
$ sudo update-initramfs -c -k 2.6.31.12

I'm running grub2 which at this point I'll update /etc/grub/40_custom to add in this kernel to boot under xen
Code:
menuentry "Xen 4.0 / Ubuntu 10.4 kernel 2.6.31.12 sxen" {
insmod ext2
set root='(hd0,2)'
search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
multiboot /boot/xen-4.0.gz dummy=dummy iommu=pv dom0_mem=1024M
module /boot/vmlinuz-2.6.31.12 dummy=dummy root=/dev/sdc2 ro
module /boot/initrd.img-2.6.31.12
}
$ sudo update-grub



NOTE [2]
Problem: debian Kernel package build of xenified kernel fails when attempting to create bzImage (it will try to create bzImage regardless of passing --zimage to make-kpkg)

hack for arch/x86/Makefile (2.6.31.12 + andrew lyon's rebased suse patches already applied as described previously)

contents of makefilehack.txt
Code:
--- arch/x86/Makefile.orig	2010-05-02 12:24:38.000000000 -0400
+++ arch/x86/Makefile	2010-05-02 12:27:01.000000000 -0400
@@ -155,18 +155,19 @@
 LDFLAGS_vmlinux := -e startup_64
 endif
 
-# Default kernel to build
-all: vmlinuz
+endif
+## Default kernel to build
+#all: vmlinuz
 
-# KBUILD_IMAGE specifies the target image being built
-KBUILD_IMAGE := $(boot)/vmlinuz
+## KBUILD_IMAGE specifies the target image being built
+#KBUILD_IMAGE := $(boot)/vmlinuz
 
-vmlinuz: vmlinux
-	$(Q)$(MAKE) $(build)=$(boot) $(KBUILD_IMAGE)
-	$(Q)mkdir -p $(objtree)/arch/$(UTS_MACHINE)/boot
-	$(Q)ln -fsn ../../x86/boot/$@ $(objtree)/arch/$(UTS_MACHINE)/boot/$@
-else
-# Default kernel to build
+#vmlinuz: vmlinux
+#	$(Q)$(MAKE) $(build)=$(boot) $(KBUILD_IMAGE)
+#	$(Q)mkdir -p $(objtree)/arch/$(UTS_MACHINE)/boot
+#	$(Q)ln -fsn ../../x86/boot/$@ $(objtree)/arch/$(UTS_MACHINE)/boot/$@
+#else
+## Default kernel to build
 all: bzImage
 
 # KBUILD_IMAGE specify target image being built
@@ -176,7 +177,7 @@
 	$(Q)$(MAKE) $(build)=$(boot) $(KBUILD_IMAGE)
 	$(Q)mkdir -p $(objtree)/arch/$(UTS_MACHINE)/boot
 	$(Q)ln -fsn ../../x86/boot/bzImage $(objtree)/arch/$(UTS_MACHINE)/boot/$@
-endif
+#endif
 
 $(BOOT_TARGETS): vmlinux
 	$(Q)$(MAKE) $(build)=$(boot) $@
I changed config option CONFIG_XEN_COMPAT_030200_AND_LATER=y (bitmonk on #xen tells me that bzImage boot may only
work on xen 3.3 or something later ). This gets by the build failure for now.

The failure that occurs when CONFIG_XEN_COMPAT_030002_AND_LATER is set (its including linux/string.h):
arch/x86/boot/compressed/misc.c:135: error: conflicting types for ‘memcpy’
/usr/src/linux-2.6.31.12/arch/x86/include/mach-xen/asm/hypercall.h:289: note: previous implicit declaration of ‘memcpy’ was here
make[2]: *** [arch/x86/boot/compressed/misc.o] Error 1
make[1]: *** [arch/x86/boot/compressed/vmlinux] Error 2
make: *** [bzImage] Error 2


the build process would be something like:

$ cd /usr/src/linux-2.6.31.12
$ patch -p0 < ~/makefilehack.txt
$ cp /boot/config-<some old version> ./ #<need good default config>
$ make oldconfig
$ make menuconfig # enable xen options and set compat option higher
$ fakeroot make-kpkg clean
$ export CONCURRENY_LEVEL=4
$ fakeroot make-kpkg --append-to-version=-sxen1 kernel_image
$ cd ../
$ sudo dpkg -i linux-image-2.6.31.12-sxen1_2.6.31.12-sxen1-10.00.Custom_amd64.deb
$ sudo update-initramfs -c -k 2.6.31.12-sxen1

<update /etc/grub.d/40_custom as noted before>
$ sudo update-grub

Last edited by Armedblade; 05-06-10 at 10:23 PM.
Armedblade is offline   Reply With Quote