That xorg.conf file is something you did as I have made no changes to nvidia-xorg.conf in the 304.32 RPM Fusion driver.
[root@main-pc leigh]# rpm -qf /etc/X11/xorg.conf
file /etc/X11/xorg.conf is not owned by any package
[root@main-pc leigh]# rpm -qf /etc/X11/nvidia-xorg.conf
[root@main-pc leigh]# cat /etc/X11/nvidia-xorg.conf
# RPM Fusion - nvidia-xorg.conf
On initial of the xorg-x11-drv-nvidia package it copies /etc/X11/nvidia-xorg.conf to /etc/X11/xorg.conf
Also FYI yum doesn't change user edited config files in /etc !
Read the manual as there have many changes in 304.xx