Of course, as soon as I post my problem, the solution arrives before anyone can reply
Turns out is all I needed was to add
option "IgnoreDisplayDevices" "DFP, TV"
to my XF86config file and problem solved.
Thanks a ton to Bailout for posting his fix to the problem.