(--) NVIDIA(0): Connected display device(s) on GeForce FX 5500 at PCI:2:0:0:
(--) NVIDIA(0): DFP-0
(--) NVIDIA(0): DFP-0: 135.0 MHz maximum pixel clock
(--) NVIDIA(0): DFP-0: Internal Single Link TMDS
There's your problem - a single link TMDS cannot cope with pumping out too many pixels per line. If you want to use higher resolutions, you'll have to use a VGA connection.
If you do decide to get a different card, make sure it has dual link TMDS. This is something that the board manufacturers can decide to reduce costs on. I would suggest avoiding any card that uses TurboCache - this uses your system memory to supplement the memory on the card (again, to reduce costs).