Carmack's .plan which you link to is not talking about dropping the nv30 path he did that much later. in your link he is talking about dropping vender specific vertex programs since they weren't needed with the new arb extension.
as for the 9800pro being as fast as the 5900u, that is because the 5900u does get the benefit of using reduced precision in the arb2 path like it once needed the nv30 path to do. look back at that link you posted to see how things once were with the arb2 path:
The NV30 runs the ARB2 path MUCH slower than the NV30 path.
Half the speed at the moment. This is unfortunate, because when you do an
exact, apples-to-apples comparison using exactly the same API, the R300 looks
twice as fast, but when you use the vendor-specific paths, the NV30 wins.
but adding partial precision to the arb2 path doesn't make it vender specific, it just lets cards which can benefit from reduced precision do so. doom3 doesn't use any advanced shaders which really require full precsision, so there is hardly reason not to let cards which can benefit from it do so.