I don't know all the tech behind it and it seems you do.
All I do know is HDR + AA in "Oblivion" was broken with 163.11 driver for me until I installed and tried nHancer with the setting I noted. It even remained broken when I tried putting back 162.22 for whatever the reason.
I didn't do anything exotic when installing 163.11. I just removed the old driver in safe mode then installed 163.11.
HDR + AA worked fine with the 162.22 driver before trying 163.11.
Let me illustrate. Below is "Oblivion" with 8xAA set from the nVidia control panel as override application with 163.11, and this is also how 8xAA appears set from nHancer with AA compatibility unticked.
Note the top two horizontal boards.
I did discover ticking AA Compatibility does work with the "Rainbow 6 Vegas" setting or "Oblivion (AA&HDR)". Below is with anti aliasing compatibility ticked under nHancer's compatibility tab. Apparently the fix is ticking anti aliasing compatibility under the compatibility tab with nHancer.
Forcing AA + HDR from the nVidia CP with 163.11 did not work for me, and Nhancer will not work either without anti aliasing compatibility ticked under the compatibility tab.
In any case, it's all good now.