yes, its possible...
you would think, there should 1st be graphic drivers with which then you build windowing system like X, but no... instead, you have X that is requisite for using OpenGL and it rather stands on the way...
i believe this is what you looking for:
myOS - Linux based OpenGL development system without X
Minimalistic GNU/Linux system, stripped down of everything, but core necessary files to compile and run OpenGL/C code. It has simplified directory structure and cleaned up internal cross referencing. Programs compiled in myOS will run on any Linux with or without X.
Copy of the OpenGL Red Book is included, together with 45 examples from the book. Examples are organized by chapters, each in separate folder, ready to compile, run and play with. Source code is modified to use Scitech drivers and is very similar to GLUT version in the Red Book. This little collection is really all one needs to learn OpenGL. In fact the whole system being very simple to use or analyze and understand, can be a tool to teach Linux basics and C/OpenGL programming.
>"..write an app that links against nvidia's openGL and..."
"unfortunately" i use scitech drivers
(no full hw support, but still pretty good considering all the supported cards and setup free installation)
not nvida drivers,
nvida only make drivers for X, even worse thats limited to particular distros, specific configurations...
i dont see reason, dont understand, or for that matter, im completly confused why they dont make drivers for low-level Linux, like scitech??!, so we could build windowing system on top of OpenGL that will surely be faster, smaller and therefore better "new X" thingy, with all that Vista, Beryl, Compiz - 3d cube, trasparency and rubbery windows stuff being easy to program and come natural to windowing API
when/if they do decide to "transfer" they drivers to lower-level and free them of X dependency, its good to know we will have complete development system built with it in only 12Mb (or less)
other "OpenGL without X" drivers/projects:
* fbdev/DRI drivers from the mesa3d project
* Scitech SNAP/MGL
i found scitech's drivers only working solution for "without X",
the rest is to big, too many dependencies, discontinued, hard to compile...
they are all based on Mesa (until nvidia helps)