View Single Post
Old 05-21-03, 04:39 PM   #1
grmoc
.. Arg!!. Uh Pinky?
 
Join Date: Jan 2003
Posts: 38
Default fragment program precision loss?

This bit of code uses a pixel shader to do a lookup into a 3d texture based on the color at that pixel fo the 2d texture (i.e. a function based on the color of the 2d texture)

char chromaFragmentProgram[]="" // this is on purpose.
"!!ARBfp1.0\n"
"TEMP lookupVal;\n"
"TEMP tmpVal;\n"
"TEX lookupVal,fragment.texcoord[0],texture[0],2D;\n"
"TEX result.color,lookupVal,texture[1],3D;\n"
"END\n";


Assume that the 3d texture is bound with S_WRAP (etc) as GL_REPEAT (i.e. the default).

Now, initialize the 3d texture as an identity function..
i.e.

// this should be bound on texture unit 1
unsigned char cubeTex[256][256][256][4];
for(int i=0;i<256;++i){
for(int j=0;j<256;++j){
for(int k=0;k<256;++k){
cubeTex[i][j][k][0]=k;
cubeTex[i][j][k][1]=j;
cubeTex[i][j][k][2]=i;
cubeTex[i][j][k][3]=255;
}
}
}

Now, take a 2d texture that 'saturates' at least one of the color channels, and bind that as the texture on texture unit 0.

When you draw this 2d texture across the screen with the fragment program enabled, you'd expect to see the 2d texture exactly (without change), however, where the 2d texture had a pixel with an channel saturated at 255, you'll see a wrap-around w.r.t. color, and the pixel will appear black instead.

This implies there is a precision loss somewhere in the fragment program! (An annoying one)

Can you confirm this for me, or suggest a workaround?

a sample program is attached.
... it won't compile, since it is missing a 'texture' library, but you should be able to easily adapt it.
Attached Files
File Type: txt sample.txt (11.1 KB, 145 views)
grmoc is offline   Reply With Quote