[solved] Clipping planes of OpenGL viewport for custom SSAO

[hg82]

Hallo everyone,

I'm currently working on a custom SSAO shader which is based around the shader by Martins Upitits http://devlog-martinsh.blogspot.de/2011/10/nicer-ssao.html
I managed to get it working but I think I got the values for the near and far clipping planes wrong.

I had a look at GZDoom's source but the only values I could find are 5.0 for the near clipping plane and 65536.0 for the far clipping plane.
These values are from the viewport of the software renderer.

But actually I think they are not quite right - at least for OpenGL.

Does anyone know where I could look up the values for the OpenGL viewport - or even better - just tell me ?

Thanks in advance.

Cheers
HG

[dpJudas]

Since you're talking about a SSAO shader that accesses depth values I'm going to assume you are implementing this on the C++ side. The values are 5.0 and 65536.0 for the GL renderer as well.

Where are you getting the depth values from? The built-in SSAO pass first creates a linearized depth buffer. You can find the code for the current SSAO pass in FGLRenderer::AmbientOccludeScene (gl_postprocess.cpp). That function calls GetZNear and GetZFar, which returns 5 and 65536.

[hg82]

Thank you very much for that info.
For now I am simply replacing the shader itself in the gzdoom.pk3 to make sure the shader is working as expected.
I'll see about anything else when the time is right.
I'm working on a standalone game and ideally I don't have to mess with the engine itself. But I'll see how this will work out. As much as I like the built-in SSAO it doesn't seem to work correctly on 3d models. That's why I decided to try to replace the shader.
A more subtle bloom, custom tonemapping, vignette and noise are already in place.
So I guess I'll have to look a little deeper into the surroundings of the SSAO code.

[dpJudas]

I haven't looked at the other SSAO shader, but perhaps it expects the real zbuffer as input instead of a linearized version.

If you are having trouble with the normal SSAO shader, make sure your models have correct valid normals. Especially md3 files from older mods tends to have md3s where the normals are broken. Earlier versions of GZDoom didn't use the normals for anything, but the attenuated lights and the SSAO pass does.

If you are absolutely certain your models have valid normals, could you be more specific in which way the current SSAO pass is not working with them? The SSAO pass you're trying to switch to is a much older SSAO algorithm that overall should produce a worse visual result than the algorithm implemented by GZDoom.

[hg82]

The Problem with the built-in SSAO pass is that 3d models don't occlude themselves. So the models occlude their environment absolutely fine but there's no occlusion on the models. I monitored that through setting CVAR gl_ssao_debug 1.
That other algorithm actually works on the 3d models. I'm pretty sure that the normals are ok... NOW
I had a lot of trouble with the exporter I originally used. It completely messed up the normals of my models. So I switched to a pretty recent md3 exporter and normals are fine now. (double checked that by letting the SSAO pass only output the normal texture)

[dpJudas]

The built-in SSAO pass doesn't have any special handling of models, but it has some special handling of null normals. The portal entrances in GZDoom generates walls that are marked using null normals and the SSAO pass ignores such pixels to make it less visible where portals are.

If a model has null normals that will cause it to not occlude itself, and if a model has normals pointing in totally wrong directions it will cause it to overly self-occlude. Your description seem to match the case where the model normals are null.

That other algorithm works on the 3d models because it doesn't take normal vectors into account at all. That avoids the problem with broken normals, but overall it creates SSAO with more self-shadowing (google for images comparing SSAO to HBAO to see the visual difference - gzdoom is a HBAO variant).

[hg82]

Hmmm... That explains a lot and now things make sense...
I will look into that...

[dpJudas]

By the way, if you "fixed" this by plugging in your own SSAO shader I'd like to get a copy of that version - assuming it uses a license compatible with GZDoom. The reason being that the paper I used for the current AO pass said it was faster than the original SSAO algorithm, but if I could confirm that for myself that would be ideal. There are some users of GZDoom that would like to use SSAO but can't due to it being too expensive and if this other version turns out to be faster it might be worth including it.

[hg82]

So it turned out that the must recent version of the exporter writes null normals. Got an earlier version and now everything's in order.
Just needed some input from someone else - thanks again...

Here is the code. It doesn't have any particular license and I couldn't find any info on the homepage...

Code: Select all • Expand view

/*
SSAO GLSL shader v1.2
assembled by Martins Upitis (martinsh) (devlog-martinsh.blogspot.com)
original technique is made by Arkano22 (www.gamedev.net/topic/550699-ssao-no-halo-artifacts/)

changelog:
1.2 - added fog calculation to mask AO. Minor fixes.
1.1 - added spiral sampling method from here:
(http://www.cgafaq.info/wiki/Evenly_distributed_points_on_sphere)
*/

in vec2 TexCoord;
out vec4 FragColor;
/*
uniform vec2 UVToViewA;
uniform vec2 UVToViewB;
uniform vec2 InvFullResolution;

uniform float NDotVBias;
uniform float NegInvR2;
uniform float RadiusToScreen;
uniform float AOMultiplier;

uniform sampler2D NormalTexture;
uniform sampler2D RandomTexture;

uniform vec2 Scale;
uniform vec2 Offset;


*/

uniform sampler2D DepthTexture;

//uniform sampler2D bgl_RenderedTexture;
//uniform float bgl_RenderedTextureWidth;
//uniform float bgl_RenderedTextureHeight;

float width = textureSize(DepthTexture,0).x; //bgl_RenderedTextureWidth; //texture width
float height = textureSize(DepthTexture,0).y; //bgl_RenderedTextureHeight; //texture height

#define PI    3.14159265

//------------------------------------------
//general stuff

//make sure that these two values are the same for your camera, otherwise distances will be wrong.

float znear = 5.0; //0.5; //Z-near
float zfar = 65536.0; //100.0; //Z-far

//user variables
int samples = 32; //ao sample count

float radius = 6.0; //ao radius *2.0
float aoclamp = 0.2; //depth clamp - reduces haloing at screen edges *0.1
bool noise = true; //use noise instead of pattern for sample dithering
float noiseamount = 0.0002; //dithering amount *0.0002

float diffarea = 0.4; //self-shadowing reduction*0.4
float gdisplace = 0.4; //gauss bell center*0.4
float aowidth = 8.0; //gauss bell width*8.0

bool mist = false; //use mist?
float miststart = 0.0; //mist start
float mistend = 20.0; //mist end *20.0

bool onlyAO = true; //use only ambient occlusion pass?
float lumInfluence = 0.0; //how much luminance affects occlusion

//--------------------------------------------------------

vec2 rand(vec2 coord) //generating noise/pattern texture for dithering
{
   float noiseX = ((fract(1.0-coord.s*(width/2.0))*0.25)+(fract(coord.t*(height/2.0))*0.75))*2.0-1.0;
   float noiseY = ((fract(1.0-coord.s*(width/2.0))*0.75)+(fract(coord.t*(height/2.0))*0.25))*2.0-1.0;
   
   if (noise)
   {
      noiseX = clamp(fract(sin(dot(coord ,vec2(12.9898,78.233))) * 43758.5453),0.0,1.0)*2.0-1.0;
      noiseY = clamp(fract(sin(dot(coord ,vec2(12.9898,78.233)*2.0)) * 43758.5453),0.0,1.0)*2.0-1.0;
   }
   return vec2(noiseX,noiseY)*noiseamount;
}

float doMist()
{
   float zdepth = normalize(texture2D(DepthTexture,TexCoord)).x;
   float depth = -zfar * znear / (zdepth * (zfar - znear) - zfar);
   return clamp((depth-miststart)/mistend,0.0,1.0);
}

float readDepth(in vec2 coord)
{
   coord.x = clamp(coord.x,0.001,0.999);
    coord.y = clamp(coord.y,0.001,0.999);
   float depth = normalize(texture2D(DepthTexture, coord)).x;
   return (2.0 * znear) / (zfar + znear - depth * (zfar-znear));
}

float compareDepths(in float depth1, in float depth2, inout int far)
{   
   float garea = aowidth; //gauss bell width   
   float diff = (depth1 - depth2)*100.0; //depth difference (0-100)
   //reduce left bell width to avoid self-shadowing
   if (diff<gdisplace)
   {
      garea = diffarea;
   }
   else
   {
      far = 1;
   }
   
   float gauss = pow(2.7182,-2.0*(diff-gdisplace)*(diff-gdisplace)/(garea*garea));
   return gauss;
}   

float calAO(float depth, float dw, float dh)
{   
   //float dd = (1.0-depth)*radius;
   float dd = radius;
   float temp = 0.0;
   float temp2 = 0.0;
   float coordw = TexCoord.x + dw*dd;
   float coordh = TexCoord.y + dh*dd;
   float coordw2 = TexCoord.x - dw*dd;
   float coordh2 = TexCoord.y - dh*dd;
   
   vec2 coord = vec2(coordw , coordh);
   vec2 coord2 = vec2(coordw2, coordh2);
   
   int far = 0;
   temp = compareDepths(depth, readDepth(coord),far);
   //DEPTH EXTRAPOLATION:
   if (far > 0)
   {
      temp2 = compareDepths(readDepth(coord2),depth,far);
      temp += (1.0-temp)*temp2;
   }
   
   return temp;
}

void main(void)
{
   vec2 noise = rand(TexCoord);
   float depth = readDepth(TexCoord);
   
   float w = (1.0 / width) / clamp(depth, aoclamp, 1.0)+(noise.x * (1.0 - noise.x));
   float h = (1.0 / height) / clamp(depth, aoclamp, 1.0)+(noise.y * (1.0 - noise.y));
   
   float pw;
   float ph;
   
   float ao;
   
   float dl = PI*(3.0-sqrt(5.0));
   float dz = 1.0/float(samples);
   float l = 0.0;
   float z = 1.0 - dz/2.0;
   
   for (int i = 0; i <= samples; i ++)
   {     
      float r = sqrt(1.0-z);
      
      pw = cos(l)*r;
      ph = sin(l)*r;
      ao += calAO(depth,pw*w,ph*h);       
      z -= dz;
      l += dl;
   }
   
   ao /= float(samples);
   ao = 1.0-ao;   
   
   if (mist)
   {
      ao = mix(ao, 1.0,doMist());
   }

   vec3 color = vec3(1.0 - normalize(texture2D(DepthTexture,TexCoord).r)); //don't know why I did that...
   
   vec3 lumcoeff = vec3(0.299,0.587,0.114);
   float lum = dot(color.rgb, lumcoeff);
   vec3 luminance = vec3(lum, lum, lum);
   
   vec3 final = vec3(color*mix(vec3(ao),vec3(1.0),luminance*lumInfluence));//mix(color*ao, white, luminance)
   
   if (onlyAO)
   {
      final = vec3(mix(vec3(ao),vec3(1.0),luminance*lumInfluence)); //ambient occlusion only
   }
   
   FragColor = vec4(final, 1.0);
}


I monitored the shader times and the SSAO above is perhaps 0.5ms slower - depending on the number of samples...

[Nash]

I'm curious to see picture comparisons between GZDoom's SSAO and this other SSAO.

[hg82]

The one I posted above isn't optimized.
It is working but the result is looking... well... it needs some tuning