Introduction

In last year SIGGRAPH, Epic games presented their real time GI solution which based on voxel cone tracing. They showed some nice results which attract me to implement the technique and my implementation runs at around 22~30fps (updated every frame) at 1024×768 screen resolution using a 256x256x256 voxel volume on my GTX460 graphic card. The demo program can be downloaded here which requires a DX11 GPU to run.

With GI

Without GI

Overview

There are 5 major steps in voxel cone tracing:

0. Given a scene with directly lighting only
1. Voxelize the triangle meshes
2. Construct sparse voxel octree
3. Inject direct lighting into the octree
4. Filter the direct lighting to generate mip-map
5. Sample the mip-mapped values by cone tracing

Step 0	Step 1	Step 2
Step 3	Step 4	Step 5

The step 1 and 2 can be done only once for static geometry while step 3 to 5 need to be done every frame. The following sections will briefly describe the above steps, you may want to take a look at the original paper first as some of the details will not be repeated in the following sections.

Voxelization pass

The first step is to voxelize the scene. I first output all the voxels from triangle meshes into a big voxel fragment queue buffer. Each voxel fragment use 16 bytes:

The first 4 bytes store the position of that voxel inside the voxel volume which is at most 512, so 4 bytes is enough to store the XYZ coordinates.

Voxel fragments are created using the conservative rasterization described in this OpenGL Insights charter which requires only 1 geometry pass by enlarging the triangle a bit using the geometry shader. So I modify my shader generator to generate the shaders for creating voxel fragments based on the material.

Scene before voxelize

Scene after voxelize

Octree building pass

I use a similar data structure that described in the voxel cone tracing paper and giga voxel paper with a large octree node buffer storing the octree/voxel node data (with 8 node grouped into 1 tile which described in the paper) and a 3D texture storing the reflected radiance and alpha from the direct lighting at that voxel (I assume all the surface reflect light diffusely so only 1 3D texture is used which is different from the paper storing incoming radiance).

Each octree node is 28 bytes with the first 4 bytes storing the child tile index with another 24 bytes storing the neighbor node byte offset from the start of the octree node buffer:

The child node tile index is also used to index the 3D texture brick(more details can be found in voxel cone tracing paper about the texture brick) which store the reflected radiance of that octree node tile. And since I use 5 bits for store different bit flags for dynamic update, so my voxel volume can only be at most 512x512x512 large.

In the octree leaf node, it stores the voxel data directly which only use 16 bytes:

the first 4 bytes store the bit flag to indicate whether that voxel is created from static mesh so that it will not be overwrite by dynamic geometry. Also the 1 byte counter stored along with the voxel normal is used to perform averaging when different voxel fragments fall into the same voxel. The steps to perform atomic average can be found in the OpenGL Insight Chapter.

So given the voxel fragment queue output from the previous steps, we can build the octree using the steps described in  OpenGL Insights Chapter and average the octree leaf node values when different voxels fall into the same node.

showing highest octree level

showing the octree with the voxels

Inject direct lighting pass

After building the voxelized scene, we need to add the lighting data into the data structure to calculate global illumination. First, we render the shadow map from the light’s point of view. Then for each pixel of the shadow map, we can re-construct the world position of the shadow map texel and then traverse down the octree data structure to calculate the reflected radiance(assume reflected diffusely) and write it to the 3D texture bricks.

In my engine, I use cascade shadow map with 4 cascades, the last cascade is used to perform the light injection, with the slope scale depth bias disabled, otherwise, the re-constructed world position may not located exactly inside the voxel. Also, it is better to stablize the shadow map (in the demo, the position of the directional light is computed from view camera)  in order to avoid flicking during cone tracing step when the camera moves. But after stablizing the shadow map, not all the voxels are filled with lighting data at 512x512x512 voxel resolution as the shadow map is not fully utilized…

This artifact will also occur when the shadow map resolution is low. Consider the the figure below, we have a directional light injecting lighting to the scene (which have a wall and floor).

As we launch one thread for each shadow map texel to determine which voxel get light injected, we can only inject light to 3 voxels(the white square in the figure) in the above case. While if the shadow map resolution is high enough, those 5 vertical voxels should all be injected with light.

Filtering pass

In order to perform cone tracing step, we need to filter the lighting data at the leaf node of the octree. The voxel lighting data is filtered anisotropically along the positive and negative XYZ directions. For example, in a 2D case(which is easier to explain and very similar to the 3D case), for each node tile, we will have 5×5 voxels which will be filtered to 3×3 voxels in the upper mip level like this:

To filter the center voxel(texel E in the figure) in the node tile in upper mip level along the +X direction, we need to consider the 9 texels (texel g, h, i, l, m, n, q, r, s in the figure) in the lower level, The texels are divided into 4 groups as the figure below:

in each group, we filter along the +X direction, for example in the upper left group, we first alpha blend the value in the +X direction and then average them to get the value for that group.

After calculate the values for all the 4 groups, we can compute the filtered center voxel value by repeating the above steps for the 4 group values.

But for the corner values(e.g. texel a, b, f, g) in a node tile we cannot access all the 9 texels to compute a filtered value as those values are in the neighbor node tile. So the group value are partially computed and store in the 3D texture first, then we rely on the transfer step described in the cone tracing paper to complete the calculation. In other words, during the transfer steps, some of the filtered value is computed by first alpha blending the neighbor group values followed by averaging them, and other filtered value will be computed by averaging first followed by alpha blending. Although this calculation is not commutative, but this can reduce the number of dispatch passes to compute the filtered value with a similar result.

So we have 6 directional values for a filtered voxels. To get a sample from it for a particularly direction, we can sample it using the method like ambient cube with 3 texture read (the following code is simplified):

1. float3 sampleAnisotropic(float3 direction)
2. {
3.     float3 nSquared = direction * direction;
4.     uint3 isNegative = ( direction < 0.0 );
5.     float3 filteredColor=
6.         nSquared.x * anisotropicFilteredBrickValue[isNegative.x] +
7.         nSquared.y * anisotropicFilteredBrickValue[isNegative.y+2] +
8.         nSquared.z * anisotropicFilteredBrickValue[isNegative.z+4];
9.     return filteredColor;
10. }

Voxel Cone Tracing pass

After doing the above steps, we finally can compute our global illumination. We first consider the simple case for ambient occlusion, which need to compute the AO integral. We approximate it by partitioning the integral with several cones:

Where each partition need to multiply with a weight W. In my implementation, 6 cones are traced with 60 degree over the hemi-sphere in the following direction (Y-axis as the up vector):

each with a weight W:

Then to calculate the visibility inside one cone, we take multiple samples from the filtered 3D texture bricks along the cone direction:

But, the remaining problem is to determine the sampling position. Since we manually filter the 3D texture bricks, the hardware quadrilinear interpolation will not work.  So to avoid performing the quadrilinear interpolation manually, It is better to make sampling position located at each mip level, having voxel size equals to the cone width at that position. This position is calculated by assuming the shape of voxel is sphere rather than cube(which is easier to calculate) as follows:

Then the sampling location can be calculated given the cone origin, cone angle, trace direction and voxel radius with some simple geometry. And here is the voxel AO result:

AO result using 512x512x512 voxel volume

Next, for diffuse indirect illumination, the calculation is very similar to AO:

The indirect diffuse calculation is done using the same cones as the AO so that both the indirect diffuse and AO can be calculated at the same time:

Finally, we calculate the indirect specular, where cone is traced in the reflected view direction along the surface normal. The cone angle is depends on the glossiness, g, of the material, currently I use the following equation to calculate the angle:

The glossiness is limited to a range so that the cone angle will not be too narrow to avoid stepping through thin walls. Here is the result of indirect specular:

In my engine, I use a light pre-pass renderer, so for the opaque objects, I calculate the indirect lighting by rendering a full screen quad after the lighting pass which is then blend on top of the direct lighting buffer using the following blend state:

With the value of AO store in alpha channel. this can apply AO to both the direct and indirect lighting. Sometimes different AO intensity is need for the direct and indirect light, this blend state can be used instead:

which apply the alpha blending to only the direct lighting and the indirect AO is applied inside the shader.

When filtering mip map directionally from the leaf node, normal direction is not taken into account, which results in more light leaking for thin objects as below. And this artifact can be hidded slightly with the AO value calculated in cone tracing:

light leak through thin geometry

apply the AO can only hide the leaking a bit

Dynamic update

To make the indirect illumination calculation faster, 5 ways are used to speed up the calculation a bit.

First, when the scene is initialized, the first 4 steps that described in the overview section are performed for all the static geometry. Then in every frame, we re-calculate all the 5 steps for voxels that affected by dynamic objects. So, only dynamic objects will be voxelized every frame (while static voxels are already stored in the octree and we don’t overwrite those data which can be identified by bit flag stored in the node). Those dynamic voxel fragments are appended to the end of the octree node buffer which can be cleared easily. Also, we need to reset the static octree node neighbor offset which points to dynamic nodes, and those static nodes that affected by dynamic voxels in previous and current frame need to re-filter again. Those nodes are found by dispatching threads for all the nodes and queue those node index into another buffer. So only those with changed value will be re-calculated.

Second, for voxelizing the dynamic geometry, the InterlockedMax() function is used to compute both the diffuse and normal for the octree voxel which is faster than performing an atomic average. Note that using InterlockedMax() function for the normal will decrease the lighting quality if the voxel volume is at a small size(e.g. 256x256x256). You can see the artifact in the below figure:

Reflect light using an average voxel normal

Reflect light using InterlockedMax() voxel normal.

So I only use InterlockedMax() function for normal in the dynamic geometry and keeping the static geometry computing an average normal. That is why you can see the octree voxel data structure(in octree building pass section) store a 1 byte counter along with the normal but not other attributes.

Third, I perform a view-frustum culling when injecting the direct lighting into the octree because there is no point to filter the light that is far from the camera where we never sample it from the current point of view. An extended frustum is calculated from the current camera (refer to the figure below) for culling.

This frustum is simply moving the camera backward a bit with increased far plane using the same field of view. The extended distance is calculated by the maximum cone tracing distance (I limited the cone tracing distance in the demo, which lost the ability to sample from far objects like the sky) and the filtered voxel size.

Fourthly, I perform the cone tracing pass at half resolution of the screen resolution. However, this result in visible artifact when up-scaling to full resolution especially at the edge of the geometry (the strength of the indirect lighting in the following screen shots are increased to show the artifact more clearly).

up-sample with only bilinear filtering

So, I first decided to just fix those pixels by finding them with an edge detection filter using depth buffer.

perform an edge detection pas

And then perform cone tracing at those pixel again at full resolution:

perform the cone-tracing again in the edge pixel at full resolution

Some of the artifacts are gone, but the frame rate drops a lot again…

So, my second attempt is to just simply blur those pixels (averaging with the neighbor pixel). The quality of blurring is not as good as re-compute the cone tracing, but it is much faster.

up sample with blurring at edge

cone tracing at full resolution for reference

Lastly, The voxel world can be updated at a different frequency than the render frame rate. By assuming the  light source/dynamic models are not moving very fast, we can take advantage of the temporary coherency and perform the voxelization at a lower frequency, say update at every 5 frames. We can trade the accuracy of the voxels for update speed, but this may result in the artifact shown below:

The voxel model is lag behind the triangle mesh due to the lower update frequency

Conclusion

The advantage of voxel cone tracing is to compute the GI in real-time with both the dynamic lighting and geometry. Also the specular indirect lighting gives a very nice glossy effect. However, it uses lots of the processing power/memory and quality of lighting is not as good as the baked solution. In my implementation, I can only use 1 directional light to compute single bounce indirect lighting. And there are still room to improve in my implementation as fewer cones can be used for tracing, using the depth buffer for view frustum culling, better up-sampling when performing cone tracing and divide the voxels into several regions to handle a larger scene like Unreal Engine does. But there is never enough time to implement all that stuff… So I decided to release the demo at this stage first. In the demo, I have added some simple interface (for changing stuffs like the light direction, indirect lighting strength) for you to play around with. Hope you all enjoy the demo.

Finally, I would like to thanks Kevin Gadd and Luke Hutchinson for reviewing this article.

 

References

[1] The Technology Behind the “Unreal Engine 4 Elemental demo” http://www.unrealengine.com/files/misc/The_Technology_Behind_the_Elemental_Demo_16x9_(2).pdf

[2] Interactive Indirect Illumination Using Voxel Cone Tracing http://maverick.inria.fr/Publications/2011/CNSGE11b/GIVoxels-pg2011-authors.pdf

[3] Octree-Based Sparse Voxelization Using the GPU Hardware Rasterizer http://www.seas.upenn.edu/%7Epcozzi/OpenGLInsights/OpenGLInsights-SparseVoxelization.pdf

[4] GigaVoxels: A Voxel-Based Rendering Pipeline For Efficient Exploration Of Large And Detailed Scenes
http://maverick.inria.fr/Publications/2011/Cra11/

[5] GPU Gems 2: Conservative Rasterization http://http.developer.nvidia.com/GPUGems2/gpugems2_chapter42.html
[6] Shading in Valve’s Source Engine http://www.valvesoftware.com/publications/2006/SIGGRAPH06_Course_ShadingInValvesSourceEngine.pdf
[7] Perpendicular Possibilities http://blog.selfshadow.com/2011/10/17/perp-vectors/

[8] A couple of notes about Z http://www.humus.name/index.php?ID=255

SSAO (Screen space ambient occlusion) is a common post processing effect that approximate how much light is occluded in a given surface by the surrounding objects. In this year SIGGRAPH, there are a few slides in “The Technology behind the Unreal Engine 4 Elemental Demo” about how they implement SSAO. Their technique can either use only the depth buffer or with the addition of per-pixel normal. And I tried to implement both version with a slight modification:

Using only the depth buffer

The definition of ambient occlusion is to calculate the visibility integral over the hemisphere of a given surface:

To approximate this in screen space, we design our sampling pattern as paired samples:

paired sample pattern

So for each pair of samples, we can approximate how much the shading point is occluded in 2D instead of integrating over the hemisphere:

The AO term for each given pair of samples will be min( (θleft + θright)/π, 1). Then by averaging the AO terms of all the sample pairs (in my case, there are 6 pairs), we achieve the following result:

Dealing with large depth differences

As seen from the above screen shot, there is dark halos around the knight. But the knight should not contribute AO to the castle as he is too far away. So to deal with the large depth differences. I adopt the approach used in Toy Story 3. If one of the paired sample is too far away from the shading point, say the red point in the following figure, it will be replace by the pink point, which is on the same plane as the other valid paired sample:

So we can interpolate between the red point and the pink point for dealing with the large depth difference. Now the dark halo has gone:

The above treatment only handle if one of the paired sample is far away from shading point. What if both of the samples have large depth differences?

dark halo artifact is shown around the sword

AO strength of this pic is increased to high light the artifact

In this case, it will result in the dark halo around the sword in the above screen shot. Remember we are averaging the all the paired samples to compute the final AO value. So to deal with this artifact, we just assign a weight to each paired samples and then re-normalize the final result. Say, for each paired sample, if both of the samples are within a small depth differences, that sample pair will have a weight of 1. If only 1 sample is far away, that pair will have a weight of 0.5. And finally if both of the samples is far away, the weight will be 0. This can eliminate most(but not all) of the artifacts:

Approximating arc-cos function

In this approach, the AO is calculated by using the angle between the paired samples, which need to evaluate the arc-cos function which is a bit expensive. We can approximate acos(x) with a linear function:  π(1-x)/2.

And the resulting AO looks much darker with this approximation:

computed with the arc-cos function

computed with the linear approximation

Note that the maximum error between the two function is around 18.946 degree.

This may affect the AO for the area of a curved surface with low tessellation. You may either need to increase the bias angle threshold or switch to a more accurate function. So my second attempt is to approximate it with a quadratic function:  π(1- sign(x) * x * x)/2.

And this approximation shows a much similar result to the one using the arc-cos function.

computed with the arc-cos function

computed with the quadratic approximation

And the maximum error of this function is around 9.473 degree.

Using per-pixel normal

We can enhance the details of AO by making use of the per-pixel normal. The per-pixel normal is used for further restricting the angle to compute the AO where the angle θleft, θright are clamped to the tangent plane :

And here is the final result:

Conclusion

The result of this AO is pleasant by taking total 12 samples per pixel and with 16 rotation in 4×4 pixel block at half resolution. I did not apply bilateral blur to the AO result, but applying the blur may gives a softer AO look. Also approximating the arc-cos function with a linear function although is not accurate, but it gives a good enough result for me. Finally more time are need to spend on generating the sampling pattern in the future where the pattern I currently used is nearly uniform distributed (with some jittering).

References
[1] The Technology behind the Unreal Engine 4 Elemental Demo http://advances.realtimerendering.com/s2012/Epic/The%20Technology%20Behind%20the%20Elemental%20Demo%2016×9.pptx
[2] Rendering techniques in Toy Story 3 http://advances.realtimerendering.com/s2010/Ownby,Hall%20and%20Hall%20-%20Toystory3%20(SIGGRAPH%202010%20Advanced%20RealTime%20Rendering%20Course).pdf
[3] Image-Space Horizon-Based Ambient Occlusion http://www.nvidia.com/object/siggraph-2008-HBAO.html
[4] http://www.wolframalpha.com/
[5] The models are export from UDK and extracted from Infinity Blade using umodel.exe

In the last few weeks, I was busy with rewriting my iPhone engine so that it can also run on the Windows platform (so that I can use Visual Studio in stead of Xcode~) and most importantly, I can play around with D3D11. During the rewrite, I want to improve the process of writing shaders so that I don’t need to write similar shaders multiple times for each shader permutation (say, for each surface, I have to write a shader for static mesh, skinned mesh, instanced static mesh… multiplies with the number of render pass), and instead I can focus on coding how the surface would looks like. So I decided to write a shader generator that will generate those shaders which is similar to the surface shader in Unity. I choose the surface shader approach instead of a graph based approach like Unreal Engine, because being a programer, I feel more comfortable (and faster) to write code than dragging tree nodes using the GUI. In the current implementation of the shader generator, it can only generate vertex and pixel shaders for the light pre pass renderer which is the lighting model used before.

Defining the surface

To generate the target vertex and pixel shaders by the shader generator, we need to define how the surface looks like by writing surface shader. In my version of surface shader, I need to define 3 functions: vertex function, surface function and lighting function. The vertex function defines the vertex properties like position and texture coordinates.

1. VTX_FUNC_OUTPUT vtxFunc(VTX_FUNC_INPUT input)
2. {
3.     VTX_FUNC_OUTPUT output;
4.     output.position = mul( float4(input.position, 1), worldViewProj  );
5.     output.normal = mul( worldInv, float4(input.normal, 0) ).xyz;
6.     output.uv0 = input.uv0;
7.     return output;
8. }

The surface function which describe how the surface looks like by defining the diffuse color of the surface, glossiness and the surface normal.

1. SUF_FUNC_OUTPUT sufFunc(SUF_FUNC_INPUT input)
2. {
3.     SUF_FUNC_OUTPUT output;
4.     output.normal = input.normal;
5.     output.diffuse = diffuseTex.Sample( samplerLinear, input.uv0 ).rgb;
6.     output.glossiness = glossiness;
7.     return output;
8. }

Finally the lighting function will decide which lighting model is used to calculate the reflected color of the surface.

1. LIGHT_FUNC_OUTPUT lightFuncLPP(LIGHT_FUNC_INPUT input)
2. {
3.     LIGHT_FUNC_OUTPUT output;
4.     float4 lightColor = lightBuffer.Sample(samplerLinear, input.pxPos.xy * renderTargetSizeInv.xy );
5.     output.color = float4(input.diffuse * lightColor.rgb, 1);
6.     return output;
7. }

By defining the above functions, writer of the surface shader only need to fill in the output structure of the function by using the input structure with some auxiliary functions and shader constants provided by the engine.

Generating the shaders

As you can see in the above code snippet, my surface shader is just defining normal HLSL function with a fixed input and output structure for the functions. So to generate the vertex and pixel shaders, we just need to  copy these functions to the target shader code which will invoke those functions defined in the surface shader. Take the above vertex function as an example, the generated vertex shader would look like:

1. #include “include.h”
2. struct VS_INPUT
3. {
4.     float3 position : POSITION0;
5.     float3 normal : NORMAL0;
6.     float2 uv0 : UV0;
7. };
8. struct VS_OUTPUT
9. {
10.     float4 position : SV_POSITION0;
11.     float3 normal : NORMAL0;
12.     float2 uv0 : UV0;
13. };
14. typedef VS_INPUT VTX_FUNC_INPUT;
15. typedef VS_OUTPUT VTX_FUNC_OUTPUT;
16. /********************* User Defined Content ********************/
17. VTX_FUNC_OUTPUT vtxFunc(VTX_FUNC_INPUT input)
18. {
19.     VTX_FUNC_OUTPUT output;
20.     output.position = mul( float4(input.position, 1), worldViewProj  );
21.     output.normal = mul( worldInv, float4(input.normal, 0) ).xyz;
22.     output.uv0 = input.uv0;
23.     return output;
24. }
25. /******************** End User Defined Content *****************/
26. VS_OUTPUT main(VS_INPUT input)
27. {
28.     return vtxFunc(input);
29. }

During code generation, the shader generator need to figure out what input and output structure are needed to feed into the user defined functions. This task is simple and can be accomplished by using some string functions.

Simplifying the shader

As I mentioned before, my shader generator is used for generating shaders used in the light pre pass renderer. There are 2 passes in light pre pass renderer which need different shader input and output. For example in the G-buffer pass, the shaders are only interested in the surface normal data but not the diffuse color while the data need by second geometry pass are the opposite. However all the surface information (surface normal and diffuse color) are defined in the surface function inside the surface shader. If we simply generating shaders like last section, we will generate some redundant code that cannot be optimized by the shader compiler. For example, the pixel shader in G buffer pass may need to sample the diffuse texture which require the texture coordinates input from vertex shader but the diffuse color is actually don’t needed in this pass, the compiler may not be able to figure out we don’t need the texture coordinates output in vertex shader. Of course we can force the writer to define some #if preprocessor inside the surface function for the particular render pass to eliminate the useless output, but this will complicated the surface shader authoring process as writing surface shader is to describe how the surface looks like, ideally, don’t need to worry about the output of a render pass.

So the problem is to figure out what the output data are actually need in a given pass and eliminate those outputs that are not needed. For example, given we are generating shaders for the G buffer pass and a surface function:

1. SUF_FUNC_OUTPUT sufFunc(SUF_FUNC_INPUT input)
2. {
3.     SUF_FUNC_OUTPUT output;
4.     output.normal = input.normal;
5.     output.diffuse = diffuseTex.Sample( samplerLinear, input.uv0 ).rgb;
6.     output.glossiness = glossiness;
7.     return output;
8. }

We only want to keep the variables output.normal and output.glossiness. And the variable output.diffuse, and other variables that is referenced by output.diffuse (diffuseTex, samplerLinear, input.uv0) are going to be eliminated. To find out such variable dependency, we need to teach the shader generator to understand HLSL grammar and find out all the assignment statements and branching conditions to derive the variable dependency.

To do this, we need to generate an abstract syntax tree from the shader source code. Of course we can write our own LALR parser to achieve this goal, but I chose to use lex&yacc (or flex&bison) to generate the parse tree. Luckily we are working on a subset of the HLSL syntax(only need to define functions and don’t need to use pointers) and HLSL syntax is similar to C language, so modifying the ANSI-C grammar rule for lex&yacc would do the job. Here is my modified grammar rule used to generate the parse tree. By traversing the parse tree, the variable dependency can be obtained, hence we know which variables need to be eliminated and eliminate them by taking out the assignment statements, then the compiler will do the rest. Below is the simplified pixel shader generated in the previous example:

1. #include “include.h”
2. cbuffer _materialParam : register( MATERIAL_CONSTANT_BUFFER_SLOT_0 )
3. {
4.     float glossiness;
5. };
6. Texture2D diffuseTex: register( MATERIAL_SHADER_RESOURCE_SLOT_0 );
7. struct PS_INPUT
8. {
9.     float4 position : SV_POSITION0;
10.     float3 normal : NORMAL0;
11. };
12. struct PS_OUTPUT
13. {
14.     float4 gBuffer : SV_Target0;
15. };
16. struct SUF_FUNC_OUTPUT
17. {
18.     float3 normal;
19.     float glossiness;
20. };
21. typedef PS_INPUT SUF_FUNC_INPUT;
22. /********************* User Defined Content ********************/
23. SUF_FUNC_OUTPUT sufFunc(SUF_FUNC_INPUT input)
24. {
25.     SUF_FUNC_OUTPUT output;
26.     output.normal = input.normal;
27.                                                                  ;
28.     output.glossiness = glossiness;
29.     return output;
30. }
31. /******************** End User Defined Content *****************/
32. PS_OUTPUT main(PS_INPUT input)
33. {
34.     SUF_FUNC_OUTPUT sufOut= sufFunc(input);
35.     PS_OUTPUT output;
36.     output.gBuffer= normalToGBuffer(sufOut.normal, sufOut.glossiness);
37.     return output;
38. }

Extending the surface shader syntax

As I use lex&yacc to parse the surface shader, I can extend the surface shader syntax by adding more grammar rule, so that writer of the surface shader can define what shader constants and textures are needed in their surface function to generate the constant buffer and shader resources in the source code. Also my surface shader syntax permit user to define their struct and function other than their 3 main functions (vertex, surface and lighting function), where they will also be copied into the generated source code. Here is a sample of how my surface shader would looks like:

1. RenderType{
2.     opaque;
3. };
4. ShaderConstant
5. {
6.     float glossiness: ui_slider_0_255_Glossiness;
7. };
8. TextureResource
9. {
10.     Texture2D diffuseTex;
11. };
12. VTX_FUNC_OUTPUT vtxFunc(VTX_FUNC_INPUT input)
13. {
14.     VTX_FUNC_OUTPUT output;
15.     output.position = mul( float4(input.position, 1), worldViewProj  );
16.     output.normal = mul( worldInv, float4(input.normal, 0) ).xyz;
17.     output.uv0 = input.uv0;
18.     return output;
19. }
20. SUF_FUNC_OUTPUT sufFunc(SUF_FUNC_INPUT input)
21. {
22.     SUF_FUNC_OUTPUT output;
23.     output.normal = input.normal;
24.     output.diffuse = diffuseTex.Sample( samplerLinear, input.uv0 ).rgb;
25.     output.glossiness = glossiness;
26.     return output;
27. }
28. LIGHT_FUNC_OUTPUT lightFuncLPP(LIGHT_FUNC_INPUT input)
29. {
30.     LIGHT_FUNC_OUTPUT output;
31.     float4 lightColor = lightBuffer.Sample(samplerLinear, input.pxPos.xy * renderTargetSizeInv.xy );
32.     output.color = float4(input.diffuse * lightColor.rgb, 1);
33.     return output;
34. }

Conclusions

This post described how I generate vertex and pixel shader source codes for different render passes by defining a surface shader which avoid me to write similar shaders multiple times and without worrying the particular shader input and output for each render pass. Currently, the shader generator can only generate vertex and pixel shader in HLSL for static mesh in the light pre pass renderer. The shader generator is still under progress where generating shader source code for the forward pass is still have not done yet. Besides domain, hull and geometry shaders are not implemented. Also GLSL support is missing, but this can be generated (in theory…) by building a more sophisticated abstract syntax tree during parsing the surface shader grammar or defining some new grammar rule in the surface shader (using lex&yacc) for easier generating both HLSL and GLSL source code. But these will be left for the future as I still need to rewrite my engine and get it running again…

References
[1] Unity – Surface Shader Examples http://docs.unity3d.com/Documentation/Components/SL-SurfaceShaderExamples.html
[2] Lex & Yacc Tutorial http://epaperpress.com/lexandyacc/
[3] ANSI C grammar, Lex specification http://www.lysator.liu.se/c/ANSI-C-grammar-l.html
[4] ANSI C Yacc grammar http://www.lysator.liu.se/c/ANSI-C-grammar-y.html
[5] http://www.ibm.com/developerworks/opensource/library/l-flexbison/index.html
[6] http://www.gamedev.net/topic/200275-yaccbison-locations/

(Note: this was originally posted on Flippfly.com)

I questioned the wisdom of writing this, since as of yet, I’ve not released a highly successful game as an independent developer since quitting my day job back in April. Forest and I have high hopes for Flippfly, but aside from our moderately successful Monkey Drum Deluxe, there’s really not a lot of inherent credibility to my words that comes with having highly successful products to back them up.

But there’s a trend of beliefs and focus among some indies that’s really kind of discouraging to me.

Namely: I think many have forgotten that the most important factor to success as a game developer is making an excellent game, and started to believe that financial success is either random, or mostly due to factors outside of the game itself.

Now – keep in mind that I said excellent (as opposed to decent, good, or even great) and that when I use that word, I mean: fun, appealing, polished, and accessible (and  don’t take accessible to mean casual or broadly appealing.)

Recently a tweet from Jon Blow (developer of Braid) made me think on this issue again:

This Gamasutra article just kind of makes me angry. I should learn not to care: gamasutra.com/view/feature/1…

— Jonathan Blow (@Jonathan_Blow) June 29, 2012

This was in response to a business-centric postmortem about a game that sold 7 copies titled “congratulations, your first indie game is a flop.”

Jon went on to explain:
“He made a game that there’s no reason for people to want, but acts like he is entitled to have people buy it / press cover it.”

Now to be fair, I think Jon’s take on this was harsh – I  found the article in question to be informative, and as pointed out by Michael Brough, talking about failures is important. The developer acknowledged mistakes, and ultimately showed no regret at having done something he loved and believed in.

My concern is that people seem to have an expectation that their game will do reasonably well as long as they get all their ducks in a row, and if this doesn’t happen, often the last thing they focus on is the game itself. Time and again I’ve seen people reference their 75% or so ratings and then go on to talk as if these are “great” reviews and that they just need to get their great game in front of people.

I think this kind of thinking is a big mistake.

Hear me out: it should be a self-evident fact that if you expect to succeed financially, you’re going to need lots of eyes on your game, especially if you’re charging a couple bucks or less for it. This can happen in a variety of ways, but it mostly boils down to two: either you spend money on marketing, or you make a game that is so good that its quality and value make it impossible to ignore. You want people to play it, share it, tweet about it, talk about it at work, review it, and feature it, not because of a great icon or an attractive promo video, but because it’s unquestionably just that good. You want it to be the game about which people say “you really have to play this!”

You should be able to think of your game like a dry pile of sticks doused in gasoline, that just needs a spark to ignite it.

It’s tempting to look at counter-examples: all the good games that somehow get passed over, and all the mediocre games that somehow manage to sell millions.

But in the absence of big marketing dollars, I would argue that:

Mediocre games usually fail.
Good games often fail.
Excellent games rarely fail.

Every other case is just noise.

So am I saying that marketing, PR, great icons, promo videos, a great website, social features, killer screenshots, and personal connections are unimportant?

Of course not!

But if your game is less than excellent, then all this stuff is like trying to push a rock up a hill in today’s market. That’s not a fulfilling way to spend your life. And the weaker your game is, the more time you’re going to spend trying to make all these supporting factors make up for it – a really bad cycle to be in when time is your most precious asset!

What’s cool about setting out to make excellent games, is that in addition to taking so much of the randomness out of your success potential, you’re going to enjoy a much more fulfilling career!

Now I feel I should make a point to say that sales isn’t the only type of success – and there is certainly room for every type of game, as Rami Ismail of Vlambeer points out. It’s a big space and not everybody in it is trying to make a living at it. We actually just submitted a little toy to the app store called “Creepy Eye” – an experiment using face-tracking and the gyroscope that we hadn’t seen explored before. Making an artful experiment or a cool diversion is a reward in itself. But my concern is when people start to feel a sense of entitlement or surprise when these experiments and “pretty good” games don’t garner any attention or sales, with sometimes lengthy and publicized business-focused analysis of where their monetization strategy failed, or scary-sounding warnings to other would-be indie developers.

If you want to sell games, and you don’t like throwing dice with your financial future, you need to be determined to produce excellence.

So do yourself a favor: take a break from your monetization strategizing, video-editing, press-emailing, buzz-creating, and icon-tuning for a minute, and ask yourself:
 

“Is this game Excellent yet?”

Continue with previous post, this post will describe how light map is calculated from the photon map. My light map stores incoming radiance of indirect lighting on a surface which are projected into Spherical Harmonics(SH) basis. 4 SH coefficients is used  for each color channels. So 3 textures are used for RGB channels (total 12 coefficients).

Baking the light map

To bake the light map, the scene must have a set of unique, non-overlapping texture coordinates(UV) that correspond to a unique world space position so that the incoming radiance at a world position can be represented. This set of UV can be generated inside modeling package or using UVAtlas. In my simple case, this UV is mapped manually.

To generate the light map, given a mesh with unique UV and the light map resolution, we need to rasterize the mesh (using scan-line or half-space rasterization) into the texture space with interpolated world space position across the triangles. So we can associate a world space position to a light map texel. Then for each texel, we can sample the photon map at the corresponding world space position by performing a final gather step just like previous post for offline rendering. So the incoming radiance at that world space position, hence the texel in the light map, can be calculated. Then the data is projected into SH coefficients, stored in 3 16-bits floating point textures. Below is a light map that extracting the dominant light color from SH coefficients:

The baked light map showing the dominant light color from SH coefficients

Using the light map

After baking the light map, during run-time, the direct lighting is rendering with usual way, a point light is used to approximated the area light in the ray traced version, the difference is more noticeable at the shadow edges.

direct lighting only, real time version

direct lighting only, ray traced version

Then we sample the SH coefficients from the light map to calculate the indirect lighting

indirect lighting only, real time version

indirect lighting only, ray traced version

Combining the direct and indirect lighting, the final result becomes:

direct + indirect lighting, real time version

direct + indirect lighting, ray traced version

Rendered with normal map

Indirect lighting with normal map

We can also applying some tessellation, adding some ambient occlusion(AO) to make the result more interesting:

Rendered with light map, normal map, tessellation and AO

Rendered with light map, normal map, tessellation and AO

Conclusion

This post gives an overview on how to bake light map of indirect lighting data by sampling from the photon map. I use SH to store the incoming radiance, but other data can be stored such as storing the reflected diffuse radiance of the surface, which can reduce texture storage and doesn’t require floating point texture. Besides, the SH coefficients can be store per vertex in the static mesh instead of light map. Lastly, by sampling the photon map with final gather rays, light probe for dynamic objects can also be baked using similar methods.

References

March of the Froblins: http://developer.amd.com/samples/demos/pages/froblins.aspx

Lighting and Material of HALO 3: http://www.bungie.net/images/Inside/publications/presentations/lighting_material.zip

In this generation of computer graphics, global illumination (GI) is an important technique which calculate indirect lighting within a scene. Photon mapping is one of the GI technique using particle tracing to compute images in offline rendering. Photon mapping is an easy to implement technique, so I choose to learn it and my target is to bake light map storing indirect diffuse lighting information using the photon map. Photon mapping consists of 2 passes: photon map pass and render pass, which will be described below.

Photon Map Pass

In this pass, photons will be casted into the scene from the position of light source. Each photon store packet of energy. When photon hits a surface of the scene, the photon will either be reflected (either diffusely or specularly), transmitted  or absorbed, which is determined by Russian roulette.

Photons are traced in the scene to simulate the light transportation

However, it is more convenient to store radiance than storing energy in photon because when using punctual light source(e.g. point light), it is hard to compute the energy emits given the light source radiance. So I use the method described in Physically Based Rendering, a weight of radiance is stored in each photon:

When a photon hits a surface, the probability of being reflected in a new random direction used in Russian roulette is:

This probability equation is chosen because photon will have a higher chance of being reflected if it is brighter. If the photon is reflected, its radiance will be updated to:

And the photon will continue to trace in the newly reflected direction.

Render Pass

In render pass, the direct and indirect lighting is computed separately. The direction lighting is computed using ray tracing.

Direct light only

The indirect lighting is computed by sampling from the photon map. When calculating the indirect lighting in a given position(in this case, the shading pixel), we can locate N nearby photons in photon map to estimate the incoming radiance using kernel density estimation. A kernel function need to satisfy the conditions:

Then the density can be computed using kernel estimator for N samples within a distance d (i.e. the distance of the photon that is the most far away in the N samples):

Then the reflected radiance at the shading position can be computed with:

However, the result showing some circular artifact:

Using the photon map directly for indirect diffuselight would show artifact

To tackle this problem, either increase the number of photon to a very high number, or we can perform a final gather step. In the final gather step, we shoot a number of final gather rays from the pixel that we are shading in random direction over the hemisphere of the shading point.

Final gather rays are casted from every shading position

When final gather ray hit another surface, then the photon map is queried just like before and the reflected radiance from this surface will be the incoming radiance of the shading pixel. Using Monte Carlo integration, the reflected radiance at the shading pixel can be calculated by sampling the final gather rays. Here is the final result:

Direct light + Indirect light, with final gather

Indirect light only, with final gather

Conclusion

In this post, the steps to implement photon map is briefly described. It is a 2 passes approach with the photon map pass building a photon map as kd-tree representing the indirect lighting data and the render pass use the photon map to compute the final image. In next part, I will describe how to make use of the photon map to bake light map for real time application.

References

A Practical Guide to Global Illumination using Photon Maps: http://nameless.cis.udel.edu/class_data/cg/jensen_photon_mapping_tutorial.pdf

Physically Based Rendering: http://www.pbrt.org/

Continue with the previous post, after filling the triangle with scan line or half-space algorithm, we also need to interpolate the vertex attributes across the triangle so that we can have texture coordinates or depth on every pixel. However we cannot directly interpolate those attributes in screen space because projection transform after perspective division is not an affine transformation (i.e. after transformation, the mid-point of the line segment is no longer the mid-point), this will result in some distortion and this artifact is even more noticeable when the triangle is large:

interpolate in screen space

perspective correct interpolation

Condition for linear interpolation

When interpolating the attributes in a linear way, we are saying that given a set of vertices, vi (where i is any integer>=0) with a set of attributes ai (such as texture coordinates), we have a function mapping a vertex to the corresponding attributes, i.e.

Say, to interpolate a vertex inside a triangle in a linear way, the function f need to have the following properties:

which means that we can calculate the interpolated attributes using the same weight ti used for interpolating vertex position. For functions having the above properties, those functions will be an affine function with the following form:

Depth interpolation

When a vertex is projected from view space to normalized device coordinates(NDC), we will have the following relation (ratio of the triangles) between the view space and NDC space:

substitute equation 1 and 2 into the plane equation of the triangle lies on:

So, 1/zview is an affine function of xndc and yndc which can be interpolated linearly across the screen space (the transform from NDC space to screen space is a linear transform).

Attributes interpolation

In last section, we know how to interpolate the depth of a pixel linearly in screen space, the next problem is to interpolate the vertex attributes(e.g. texture coordinates). In view space, we know that those attributes can be interpolated linearly, so those attributes can be calculated by an affine function with the vertex position as parameters e.g.

Similar to interpolate depth, substitute equation 1 and 2 into the above equation:

Therefore, u/zview is an another affine function of xndc and yndc which can be interpolated linearly across the screen space. Hence we can interpolating u linearly by first interpolate 1/zview and u/zview across screen space, and then divide them per pixel.

The last problem…

Now, we know that we can interpolate the view space depth and vertex attributes linearly across screen space. But during the rasterization state, we only have vertices in homogenous coordinates (vertices are transformed by the projection matrix already), how can we get the zview to do the perspective correct interpolation?

Consider the projection matrix (I use D3D one, but the same for openGL):

After transforming the vertex position, the w-coordinate will be the view space depth!

i.e. w-homogenous = zview 

And look at the matrix again and consider the transformed z-coordinates, it will in a form of:

After transforming to the NDC,

So the depth value can be directly interpolated using z-NDC  for depth test.

Demo

A javascript demo to rasterize the triangles can be viewed here(although not optimized…). And the source code can be downloaded here.

Screen shot of the demo

Conclusion
In this post, the steps to linear interpolate the vertex in screen space is described. And for rasterizing the depth buffer only (e.g. for occlusion), the depth value can be linearly interpolated directly with the z coordinate in NDC space which is even simpler.

References
[1] http://www.lysator.liu.se/~mikaelk/doc/perspectivetexture/
[2] http://www.gamedev.net/topic/581732-perspective-correct-depth-interpolation/
[3] http://chrishecker.com/Miscellaneous_Technical_Articles
[4] http://en.wikipedia.org/wiki/Affine_transformation

 

Points in Homogenous coordinates
In our usual Cartesian Coordinate system, we can represent any points in 3D space in the form of (X, Y, Z). While in Homogenous coordinates, a redundant component w is added which resulting in a form of (x, y, z, w). Multiplying any constant (except zero) to that 4-components vector is still representing the same point in homogenous coordinates. To convert a homogenous point back to our usual Cartesian Coordinate, we would multiply a point in homogenous coordinates so that the w component is equals to one:

In the following figure, we consider the x-w plane, a point (x, y, z, w) is transformed back to the usual Cartesian Coordinates (X, Y, Z) by projecting onto the w=1 plane:

figure 1. projecting point to w=1 plane

The interesting point comes when the w component is equals to zero. Imagine the w component is getting smaller and smaller, approaching zero, the coordinates of point (x/w , y/w , z/w, 1) will getting larger and larger. When w is equals to zero, we can represent a point at infinity.

Line Segments in Homogenous coordinates
In Homogenous coordinates, we still can represent a line segment between two points P0= (x0, y0, z0, w0) and  P1= (x1, y1, z1, w1) in parametric form:

Then we can get a line having the shape:

figure 2. internal line segment

The projected line on w=1 is called internal line segment in the above case.
But what if the coordinates of P0 and P1 having the coordinates where w0 < 0 and w1 > 0 ?

figure 3. external line segment

In this case, it will result in the above figure, forming an external line segment. It is because the homogenous line segment have the form L= P0 + t * (P1-P0), when moving the parameter from t=0 to t= 1, since w0 < 0 and w1 > 0, there exist a point on the homogenous line where w=0. This point is at infinity when projected to the w=1 plane, resulting the projected line segment joining P0 and P1 passes through the point at infinity, forming an external line segment.

The figure below shows how points are transformed before and after perspective projection and divided by w:

figure 4. region mapping

figure 5. external triangle

In the above figure, the shaded area represents the external triangle formed by the points P0, P1 and P2. This kind of external triangles may appear after the perspective projection transform. And this happens in our real world too:

an external triangle in real world

the full triangle of the left photo

In the left photo, it shows an external triangle with one of the triangle vertex far behind the camera while the right photo shows the full view of the triangle and the cross marked the position of the camera where the left photo is taken.

Triangles clipping
To avoid the case of external triangles, lines/triangles should be clipped in homogenous coordinates before divided by the w-component. The homogenous point (x, y, z, w) will be tested with the following inequalities:

(The z clipping plane inequality is 0 <= z <= w in the case for D3D, it depends on how the normalized device coordinates are defined.) Clipping by inequality 1,2,3 will effectively clip all points that with w < 0 because if w < 0, say w = -3:

3 <= x <= -3     =>     3 <= -3

which is impossible. But the point (0, 0, 0, 0) is still satisfy the first 3 inequalities and forming external cases, so inequality 4 is added. Consider a homogenous line with one end as (0, 0, 0, 0), it will equals to:

L= (0, 0, 0, 0) + t * [ (x, y, z, w) - (0, 0, 0, 0) ] = t * (x, y, z, w)

which represent only a single point in homogenous coordinates. So triangle (after clipped by inequality 1, 2, 3) having one or two vertices with w=0 will result in either a line or a point which can be discarded. Hence, after clipping, no external triangles will be produced when dividing by w-component. To clip a triangle against a plane, the triangle may result in either  1 or 2 triangles depends on whether there are 1 or 2 vertex outside the clipping plane:

figure 6. clipping internal triangles

Then the clipped triangles can be passed to the next stage to be rasterized either by a scan line algorithm or by a half-space algorithm.

Below is the clipping result of an external triangles with 1 vertex behind the camera.

clipping external triangle in software rasterizer

Below is another rasterized result:

rasterized duck model

reference of the duck model

Conclusion
In this post, the maths behind the clipping of triangles are explained. Clipping should be done before projecting the homogenous point to the w=1 to avoid taking special cares to clip the external triangles. In the next post, I will talk about the perspective interpolation and the source code will be given in the next post (written in  javascript, drawing to html canvas).

And lastly special thanks to Fabian Giesen for giving feedback during the draft of this post.

References
[1] http://research.microsoft.com/pubs/73937/p245-blinn.pdf
[2] http://medialab.di.unipi.it/web/IUM/Waterloo/node51.html
[3] http://kriscg.blogspot.com/2010/09/software-occlusion-culling.html
[4] http://www.slideshare.net/guerrillagames/practical-occlusion-culling-in-killzone-3
[5] http://www.slideshare.net/repii/parallel-graphics-in-frostbite-current-future-siggraph-2009-1860503
[6] http://fgiesen.wordpress.com/2011/07/05/a-trip-through-the-graphics-pipeline-2011-part-5/
[7] http://devmaster.net/forums/topic/1145-advanced-rasterization/

• Hierarchical Z-Buffer Occlusion Culling – Generating Occlusion Volumes
• Robust Inside and Outside Solid Voxelization

Lets start with the newest and most important information…

Update 4/13/2012

The project is now hosted on BitBucket, http://bitbucket.org/NickDarnell/oxel/overview feel free to contribute or just follow along :)

It’s a Tool Now!

Download

Oxel 1.0.0 – Win32.zip

Requirements

• .Net 4.0
• VS2010 C++ Runtime

Description

Oxel is a tool for generating occluders – primarily for use with the Hierarchical Z-Buffer method of occlusion culling.  Open an .obj file then go to Build > Voxelize to generate the proxy.  Try it out, let me know what you think.

There are some further improvements that have been made over the method that is laid out in the original Generating Occlusion Volumes post,

• Retriangulation
• Evaluating Occlusion-ness
• Filtering Polygons

Retriangulation

I went back to the CSG article Sander van Rossen and Matthew Baranowski wrote.  I noticed near the end that they recommended retriangulating the final CSG mesh because their library doesn’t handle it.

The easiest method I found to do the retriangulation was to just use David Eberly’s Wild Magic math library which can do it, see here.  Before you perform the retriangulation you need to collect all the external/internal edge loops on each surface and then merge collinear edges before performing the retriangulation.

It’s was definitely worth doing, performing the retriangulation saved about 30% of the triangles.

Evaluating Occlusion-ness

When you’re generating occluder geometry you need to ensure that the volumes you’re adding are useful enough to pay the additional polygon tax they will incur.

Oxel achieves this by measuring the number of pixels written to the color buffer by rendering the original visual mesh into the stencil buffer, and then using the stencil buffer to clip a full screen quad while a “samples passed” hardware query is performed to record the number of pixels written.  This is performed from about 64 different camera angles – then when all the samples are summed together this defines the ground truth occlusion of the mesh.

Knowing that information we can evaluate every new box we plan to add to the final occlusion proxy geometry and ask – Does this increase the coverage enough to make it worth it?.  The default threshold is set to 3%.  If a new volume does not at least cover 3% of new silhouette pixels, we don’t include it in the final occluder mesh.

Filtering Polygons

Not every game needs the occluders to be visible from all angles.  For example most racing games will never have the cars jumping so high they see on top of a building.  In those circumstances you may not want to have the occluders bothering to have polygons on the top.

So the tool offers the ability to remove Top and Bottom polygons from the final occluder mesh after everything has finished being processed.

If you decide to filter out all top polygons they’re all removed.  For bottom surfaces this has to be handled slightly differently, see below -

Imagine a bridge structure or a large over hang on a building.  You wouldn’t want the bottom portion of the occluder to be removed from the overhang since it would allow seeing into and through the occluder – because presumably you wouldn’t have double sided rendering enabled.  So to distinguish between bottom surfaces not meant to ever be seen by the player, and bottom surfaces that may be important to higher up potions of the structure, only bottom surfaces within 1 voxel’s height from the base of the mesh are removed.  (See picture above)

Work Continues

I’ve got some more ideas I need to test out but I wanted to go ahead and cut a version of where I’m at and let others take a look and give me some feedback.

One area I need to investigate next is a better way to generate the boxes.  Currently they are just expanded in all directions equally, but that’s not ideal.  I’m thinking about trying a parallel brute force method that would test all possible boxes at a specific origin point to find the best shaped box to generate from that point.

Oxel: A Tool for Occluder Generation @ nickdarnell.com

Tiled Resources for Xbox 360 and Direct3D 11 – Matt Lee

This talk was about mega-texturing in DirectX 11/Xbox 360. Matt Lee was showing a new DirectX SDK sample that’s coming in the next SDK release giving a reference implementation of a mega-texturing run-time. I’ve only skimmed mega-texturing papers so I got a lot out of this talk since he walked through all the steps in the run-time.

The sample shows off how you begin by creating different tiles for different resource formats. Each pool is dedicated to a different texture format. The tiles in the pool are all the same size; However the tiles may vary in size depending upon the texture format to maximize cache efficiency. When you render the scene you have a shader that can write out texture look-up failures. When the UV coordinates and mip level are not found to be resident in memory a failure is added to this list. After the shader completes you read back the failures and proceed to load the tiles that will fit in your established pools.

Unlike most texture streaming systems you’re not loading an entire mip level or the entire mip chain of the texture. You’re only ever loading into the tiles a sub-region of a texture (like a 64×64 pixel region), which overcomes one common texture streaming problem, texture memory fragmentation. Because the tile pools you create are never deallocated you don’t have to worry about fragmenting your texture memory because of different sized textures being streamed in and out.

Now the sample is not without its short comings, but that is mostly due to hardware limitations. Ideally the virtual texture system would be transparent, you wouldn’t need to write a shader that recorded look-up failures. The GPU and DirectX would simply report when a failure occurred and allow you to handle it. Maybe some day…

Gesture Detection Using Machine Learning – Claude Marais

If you have ever been interested in machine learning this is a worthwhile presentation to check out when the slides are posted. Claude Marais talked about a case study they performed to try and use machine learning to detect a Punch and a Kick. For their experiment they used Adaboost which is a machine learning technique that combines thousands of weak classifiers that ‘boost’ each other and provide you with a high degree of accuracy in the results.

The classifiers are all extremely simple things, for example you may have a classifier like:

if (elbow_joint_angle > ANGLE)
    return 1;
return -1;

Then simply create a macro and have 180 variants of this classifier one for each ANGLE. If you can imagine all the different things you could measure about the skeleton, creating simple variants of the kernels for each of the possible test cases will explode the number of weak classifiers you have; Claude had around 21,000 weak classifiers for his system.

The training phase looks at labeled data sets to know what examples of punches look like (positive examples) and what -not- punches look like (negative examples). It uses the +1/-1 scores each weak classifier provides to determine the weights to apply to each classifier. After it has determined the best weak classifiers to detect a punch and not detect a negative example as a punch on accident you can use the classifiers at run-time with the weights applied to detect a punch.

The results were undeniable; they had a demo setup the the expo area that was really good at detecting a punch and kick.

The only real drawback to this solution is the data collection; they needed something on the order of 70,000 examples of punches and 7x that in negative not a punch examples before the training produced accuracies over 90-95% from the chart they had; if my memory is correct.

In training the system they had 70,000 frames worth of recorded training data.  The actual number of recorded punches used to train the system was 25 different people doing 10 punches, so around 250 punch examples.  Then they had about 7x that number in negative training examples, which might be things like waves, or other actions that SVM can use to differentiate between random movement and an intentional punch. (Thanks to Claude for clarifying this)

Kinect and Kids: Pitfalls and Pleasantries – Deborah Hendersen

If you had asked me to make a Kinect game for kids (ages 3-6) before seeing this presentation I likely would’ve designed something with a dumb-me as the target audience. What I quickly realized is how wrong I would’ve been to make that assumption. At that stage of development kids are not capable of interacting with games I’m used to playing.

Something as simple as a menu of options is an impossibility since they are illiterate. How many games have you seen that you could play without knowing how to read?

When interacting with an onscreen character, the kids ignore social norms of waiting for the person to finish talking. They may just jump the gun if they already know what is expected and get frustrated if they can’t do it when they want to.

Kids are distracted very easily and will make their own games out of game behavior. Deborah mentioned one story where a kid stopped playing the game because he realized he could get the game to react to leaving the play area and Kinect could no longer detect him the game would do something. So he made up his own game of jumping in and out of the play area to activate this condition; utterly boring for adults, completely entertaining for this kid.

You almost have to design the game like passive experience like a children’s TV show. Where on TV because there is no feedback, the TV show host asks the kid, “Can you find _______?” and the kid at home says something, and expecting this the show simply pauses while he waits for the response. The game has to function in essentially the same way, regardless of the kid participating in the expected fashion the game has to move forward. If it functions like a state machine that requires proper actions to move forward the kid may become bored and simply want to move on. If the game refuses to let them move on, they’ll just walk away.

I really enjoyed this presentation because it was very clear how difficult the problem space is and it was interesting to hear how they tried to solve each one.

Kinect Hands: Finger Tracking and Voxel UI – Abdulwajid Mohamed and Tony Ambrus

This presentation was broken into two completely different parts, the first part was on finger tracking with Kinect. This is one area I’ve been playing around in for awhile so it was interesting to see someone else’s attempt to solve the problem. Because the Kinect is a structured light depth camera you don’t necessarily have depth at each pixel like you would on a time of flight depth camera. Structured light cameras build a topology of depth using the light pattern it projects into the scene, viewed from a different angle it can discern depth, but a single dot does not give you depth. It connects groups of them when determining the depth of a surface. This means that even though your hand can be seen by Kinect, the further you back away from the sensor, the more like a mitten your hand becomes. The gaps between your fingers disappear until they are just clumps on your wrist.

Because of this limitation you can’t go past 10 feet, there simply isn’t enough data. Ideally the user is at 6 feet or closer, past 6 feet the accuracy begins to break down.

The way Microsoft tackled the problem was to first capture lots of hand examples and then to train an SVM (Support Vector Machine) against a curvature analysis of the hands. So once you know all the pixels that make up a persons hand you find the points on the hand that result in the largest changes in curvature. On an open hand these curves are your fingers and if you’re close enough to the camera that it can see the gaps between fingers it’s a very large change in curvature. A closed hand has more or less a uniform curvature change viewed from any angle. By training the SVM against a set of closed hand curvature examples vs. open hand curvature examples they were able to get pretty accurate results at about the 6-8 foot range for an adult, 5-7 feet for kids.

Because the detector is instantaneous i.e. it can tell you in a single frame is the hand open or closed, you need some something to counteract a single/couple misinterpreted frame. So they trained an HMM (Hidden Markov Model) on examples of a flaky transition where the system is quickly switching between 2 states because the hand is at an odd orientation confusing the SVM; I thought it was an interesting solution to the problem. I’ve only ever tried something simple like requiring 3 contiguous frames of agreement to have a state change.

The second half of the presentation was on a 3D (not stereoscopic) UI for Kinect. One of the problems with navigating a ‘push to click’ interface is that it’s hard to correct for user drift. When a user pushes forward they may do several things,

• Push toward the TV
• Push toward the sensor
• Push forward (wherever forward happens to be at that moment in time)

Depending upon what you’re expecting them to do there’s going to be drift away from the thing on the screen they are trying to click. To attempt to correct this Abdulwajid presented a UI where the hands are visualized as voxelized clumps of boxes in a 3D environment with 3D buttons that could be mashed. Seeing the hand in the same space as the button appeared to make it much easier to perform the click.

One thing I noticed that was not called out was his use of 2 directional shadow casting lights. By having 2 directional lights facing each other both casting shadows, the resulting effect is a focal point. As the hand gets closer to a surface the eye perceives the two shadows heading towards each other and can see the point where they will meet. I thought that was and additional powerful indicator of where your hand was moving in the space and made it much easier to correct drift.

Trip Report: Gamefest 2011 – Seattle @ nickdarnell.com

Ode to an Unsung Hero

1
2
3
4
5
6
7
8
9
10
11
12
13

struct VS_Output
{  
    float4 Pos : SV_POSITION;              
    float2 Tex : TEXCOORD0;
};
 
VS_Output VS(uint id : SV_VertexID)
{
    VS_Output Output;
    Output.Tex = float2((id << 1) & 2, id & 2);
    Output.Pos = float4(Output.Tex * float2(2,-2) + float2(-1,1), 0, 1);
    return Output;
}

What does it do? Well, let’s pretend we’ve bound the shader and are using it to draw a single triangle:

1
2
3

pImmediateContext->VSSetShader(fxaaVS, ..., ...);
pImmediateContext->IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY_TRIANGLELIST);
pImmediateContext->Draw(3, 0);

The shader’s only input is the vertex ID, a system-generated value that starts at zero and increases for every new vertex. So, the shader will be invoked three times with IDs 0, 1, and 2. This produces the following output:
ID=0 -> Pos=[-1,-1], Tex=[0,0]
ID=1 -> Pos=[ 3,-1], Tex=[2,0]
ID=2 -> Pos=[-1,-3], Tex=[0,2]
If we clip the resulting triangle into homogenous clip space (-1..1 along each axis), we see that it just barely fills the XY plane, and that the texture coordinates range from [0,0] in the upper-left corner to [1,1] in the lower-right. Aha, a full-screen quad!

So what? Full-screen quads are easy! What makes this one interesting is that the shader is completely self-contained, with zero dependencies on external data. No vertex buffer, no index buffer, no constant buffers, no transformation matrices, and no unusual render state. Just bind the shader, draw a single triangle using auto-generated indices, and you’re done. It doesn’t even rely on any fancy HLSL-specific language features, so you could easily concoct a similar shader in Cg/GLSL/etc. Neat! It’s almost as easy as the old immediate-mode days, before every draw call had to wrangle multiple buffers in GPU-mapped memory.

Digging (Arguably) Too Much Deeper

Well, the vertex ID is a full 32-bit integer, and we’re only using the lowest two bits. The D3D11 Draw() function allows you to specify the ID of the first vertex to draw. Since we’re already not using the ID to index into memory, why not pack the viewport’s width and height into the high 30 bits?

1

pImmediateContext->Draw(3, ((viewWidth&0x7FFF)<<17) | ((viewHeight&0x7FFF)<<2) | 0);

Unfortunately, this doesn’t work. No matter what starting index you pass, the SV_VertexID you see in HLSL always starts at zero. There’s probably a damned good reason for this, but for now it’s ruined my day. There’s also the possibility that you’ll run into an over-eager graphics driver that will try to pre-fetch some ridiculously far-away vertex even though you’re not using it, and probably end up triggering a segmentation fault.

There is another option: in the pixel shader, we could use the ddx()/ddy() functions to compute the screen-space partial derivative of the incoming texture coordinates. Since our UVs range from 0 to 1 across the screen, the partial derivative in each axis is exactly the inverse viewport resolution! But now you’re doing an extra bit of completely redundant ALU work in every single invocation of the pixel shader; on one hand, ALU is cheap (and getting cheaper), but most likely this is still too steep a price to pay for a bit of extra elegance and CPU-side simplicity.

[Update: As readers have pointed out in the comments, HLSL Shader Model 4+ provides a mechanism for querying the dimensions of a texture (via the GetDimensions() method), as well as applying an integer offset to texture reads (via the optional offset parameter to Sample()). In many full-screen passes, the source texture has the same dimensions as the destination viewport; if so, you're golden!]

That’s it; nothing earth-shattering, just a clever little nugget.

The Problem at Hand

For those of you that might be reading this and haven’t drank the graphics programming Kool-Aid, I’m going to fill you in a little bit as to why we care about anti-aliasing.

In our line of work, the end result of a rendered scene is typically a 2-dimensional array of colors that is displayed on the user’s monitor. Just like how audio (which is originally analog), is quantized when it is made digital, the image is built into a discrete number of pixels during the process of rasterization (i.e. filling in each triangle). The measure of how many pixels are used to display the image is of course known as the resolution, which is why things become increasingly blocky as you lower a game’s resolution, due to there being fewer dots per inch.

Even at high resolutions, individual pixels can still often be identified by the viewer, because the lines and edges of triangles form distracting “jaggies”. This is noticeable along the outline of an object being displayed due to rasterization, but can also occur on interior surfaces for other reasons such as shadows and texture map resolution. The user typically picks the resolution their monitors are set at, but most console games provide 720p or 1080p images to the TV, and monitors tend to follow suit (my laptop is set to ~720p). Other than relying on hardware to pack more pixels into the same physical area by increasing the dots per inch (which is something that Apple is claiming with it’s retina displays, but check out this post for an interesting look at that), we have to find ways to smooth the transition between pixels. Perhaps the simplest strategy to dealing with aliasing is to render into a texture twice as large as the target resolution and then downsampling it to actual output resolution. This allows you to take the average of every four pixels, softening out the boundaries by averaging the edge colors together. However, this also just straight up sucks for performance. You end up processing 4x as many fragments from the higher resolution, and you have 4x the memory usage for the buffer. This is way too high a cost to pay for some smooth edges.

The Hardware Option

There is a hardware accelerated method, MSAA (Multi-Sample Anti-Aliasing). This operates by computing additional samples when rendering the frame instead of rendering additional pixels and downsampling. The fragment shader is only run once for each group of samples. The problem is that it doesn’t work with the increasingly popular deferred rendering techniques, because you can’t really take more samples of a buffer you’ve already rendered. In short, the damage is already done, the data has been discretized to a particular resolution. Furthermore, I’ve always found that MSAA is still pretty expensive (but then again, the only way to make your rendering take 0 ms is to quite doing rendering and switch to a job in finance).

New Maps of AA-land

The desire to use deferred rendering has pushed alternate forms of anti-aliasing to get a lot of attention (I mean who doesn’t need more acronyms, right?). Perhaps the most prevalent that you may have heard of is MLAA, but most of the techniques that I’m referring to here are post-processing techniques that rely on detecting and softening edges. By doing anti-aliasing as a post-process, it will work seamlessly with deferred rendering, and pretty much anything else for that matter. If the technique only needs the color buffer access, then it can even be applied to video/screenshots/whatever of existing games, which I think researchers in this area have found as a great method of showing off how their work.

MLAA was originally a CPU based technique developed by Intel, that has since then been adopted to be done on the GPU. This has been outlined in GPU Pro 2, Game Developer Magazine, and around the net, but if you’re not familiar with it I’ll make a few brief points about it. It essentially boils down to storing edges in a texture with different colors indicating which side of the pixel the edge is located at, and an additional buffer is used to calculate the blending weights for the blurring.

Impressively, they found results falling somewhere between 4x and 8x MSAA, while being 11x faster than 8x MSAA. The memory footprint of the technique is 1.5x or 2x the size of the back buffer depending on the hardware (2x for the Xbox 360 for all you console devs who probably have already heard everything I’m saying). That’s pretty good for something that solves the problems encountered with deferred rendering at the same time. This is undoubtably why the technique has garnered so much attention, and I really recommend the article in GPU Pro 2 if you want a clear view of all of the details.

Enter: FXAA

As you may know from reading other posts, my big side-project/hobby/thing is that when I give a new technique a try, I do it in Unity because it a) is usually not straightforward and requires actually understanding the technique to get it working and b) can be evaluated using the many projects I’ve done in Unity previously. From looking over the details of MLAA, I knew it would probably take a full weekend to get it right, and I had been procrastinating quite a bit with getting around to doing it.

When the third iteration of FXAA rolled out I decided to take a look at what it entailed. I knew in the back of my mind that FXAA was a strictly luminosity based technique, which is interesting to me. The authors of MLAA recommend using depth to determine edges for best results and performance. However, luminosity based techniques offer the advantage/disadvantage of smoothing boundaries that exist in places other than depth, such as with aliasing on texture maps and with shadows. The downside is that this can produce results that are too blurry in places you don’t want blur, such as with text on a prop in 3D space. I once tried a very, very simple luminosity based AA filter, that resulted in too many cons (especially with blurry text) for me to seriously use. I was curious if FXAA would give me similar problems.

With the intention of just looking over the code briefly, I suddenly found myself staring at a very simple and easy to use code base offering a ton of well explained pre-processor options for target platform and quality. Being around midnight when I started looking, I quickly decided that porting the higher-quality PC version of the HLSL code to CG/Unity would be fun. There were two steps involved:

1) At the end of all the other post-processing, calculate Luminosity and slam it into the alpha channel (super easy to do).
2) Perform the FXAA pass. Porting mostly involved fixing texture look-up syntax.

No extra buffers involved. This cuts down on the extra memory needed for MLAA, and the code seemed simple enough that it would probably be pretty fast. I ported all the code within 2 hours and then went to sleep. The next morning I finished setting it up for use in Unity… and was blown away by the speed and the results. Here’s a breakdown of what I got running in the Unity editor at ~720p, using Dust, a previous project of mine, to test it. These are PNG’s cropped down at native resolution:

Shot 1: No Anti-aliasing

Shot 2: FXAA3

Shot 3: 6x MSAA

It takes FXAA3 only ~1 ms [Note: corrected from an erroneous order of magnitude type when I first posted this article] on my laptop (MacBook Pro with an NVIDIA GeForce 320M) to be completed (including the luminosity calculation), and as I mentioned, no additional memory either. I’ll pay that millisecond any day of the week for that quality of anti-aliasing. Furthermore, the blurriness on text was much more acceptable than the fast blur I had tried previously. Note that this text is not really meant to be read, but rather recognized to match the voice over so the player can understand that the voice is that of the journal’s author. I wish I could credit the sources where I put together the fast blur from, but it’s been more than a few months since I implemented/ditched it. Here’s a comparison:

Shot 1: No Anti-aliasing

Shot 2: FXAA3

Shot 3: Fast Blur

This is getting a bit more to the personal opinion end of things, but even though FXAA3 does blur the text, I feel like the fast blur makes it almost uncomfortable to look at. I get the sensation that I’m getting tested for a new prescription of glasses, and have to recognize out of focus letters. To me, that seems like a pretty good indication that your luminosity based AA technique isn’t up to snuff if you get that type of blurring. FXAA3 on the other hand seems acceptable enough that it’ll definitely be enabled in future builds of Dust that get pushed up onto my website.

What did we learn?

Hopefully we learned that FXAA is both fast and doesn’t require additional memory, and is trivially simple to implement or port. There are versions for the 360, PS3, and PC, as well as HLSL and GLSL variants. It’s a luminosity based solution, and comes with the associated pros/cons those techniques have, but I’ve found FXAA to minimize the cons. It should literally take you at most a few hours to get it up and running in your game and evaluate if it’s a good fit for what you’re doing. Furthermore, it is in the public domain (credit to the post by Eric Haines for this twitter snippet):

Although at this point, I’d the amount of quality you’re getting for the effort of implementation, it might as well be under the beer license. Finally, I would point out that Timothy Lottes is still improving on the code, and has already released version 3.9 in the time since I first touched it. You can find the latest source links and updates on his blog: http://timothylottes.blogspot.com/.

From a performance perspective, this is suboptimal, to say the least. We can never utilize multiple cores with this design, and we will be lucky to utilize 100% of a single core. In particular, vertical sync has a way of introducing long, useless pauses into our UI thread, like this:

The pedants among you may note that the GPU can cheat and help us out by moving that ‘wait for vertical sync’ pause to the beginning of the next redraw. This is true, but the problem is that we are still stuck executing serially on one thread. We have to be extremely lucky to hit 100% utilization on that CPU, and we’ll never utilize other CPUs.
Depending on the kind of game you’re building, you may be able to get some easy wins here – for example, let’s say you have enough objects that you can update them on two threads:

This is better. Now, the update work we’re doing can utilize two cores. Depending on your design, you can easily move up to 4 or 8 cores for your update code. You’ll notice that during a redraw, these threads have to sit idle, because there’s nothing else for them to do – the UI thread is using all the game state in order to draw, so no updating can occur.
There’s a common solution for this sometimes known as ‘double buffered multithreading’, where you create two ‘buffers’ that hold copies of your game state. It looks something like this:

Double-Buffered Multithreading

(In these diagrams, a red arrow indicates that one thread asks another to perform work, and a blue arrow indicates that one thread wakes another thread up from a waiting state.)

We now have two distinct threads of execution that synchronize at specific points. In the example you can see red arrows indicating where the update thread tells the render thread to do work. Because we write to alternating buffers, the render thread can take as much time with Buffer A as it wants, and we can use that time to fill Buffer B. We only have to stop and wait when we need to update Buffer A again, at which point we need to be sure the render thread is done with it.

We can utilize multiple hardware threads to handle our update and render logic in this design – the ‘update thread’ in this example could be four threads working on update logic in parallel. We can do something similar with the render thread, though the threading restrictions in Direct3D and OpenGL mean that eventually we become single-threaded again.
The ‘double buffered’ model is pretty easy to understand. I’m now going to describe an alternate model that has some distinct advantages. I’m going to call this the ‘batched model’:

Batched Multithreading

The concept behind this model is that each update is turned into a ‘batch’ full of rendering commands that can be sent to the video card. A batch contains all the information necessary to draw a frame, but does not contain any other game state. This makes it smaller than a full copy of the game state. Building a batch containing your rendering commands is actually quite simple, because it’s what already happens behind the scenes: Direct3D/OpenGL calls turn into batches of native rendering commands that are sent to the GPU.

At first glance, this model might seem worse. It’s certainly more complicated. Your game loop has become a pipeline, with three stages: Update, Prepare, and Render. In practice, however, this model works quite well. One of the key advantages is that the pipelines are independent. Preparing could take a long time, for example:

In fact, updating, preparing, and drawing can all take a long time, and things look pretty good:

With this model, each of your pipelines being independent means that correctly written code will allow each pipeline to utilize a full CPU, as long as the pipeline’s got work to do.

Performance Advantages

Even better, pipelines can run on multiple cores. Here is where the value of the ‘prepare’ stage comes in. In a normal single-threaded model, drawing a group of sprites would look like this:

If you’re observant, you’ll note that the only real input to the ‘build vertices’ stage is the location of the sprite (along with information that changes infrequently, like the size of the sprite). As a result, we can defer the ‘build vertices’ stage until later, or even do it on another thread. Even better, this results in each thread doing the same work repeatedly. When a thread is doing the same work over and over, the CPU’s instruction and data caches are better utilized, improving performance.

This model also provides some other natural advantages. For example, a common performance issue when writing Direct3D code is that changing states too frequently is expensive. If you change rendering states every time you draw an object, you pay a tremendous price for doing so and the GPU isn’t able to operate at peak performance. Using this pipeline model, you can build up a long list of drawing commands, and then sort them by rendering state – enabling you to minimize the number of state changes. The performance benefits from this can be tremendous.

Scaling Out Further

Speaking of GPUs, this model is also useful for taking advantage of the GPU to perform computation. In the sprite example above, you can trivially move the hard work from the ‘build vertices’ stage of the pipeline into a vertex shader running on the GPU. After doing so, things would look like this:

Furthermore, because each pipeline only relies on its inputs, you can take those inputs and split them into pieces and run the pipeline on multiple threads without synchronization. As long as the pipeline doesn’t hand its outputs to another thread before they’re finished, you don’t need locks and you don’t have to wait for other threads. Each pipeline pulls its inputs off a queue, processes them to produce its outputs, and shoves those outputs onto another queue. Most modern CPUs and GPUs are built in a similar pipelined fashion.

I hope this post has given you some ideas about how to scale your game up to make better use of multiple cores!

For example, let’s consider a common OpenGL error:

glEnable(GL_DEPTH); // BAD!

Of course, you and I know this is a bug (GL_DEPTH is not a valid argument to glEnable — perhaps you meant glEnable(GL_DEPTH_TEST) or glDepthMask(GL_TRUE)?). By default, this call will quietly fail, and the scene will not render correctly. If you call glGetError() at some point later, you will learn that a GL_INVALID_ENUM error occurred (…somewhere…), but that’s it. No indication of which function triggered the error, or which parameter was invalid. Weak!

With the shiny new debug output extension, something like the following message will appear immediately upon executing this misguided glEnable() call:

OpenGL: glEnable parameter has an invalid enum '0x1801' (GL_INVALID_ENUM)
[category=API_ERROR severity=HIGH id=1001]

Even better, you can put a breakpoint in your error-handling callback. When an error occurs, you’ll drop immediately into the debugger with a full call stack leading right back to the problematic API call. Keen!

Unfortunately, integrating this new functionality can be a bit of a headache. Driver support for this extension is still far from universal, and getting it working involves messing with some platform-specific windowing-system code that you probably haven’t looked at in years (if ever). On the other hand, the only code you need to modify is in the initialization stage of your application — the boilerplate stuff that gets pasted unmodified into every new project. You can write this code once and enjoy it for the rest of your OpenGL-using days. So if you’ve ever experienced the sheer unadulterated joy of sprinkling dozens of calls to glGetError() throughout your codebase to track down the source of an oh-so-helpful GL_INVALID_OPERATION error, please read on!

If I had my way, every GL_INVALID_ENUM error would be accompanied by this image.

Step 1: Is the extension available?

Before you can use the new debug output features, your drivers need to support them. For Windows and Linux developers, NVIDIA added support in August 2010 (driver version 259.31). AMD added preliminary support in May 2011 (driver version 11.5a), with full support expected in a few months time. However, AMD has their own vendor-specific GL_AMD_debug_output extension which offers near-identical functionality with a very similar interface. On certain other white, hermetically-sealed platforms which shall remain nameless, your mileage may vary. Fortunately, the absence of superior debug output is hardly a fatal error; if something goes wrong, your app can just soldier on as if nothing had happened.

The topic of querying support for OpenGL extensions and setting up the necessary function pointers could warrant a whole article by itself. The short version is that you should use a library such as GLEW to handle the gruntwork for you. In the code below, I assume that GLEW has already been initialized.

Step 2: Create a debug OpenGL context

Time to dust off some cobwebs! Even if you’ve been working with OpenGL for many years, it’s entirely possible you’ve never given any thought at all to creating an OpenGL context (the object through which all future OpenGL calls interact with your graphics hardware, behind the scenes). Prior to OpenGL 3.0, there wasn’t much reason to care. The process is tedious, platform-specific, and completely independent of anything else the application is doing — in other words, exactly the sort of code you want to embed in some larger engine or framework and never touch again.

When OpenGL 3.0 introduced a deprecation model for certain older/obsolete functionality, the context creation process became slightly more involved. You can now ask for a context which implements a particular version of the OpenGL specification, and can also request that the context should be “core” (strictly non-deprecated functionality only) or “compatibility” (deprecated features supported where possible). This additional flexibility is exposed in a series of platform-specific extensions, such as WGL_ARB_create_context (for Windows) and GLX_ARB_create_context (for Linux). These extensions also allow you to flag a new OpenGL context as a “debug context”, which is required by GL_ARB_debug_output. If you try to enable debug output on a non-debug context, at best it will silently fail; at worst, it may not even report that the extension is supported!

The mechanism for creating a debug context varies from platform to platform. I’ll cover the Windows implementation here, but the general idea should be similar elsewhere (when in doubt, read the extension spec). The full process is beyond the beyond the scope of this article; instead, I’ll focus on how to extend pre-existing context creation code to support this new functionality. Basically, somewhere in the depths of your engine there will be code that looks something like this:

HDC g_hdc;
HGLRC g_glContext = wglCreateContext(g_hdc);
wglMakeCurrent(g_hdc, g_glContext);

If wglCreateContext() is called more than once, focus on the final call; the earlier ones are temporary dummy contexts which are probably deleted a few lines later. We need to use this final GL context to load a pointer to the wglCreateContextAttribsARB() function (if available), and use it to create our new final GL context.

bool enableDebugContext = true; // This should be specified by the application
if (wglewIsSupported(“WGL_ARB_create_context”))
{
    // Retrieve a pointer to the wglCreateContextAttribsARB() function
    PFNWGLCREATECONTEXTATTRIBSARBPROC my_wglCreateContextAttribsARB;
    my_wglCreateContextAttribsARB = (PFNWGLCREATECONTEXTATTRIBSARBPROC*)
       wglGetProcAddress(“wglCreateContextAttribsARB”);
    if (my_wglCreateContextAttribsARB != NULL)
    {
        // Clean up and delete old GL context
        wglMakeCurrent(NULL, NULL);
        wglDeleteContext(g_glContext);

        // Create a new (optionally debug/core/compatibility) context
        int contextAttribs[] =
        {
            WGL_CONTEXT_FLAGS_ARB,
            enableDebugContext ? WGL_CONTEXT_DEBUG_BIT_ARB : 0,
            0
        };
        g_glContext = my_wglCreateContextAttribsARB(g_hdc, 0, contextAttribs);
        if (g_glContext == NULL)
        {
            int err = GetLastError();
            // handle error here
        }
        wglMakeCurrent(g_hdc, g_glContext);
    }
}

Two things are worth noting in the above code. First, debug contexts do carry a performance cost, so they should only be enabled in development builds. Also note the contextAttribs[] array, which is a zero-terminated list of key/value pairs specifying which features you’d like the new GL context to have. Refer to the WGL_ARB_create_context extension specification for the full list of possible attributes; for our purposes, only the debug bit matters. Set this bit during context creation, and if all goes well you’ll notice a difference in the string returned by glGetString(GL_VERSION):

3.3.10834 Compatibility Profile Context       // default context
3.3.10834 Compatibility Profile/Debug Context // debug context

Step 3: Hook up a custom error-printing callback

If you’ve come this far, congratulations! The end is in sight! All that’s needed now is to register the callback function that will run whenever the OpenGL runtime detects an error. First, let’s whip up a utility function to format the various error codes into a human-readable string:

void FormatDebugOutputARB(char outStr[], size_t outStrSize, GLenum source, GLenum type,
    GLuint id, GLenum severity, const char *msg)
{
    char sourceStr[32];
    const char *sourceFmt = "UNDEFINED(0x%04X)";
    switch(source)

    {
    case GL_DEBUG_SOURCE_API_ARB:             sourceFmt = "API"; break;
    case GL_DEBUG_SOURCE_WINDOW_SYSTEM_ARB:   sourceFmt = "WINDOW_SYSTEM"; break;
    case GL_DEBUG_SOURCE_SHADER_COMPILER_ARB: sourceFmt = "SHADER_COMPILER"; break;
    case GL_DEBUG_SOURCE_THIRD_PARTY_ARB:     sourceFmt = "THIRD_PARTY"; break;
    case GL_DEBUG_SOURCE_APPLICATION_ARB:     sourceFmt = "APPLICATION"; break;
    case GL_DEBUG_SOURCE_OTHER_ARB:           sourceFmt = "OTHER"; break;
    }

    _snprintf(sourceStr, 32, sourceFmt, source);
 
    char typeStr[32];
    const char *typeFmt = "UNDEFINED(0x%04X)";
    switch(type)
    {

    case GL_DEBUG_TYPE_ERROR_ARB:               typeFmt = "ERROR"; break;
    case GL_DEBUG_TYPE_DEPRECATED_BEHAVIOR_ARB: typeFmt = "DEPRECATED_BEHAVIOR"; break;
    case GL_DEBUG_TYPE_UNDEFINED_BEHAVIOR_ARB:  typeFmt = "UNDEFINED_BEHAVIOR"; break;
    case GL_DEBUG_TYPE_PORTABILITY_ARB:         typeFmt = "PORTABILITY"; break;
    case GL_DEBUG_TYPE_PERFORMANCE_ARB:         typeFmt = "PERFORMANCE"; break;
    case GL_DEBUG_TYPE_OTHER_ARB:               typeFmt = "OTHER"; break;
    }
    _snprintf(typeStr, 32, typeFmt, type);

 
    char severityStr[32];
    const char *severityFmt = "UNDEFINED";
    switch(severity)
    {
    case GL_DEBUG_SEVERITY_HIGH_ARB:   severityFmt = "HIGH";   break;
    case GL_DEBUG_SEVERITY_MEDIUM_ARB: severityFmt = "MEDIUM"; break;
    case GL_DEBUG_SEVERITY_LOW_ARB:    severityFmt = "LOW"; break;
    }

    _snprintf(severityStr, 32, severityFmt, severity);
 
    _snprintf(outStr, outStrSize, "OpenGL: %s [source=%s type=%s severity=%s id=%d]",
        msg, sourceStr, typeStr, severityStr, id);
}

With that out of the way, writing the callback itself is a cakewalk. In this implementation, I use the callback’s userParam argument to store a FILE pointer to which the output should be printed:

void DebugCallbackARB(GLenum source, GLenum type, GLuint id, GLenum severity,
                     GLsizei length, const GLchar *message, GLvoid *userParam)
{
    (void)length;
    FILE *outFile = (FILE*)userParam;
    char finalMessage[256];
    FormatDebugOutputARB(finalMessage, 256, source, type, id, severity, message);
    fprintf(outFile, “%s\n”, finalMessage);

}

And finally, we need to register our callback with OpenGL during our application’s initialization (after the debug context is initialized, of course). We also enable synchronous output, which forces the driver to invoke the callback immediately when an error occurs instead of at some unspecified future time.

if (glewIsSupported(“GL_ARB_debug_output”)
{
    glDebugMessageARB(DebugCallbackARB, stderr); // print debug output to stderr
    glEnable(GL_DEBUG_OUTPUT_SYNCHRONOUS_ARB);
}

Appendix for impatient AMD users

As mentioned above, AMD’s official drivers do not yet support the GL_ARB_debug_output extension. In the meantime, they do support the eeriely similar GL_AMD_debug_output extension (upon which the final ARB extension was clearly based). Both extensions require the same debug OpenGL context initialization, so it’s simple enough to include support for both in the same application; you need only provide separate FormatDebugOutputAMD() and DebugCallbackAMD() functions. I’ve provided implements below for convenience; there shouldn’t be any surprises.

void FormatDebugOutputAMD(char outStr[], size_t outStrSize, GLenum category, GLuint id,
                         GLenum severity, const char *msg)
{
    char categoryStr[32];
    const char *categoryFmt = "UNDEFINED(0x%04X)";
    switch(category)
    {
    case GL_DEBUG_CATEGORY_API_ERROR_AMD:          categoryFmt = "API_ERROR"; break;
    case GL_DEBUG_CATEGORY_WINDOW_SYSTEM_AMD:      categoryFmt = "WINDOW_SYSTEM"; break;
    case GL_DEBUG_CATEGORY_DEPRECATION_AMD:        categoryFmt = "DEPRECATION"; break;
    case GL_DEBUG_CATEGORY_UNDEFINED_BEHAVIOR_AMD: categoryFmt = "UNDEFINED_BEHAVIOR"; break;
    case GL_DEBUG_CATEGORY_PERFORMANCE_AMD:        categoryFmt = "PERFORMANCE"; break;
    case GL_DEBUG_CATEGORY_SHADER_COMPILER_AMD:    categoryFmt = "SHADER_COMPILER"; break;
    case GL_DEBUG_CATEGORY_APPLICATION_AMD:        categoryFmt = "APPLICATION"; break;
    case GL_DEBUG_CATEGORY_OTHER_AMD:              categoryFmt = "OTHER"; break;
    }
    _snprintf(categoryStr, 32, categoryFmt, category);

    char severityStr[32];
    const char *severityFmt = "UNDEFINED";
    switch(severity)
    {
    case GL_DEBUG_SEVERITY_HIGH_AMD:   severityFmt = "HIGH";   break;
    case GL_DEBUG_SEVERITY_MEDIUM_AMD: severityFmt = "MEDIUM"; break;
    case GL_DEBUG_SEVERITY_LOW_AMD:    severityFmt = "LOW";    break;
    }
    _snprintf(severityStr, 32, severityFmt, severity);

    _snprintf(outStr, outStrSize, "OpenGL: %s [category=%s severity=%s id=%d]",
        msg, categoryStr, severityStr, id);
}
void DebugCallbackAMD(GLuint id, GLenum category, GLenum severity, GLsizei length,
                     const GLchar *message, GLvoid *userParam)
{
    (void)length;
    FILE *outFile = (FILE*)userParam;
    char finalMsg[256];
    FormatDebugOutputAMD(finalMsg, 256, category, id, severity, message);
    fprintf(outFile, "%s\n", finalMsg);    
}

Further reading

• The extension specs for GL_ARB_debug_output, GL_AMD_debug_output, WGL_ARB_create_context and GLX_ARB_create_context.
• The GL Extension Wrangler Library (GLEW)
• A more in-depth GL_ARB_debug_output tutorial by Aks.