• Hierarchical Z-Buffer Occlusion Culling – Generating Occlusion Volumes
• Robust Inside and Outside Solid Voxelization

Lets start with the newest and most important information…

Update 4/13/2012

The project is now hosted on BitBucket, http://bitbucket.org/NickDarnell/oxel/overview feel free to contribute or just follow along :)

It’s a Tool Now!

Download

Oxel 1.0.0 – Win32.zip

Requirements

• .Net 4.0
• VS2010 C++ Runtime

Description

Oxel is a tool for generating occluders – primarily for use with the Hierarchical Z-Buffer method of occlusion culling.  Open an .obj file then go to Build > Voxelize to generate the proxy.  Try it out, let me know what you think.

There are some further improvements that have been made over the method that is laid out in the original Generating Occlusion Volumes post,

• Retriangulation
• Evaluating Occlusion-ness
• Filtering Polygons

Retriangulation

I went back to the CSG article Sander van Rossen and Matthew Baranowski wrote.  I noticed near the end that they recommended retriangulating the final CSG mesh because their library doesn’t handle it.

The easiest method I found to do the retriangulation was to just use David Eberly’s Wild Magic math library which can do it, see here.  Before you perform the retriangulation you need to collect all the external/internal edge loops on each surface and then merge collinear edges before performing the retriangulation.

It’s was definitely worth doing, performing the retriangulation saved about 30% of the triangles.

Evaluating Occlusion-ness

When you’re generating occluder geometry you need to ensure that the volumes you’re adding are useful enough to pay the additional polygon tax they will incur.

Oxel achieves this by measuring the number of pixels written to the color buffer by rendering the original visual mesh into the stencil buffer, and then using the stencil buffer to clip a full screen quad while a “samples passed” hardware query is performed to record the number of pixels written.  This is performed from about 64 different camera angles – then when all the samples are summed together this defines the ground truth occlusion of the mesh.

Knowing that information we can evaluate every new box we plan to add to the final occlusion proxy geometry and ask – Does this increase the coverage enough to make it worth it?.  The default threshold is set to 3%.  If a new volume does not at least cover 3% of new silhouette pixels, we don’t include it in the final occluder mesh.

Filtering Polygons

Not every game needs the occluders to be visible from all angles.  For example most racing games will never have the cars jumping so high they see on top of a building.  In those circumstances you may not want to have the occluders bothering to have polygons on the top.

So the tool offers the ability to remove Top and Bottom polygons from the final occluder mesh after everything has finished being processed.

If you decide to filter out all top polygons they’re all removed.  For bottom surfaces this has to be handled slightly differently, see below -

Imagine a bridge structure or a large over hang on a building.  You wouldn’t want the bottom portion of the occluder to be removed from the overhang since it would allow seeing into and through the occluder – because presumably you wouldn’t have double sided rendering enabled.  So to distinguish between bottom surfaces not meant to ever be seen by the player, and bottom surfaces that may be important to higher up potions of the structure, only bottom surfaces within 1 voxel’s height from the base of the mesh are removed.  (See picture above)

Work Continues

I’ve got some more ideas I need to test out but I wanted to go ahead and cut a version of where I’m at and let others take a look and give me some feedback.

One area I need to investigate next is a better way to generate the boxes.  Currently they are just expanded in all directions equally, but that’s not ideal.  I’m thinking about trying a parallel brute force method that would test all possible boxes at a specific origin point to find the best shaped box to generate from that point.

Oxel: A Tool for Occluder Generation @ nickdarnell.com

The Theme

The Ouroboros – but no explanation was given.  We just showed the jammers the image and it was up to them to interprete it.

LOTR Body Snatching

Rock, Paper, Scissors with body snatching!
Avoid or possess enemies to stay alive and win.

Low Power

In order to survive Low Power you must consume yourself (like the Ouroboros) to survive.  You control a robot whose battery is constantly draining but there are valuable energy cores scattered throughout the level that will sustain your life.  There are all kinds of dangerous environmental hazards that unless you enable your sensors/abilities (lights, shield, ground sensor, microphone) you’ll never survive.  However enabling the different systems will drain your battery even faster!

Parasite

You are a Parasite that leeches off battle ships. Enemies will constantly come in formations to destroy the ship you are occupying. Using the possessed ship, you must defeat your enemies! However, as you leech your ship it slowly dies, you will constantly need to take control of and leech of another ship. With out a ship for protection you are helpless!

Planes on a Snake

A rift in spacetime has resulted in a large number of World War II era planes getting stuck on the world snake.  Join the frequent fliers club of Ouroboros Airlines, racking up points while taking advantage of the torus nature of your new environment.

Roller Snake: The Quest for Chili Dogs

Control “Hardy the Hoop Snake, Jr., III, IV, and V” in his quest to eat as many chili dogs as possible while winding through ‘Catastrophe Canyon’

Snake Run

You are the snake’s guardian, you must ensure that balance is maintained.  Let enough creatures be eaten by the snake to ensure he doesn’t starve, but not enough that he frenzies and destroys everything.  All the while outrunning him!

Global Game Jam 2012 Wrap-Up @ nickdarnell.com

I’ll start with the worst input paradigm we tried go from there.

#5 The Faux Mouse

The Idea
First we just tried mapping the hands to the screen as if they were each a mouse or finger.  Each would control a hand like cursor and move it around the screen.  Clicks could either be performed by grasping the hands or pushing towards the screen.  The game would be played normally – except you’d click on everything using your hand.

The Reality
This input method was by far the worst.  Your hands are simply not mice, they get tired much faster and are not as dexterous.  Even with heavy filtering and snap-able buttons, the interaction is just too nuanced and motor skill intensive an operation to click on a button or object on the screen using your hands as mice.  The time it took us to play a level vs. someone playing it on the iPhone was an order of magnitude or longer a task.

When pushing towards the screen a lot of care had to be taken to deal with user drift.  When a user pushes towards the screen, they may in fact be pushing towards many possible locations.  They may drift towards the TV, or the camera if that’s their focus.  They may also consciously attempt to maintain a straight hand as they push forward.  No matter the case – a user will drift even off their intended target.

When we tried grasp detection using computer vision – again we saw drift.  When a user opens and closes their hand the volume of the hand is changed from the point of view of the camera.  The result ends up displacing the center of mass of the hand causing it to shift downward when the hand is closed.  There are solutions to this problem such as eliminating the fingers from the hand volume calculation but this is a difficult problem given the quality of the data.

In the end the drift issues prevented us from playing as well as we could with the real mouse or on the iPhone and since Angry Birds is both a game of quick victory and defeat as well as a game of precision.  Having both large problems with precision and game play progression speed – we ditched this idea.

#4 Voice

The Idea
This one is exactly what you might expect – firing the bird using your voice.  Either by saying “Fire!” or maybe… “Ca Cah!”.

The Reality
We really were not sure how this method would perform.  The voice portion was layered onto the mouse control system.  Essentially you would click on the slingshot – move your hand to aim and exclaim viciously to let loose the birds of war.  The hope was that it would alleviate the drift problems, which it did.  Without having to push your hand forward to fire the drift was eliminated from firing.

However there were other problems, there was delay in the speech recognition and often outright failure if you didn’t say the words just right.  We tried several tricks, like using multiple words to identify “Fire”.  One good way to generate that list is to take the top 10 words it mistakened someone saying fire for and add those to the dictionary as triggers for firing the bird.

Moving Away From Faux Mouse

After the failures with both #4 and #5 we went back to the drawing board.  We needed to get away from the mouse or touch device centered thinking.  The worlds are simply too different to treat it the same.  So we prototyped 3 other solutions that ended up all being better than the mouse style interface.

The ideas all stem from an understanding that when you go to port an application from a touch enabled world you need to think about 2 things primarily,

• Context
• Automation

Context – How can you reduce and scope the options to the user so that a broad array of options can be presented to the user – but with only a few usable at any given time with a small vocabulary of motions.

An example from Angry Birds is all the options the user can perform in the game:

• Fire Bird
• Activate Bird Special Attack
• Restart
• Pan
• Zoom
• Return to the Menu

We needed to find a way to contextualize these options when moving away from a mouse driven style interface.

Automation – Find the items in the application that everyone does without thinking about it and automate them.  If they aren’t relevant to game play find a way to make them irrelevant in a NUI application.

An example from Angry Birds is activating the slingshot.  You probably don’t think about it when playing the game but to fire a bird you must first place your finger on the slingshot before drawing back the bird to fire it.  While this is unbelievably trivial to the point of not thinking about it on an iPhone, it’s a huge pain in an environment where you have to get a virtual hand cursor over it, even more so if you then need to push forward to activate it.

So we needed to find a way to automate clicking the slingshot.  That way instead of clicking the slingshot explicitly it would be implicitly activated by performing some gesture to begin the act of firing a bird, that would be disconnected from the onscreen location of the hand relative to the slingshot on the screen.

#3 Arclight

The Idea
You would draw back the slingshot by bringing your hands together.  Then bring your hands apart and rotate them around your center of mass to change the firing angle on the slingshot.  Then once you’ve settled on the firing angle, bring your hands together fast to trigger the fire.

The Reality
The problem with this kind of activation of the slingshot was the drift when the hands come together.  This can be partially accounted for but it’s heuristically based and can be erroneous.  Additionally activating the bird’s special power was difficult.  You would have to choose a different kind of interaction to activate the special power which would complicate the process.

#2 Stretch and Snap

The Idea
This idea grew out of the Arclight firing system.  To attempt to solve the problem of drift, have the slingshot fire as soon as the arms reach a certain distance apart.

The Reality
With this firing system you still have the problem of determining how to activate the bird special power.  You also introduce a new problem – all birds are fired at maximum drawback.  You also need to make sure to provide the user with feedback so that they know how close the user is to the *snap*, some kind of progress firing bar.

#1 Axis Separated

The Idea
For this idea I separated the functions of the hands into distinct responsibilities.  Your left hand activates the slingshot by pushing forward (doesn’t matter where).  After a threshold is crossed the slingshot is activated, from then on an angle is calculated between the shoulder location and the left hand’s location relative to it, to produce the slingshot firing angle.  To fire the bird the right hand is pushed forward and pulled back, this sends the bird flying.  To activate the bird’s special power you again push the right hand forward.

The Reality
This method ended up working perfectly.  It doesn’t result in any drift when firing is activated. It is also easy to perform because all the motions can be performed with your arms down by your sides, reducing exhaustion in long game play sessions.

Lessons Learned

You hear it all the time but it is critically important to prototype ideas when it comes to creating Kinect controls.  They simply don’t work as well as you would like in reality as they do in your head. Here’s a demonstration video of the end result,

Creating Kinect Controls for Angry Birds @ nickdarnell.com

Anthropometry (Greek anthropos (άνθρωπος – “man”) and metron (μέτρον – “measure”) therefore “measurement of man”) refers to the measurement of the human individual.

Ask any Kinect developer what the hardest problem is developing a game or application that uses the Kinect – or any other depth camera.  The answer you’ll get most often will be creating something that works well for 95% of target users.

This is something you have to consider for all your gestures, poses, and UI interaction.

• Is this gesture too difficult for your average user?
• Does this pose require too much flexibility?
• Is this UI interaction comfortable and easy to perform?

One area that can benefit from anthropometry data is UI interaction.  When you think about UI interaction with Kinect you’ve got to picture it as a real world space (box, sphere, cylinder, other) located somewhere around the body that you’re mapping the hand position in that space to the 2D or 3D UI coordinate plane.

When determining the size and location of this real world space and how it maps user hand locations onto the UI coordinate system the largest question you need to consider is:

Where will they be most comfortable?

Generally speaking you want a space that minimizes upper arm movement – as that is much more strenuous compared to forearm movements.

However, since the 3d position we’re mapping into our 2d/3d UI coordinate system varies based upon user skeleton size we can’t choose a single set of real world dimensions that will work for everyone.  We’ll have to make educated guesses about the size and location of our real world UI coordinate frame based on size and location of the users bones.

So how does anthropometry fit in?

Because the skeleton you get from Kinect – and other SDKs can be unreliable in certain poses you often find yourself heavily filtering any kind of data you’re tracking about the user.  Especially things like the user’s arm length – which can vary dramatically over a session.

So one thing I prefer to do is use anthropometry tables to ensure a more consistent size and location and doesn’t fluctuate as much as the user’s skeleton.  Using anthropometry tables we can estimate the users arm length or hand size based on other bones in their body, bones that are more stable in your skeleton SDK of choice (Kinect, OpenNI, Iisu, Omek…etc).

You can also use anthropometry tables to estimate the size of body parts that the skeleton SDK you’re using doesn’t provide – such as the size of the users hand.

But where do you find that kind of anthropometry data?

Luckily such a resource has already been painstakingly cataloged for us by the FAA – The Human Factors Design Guide.  The HFDG was put together so that planes could be constructed so that almost anyone would fit and be able to operate anything from their seat.

The anthropometry data that’s valuable to us starts in chapter 14, page 791.  For example, these lovely tables from page 818 show the functional reach and the extended functional reach of men and women broken down by population percentiles.

Kinect, Anthropometry and You @ nickdarnell.com

If two artists touch the same level, have your tools programmers made enough of an effort to make merging possible and if possible as painless as can be?

Here’s a handy checklist,

1 – Use XML/JSON (or some other text based file format)

It’s slower, it’s bigger but it’s going to make merging possible.  If your level files are binary blobs merging without a custom tool just isn’t going to be possible.  Using XML or JSON are the simplest text based alternatives because there are already many libraries for reading and writing them.

2 – For XML – Give Each Attribute a Line

If you’re using XML, you should make sure to write out the files such that every element and every attribute receives its own line.  If you don’t do this it will make conflicts a lot more likely if two users touch the same object.

If you’re using JSON, having each attribute on its own line is the norm unless the library is attempting to keep the text compact.

Here’s a quick example, note that after separating the attributes the merge tool handles the conflict correctly while it fails to do so when they are on the same line.

Figure 1 – Attributes all on the same line

Figure 2 – Attributes on different lines

If you’re using .Net this can be achieved very easily by changing the default XmlWriterSettings with the NewLineOnAttribute property set to true.

var settings = new XmlWriterSettings()
{
    NewLineOnAttributes = true
};
 
using (XmlWriter writer = XmlWriter.Create(filestream, settings))
{
    // Write out document...
}

3 – Maintain Order

Always write out the order of the data the same.  This one is pretty easy, the only real pitfall is to make sure your data structure and undo/redo system are working correctly.  For example, if you happen to store your objects in a list and someone removes and object at the front, but when it’s undone it’s inserted at the end of the list (Even if the UI doesn’t reflect this) this could affect your output order.

4 – Don’t Add New Objects At The End

When you go to save the entities don’t write them out in the order they were created.  This will definitely result in a merge conflict.  Because it ensures that if two users edit the file and both add an entity they’ll both show up in the same location and confuse the merge tool.

One thing you could do is to sort them based on a GUID that is stored as part of the objects data.  Sorting based on GUID ensures a lower probability of collisions occurring when two users both add objects.

Alternatively a sorting based on a string containing the machine name of the original creator of the object is another idea.  It would ensure every user touching the file would be inserting to their own section of the file instead of everyone inserting to the tail.

Making Your Files Merge Friendly @ nickdarnell.com

While wrapping up my post for generating simplified occluders for Hierarchical Z-Buffer Occlusion Culling, I was pointed to a paper called Complete Polygonal Scene Voxelization. Afterwards I found time to read it thoroughly and implement it as a replacement for my existing ray casting based solid voxelization method.

The problem with the solid voxelization technique I was using previously was that it used ray casting; making it impossible to perform solid voxelization unless the mesh is watertight in addition to having no anomalies like intersecting geometry.

However, that restriction makes it an unrealistic solution in the real world because game art typically has holes in the locations players never see; such as the bottom cap on a building, which is rarely modeled.

The New Solution

The Complete Polygonal Scene Voxelization paper’s solution to voxelizing a scene is pretty clever; It applies a heuristic model to the problem of determining the inside/outside status of each voxel or octree cell. Allowing it to overcome the problem of holes and intersecting geometry making it suitable as a real world solution.

How It Works

You can download the paper and read it for yourself, but let me go ahead and summarize the algorithm for brevity’s sake so that the rest of the article makes sense.

The algorithm takes place in 3 stages:

1. Create Octree
2. Find Seed Cell
3. Propagate Seed Cell

Create Octree

First you create an octree around the mesh that continues to subdivide each cell until either the cell no longer intersects with any triangle or a maximum depth is reached. A typical maximum octree depth that will work for most meshes is 5. If the mesh has some exceptionally thin walls that you want the cells to be small enough to fill you may need to go as high as 7 or 8.

I was having some problems with the GPU AABB/Triangle overlap test I used for voxelization in the Hierarchical Z-Buffer Occlusion Culling article and so I ended up porting the Möller implementation of AABB/Triangle overlap test to C# and just used it instead.

Also if you ever need to lookup how an intersection is performed I highly recommend the gigantic matrix of intersections over at realtimerendering.com. It was a handy resource since I don’t keep the algorithm for AABB/Triangle overlap stored in my brain.

After you’ve created the octree we need to process each cell that wasn’t intersecting with a triangle to determine if it is inside or outside.

Find Seed Cell

Before we can determine if a cell is inside or outside the mesh we need to find a seed cell. The seed cell is sort of the ground truth example cell that we use to propagate its status to the other cells that it can see. The seed cell’s status is determined by rendering a cube map centered inside the cell with the near plane placed at the cell edge.

When rendering each side of the cube map, you render the scene such that all front facing polygons are blue and all back facing polygons are red. You then read back each cube map surface from the GPU and determine the percentage of red and blue pixels seen at each face.

If at least 4 sides of the cube map contain red pixels, the cell is determined to be inside the mesh.

The paper says that NO red pixels can be seen for a seed cell to be determined to be outside however I found this problematic since occasionally a red pixel can be seen just through tiny rendering artifacts.

So I feel a better solution is one like the following:

MIN_INSIDE_FACES = 4;
MIN_INSIDE_PERCENTAGE = 0.03f;
 
int cubemap_sides_seeing_inside = 0;
 
for (int i = 0; i < 6; i++)
{
    RenderCubeMapSide(i);
    float backfacePercentage = CalculateBackfacePercentage(i);
 
    if (backfacePercentage > MIN_INSIDE_PERCENTAGE)
        cubemap_sides_seeing_inside++;
}
 
if (cubemap_sides_seeing_inside >= MIN_INSIDE_FACES || cubemap_sides_seeing_inside == 0)
{
    if (cubemap_sides_seeing_inside >= MIN_INSIDE_FACES)
        cell.Status = CellStatus.Inside;
    else // cubemap_sides_seeing_inside == 0
        cell.Status = CellStatus.Outside;
 
    // Propagate cell status...
}
else
{
    // Unable to solve status exactly.
}

Where you don’t count a face as inside unless at least 3% of the total red and blue pixels are red. The percentage is just something I picked out of thin air, it feels like a number small enough to be easily overcome by any truly inside cube face, but high enough to allow me to ignore tiny artifacts.

Propagate Seed Cell

The last step is to propagate the seed cell’s status to other cells. After classifying a seed cell you need to test every unknown cell against each the depth map and frustum of each cube map surface.

You’re performing a test to see if any of the 8 corners of the octree cell when projected into the camera space of each cube face are closer than the depth value at that pixel. If it is, then then the entire octree cell likely visible.

If the cell is visible from the seed cell, then in all likelihood the cell has the same status as the seed cell. However because having holes means 2 seed cells (one inside and one outside) can potentially see the same cell you want several seed cells to confirm the status of a cell before committing to it.

So once you’ve determined a cell is visible from the seed cell you’ll increment a counter on the cell for that status. Once one of the statuses reaches a threshold, like for example 16 you’ll change the status of the cell from Unknown to whatever status counter overcame the threshold and no longer process that cell.

It should be noted that only seed cells propagate their status. Cells that you propagate to do not propagate their own status.

Repeat

After you’ve found a seed cell and propagated its status you’ll continue to repeat finding a seed cell and propagating the seed cell until all cells have a status of inside, outside or intersecting. You can rarely end up with some cells whose status simply can’t be determined so make sure your code can handle that scenario and not loop forever.

Improvements

While implementing the paper I made some additional improvements to the proposed solution. I sped up the process by taking advantage of hardware improvements to render the scene using a single pass. I also improved the conservativeness of the algorithm in situations where you’re using square voxels. When a mesh is wider than it is tall in those sitations there will be padding below the mesh; if the bottom of the mesh is uncapped it can lead to inside cells ‘leaking’ their status outside.

Single Pass Rendering

The paper was published back in 2002 and due to limitations at the time the simplest method of rendering front faces one color and back faces another was to render the scene twice and flip the winding order and the color of the triangles being rendered. However this method is slower than just using a simple pixel shader to change the color of front/back faces.

In OpenGL you can use gl_FrontFacing and in DirectX 11 you can use SV_IsFrontFace.

void main()
{
     gl_FragColor = gl_FrontFacing ? vec4(0,0,1,1) : vec4(1,0,0,1);
}

Intersect Mesh Bounds and Clip To Bounds

One problem I found is that when a mesh (like a building) is uncapped at its base but is wider than it is tall there will be several cells below the base of the mesh. Cells that will have the status of inside spread to them, even though a human could easily see those cells are outside.

So one improvement I ended up adding is that when testing for triangle intersection, you should also test against intersection against the mesh bounds. Additionally you immediately mark a cell as Outside if is outside the mesh bounds, since it simply is not possible for that cell to be inside the mesh, but don’t treat that cell like it’s a seed cell; just mark it as outside and move on.

I needed some real game art to properly test the solution so I exported a roof structure from the Necropolis map from UDK’s UTGame sample. Here you can really see the difference it makes to clip to the bounds of the mesh. Note how many additional voxel/octree cells (purple lines) are determined to be ‘inside’ because of how many backfaces (red triangles) they can see.

Figure 1 – Roof Inner Voxelization Not Bounded (Before)

Figure 2 – Roof Inner Voxelization Bounded (After)

Future Improvements

When the cube map for each seed cell is processed it’s read back from the GPU and each pixel is checked on the CPU. This is wildly inefficient when all we care about is the percentage of red to blue pixels on each face.

So that processing can be moved to OpenCL to improve the performance significantly. I would prefer to have all cells be seed cells since that makes it easier to define the rules of what cells are inside vs. outside. Allowing the cells to propagate has a higher potential to cause problems I suspect on meshes with very nasty artifacts. Giving each cell the ability to individually determine their status will be more stable and more predictable.

Currently my cube map rendering is performed in 6 passes. Shifting over to a single pass method using the geometry shader will likely add additional speed improvements, but I don’t know for sure.

If I move enough of the processing to the GPU it may allow me to make more cells seed cells (perhaps all?) and still maintain an acceptable performance footprint for tool time usage. For offline usage though this method is already very acceptable (a few seconds for an average mesh and a maximum depth of 5) even with all the CPU read-backs I’m performing.

Sample Code

I’m still working on an improved version of the Generating Occluders for Hierarchical Z-Buffer Occlusion Culling sample. So the code is a bit tied up a the moment. However the next post I do on generating the occluders will contain it, which should be soon.

Robust Inside and Outside Solid Voxelization @ nickdarnell.com

Tiled Resources for Xbox 360 and Direct3D 11 – Matt Lee

This talk was about mega-texturing in DirectX 11/Xbox 360. Matt Lee was showing a new DirectX SDK sample that’s coming in the next SDK release giving a reference implementation of a mega-texturing run-time. I’ve only skimmed mega-texturing papers so I got a lot out of this talk since he walked through all the steps in the run-time.

The sample shows off how you begin by creating different tiles for different resource formats. Each pool is dedicated to a different texture format. The tiles in the pool are all the same size; However the tiles may vary in size depending upon the texture format to maximize cache efficiency. When you render the scene you have a shader that can write out texture look-up failures. When the UV coordinates and mip level are not found to be resident in memory a failure is added to this list. After the shader completes you read back the failures and proceed to load the tiles that will fit in your established pools.

Unlike most texture streaming systems you’re not loading an entire mip level or the entire mip chain of the texture. You’re only ever loading into the tiles a sub-region of a texture (like a 64×64 pixel region), which overcomes one common texture streaming problem, texture memory fragmentation. Because the tile pools you create are never deallocated you don’t have to worry about fragmenting your texture memory because of different sized textures being streamed in and out.

Now the sample is not without its short comings, but that is mostly due to hardware limitations. Ideally the virtual texture system would be transparent, you wouldn’t need to write a shader that recorded look-up failures. The GPU and DirectX would simply report when a failure occurred and allow you to handle it. Maybe some day…

Gesture Detection Using Machine Learning – Claude Marais

If you have ever been interested in machine learning this is a worthwhile presentation to check out when the slides are posted. Claude Marais talked about a case study they performed to try and use machine learning to detect a Punch and a Kick. For their experiment they used Adaboost which is a machine learning technique that combines thousands of weak classifiers that ‘boost’ each other and provide you with a high degree of accuracy in the results.

The classifiers are all extremely simple things, for example you may have a classifier like:

if (elbow_joint_angle > ANGLE)
    return 1;
return -1;

Then simply create a macro and have 180 variants of this classifier one for each ANGLE. If you can imagine all the different things you could measure about the skeleton, creating simple variants of the kernels for each of the possible test cases will explode the number of weak classifiers you have; Claude had around 21,000 weak classifiers for his system.

The training phase looks at labeled data sets to know what examples of punches look like (positive examples) and what -not- punches look like (negative examples). It uses the +1/-1 scores each weak classifier provides to determine the weights to apply to each classifier. After it has determined the best weak classifiers to detect a punch and not detect a negative example as a punch on accident you can use the classifiers at run-time with the weights applied to detect a punch.

The results were undeniable; they had a demo setup the the expo area that was really good at detecting a punch and kick.

The only real drawback to this solution is the data collection; they needed something on the order of 70,000 examples of punches and 7x that in negative not a punch examples before the training produced accuracies over 90-95% from the chart they had; if my memory is correct.

In training the system they had 70,000 frames worth of recorded training data.  The actual number of recorded punches used to train the system was 25 different people doing 10 punches, so around 250 punch examples.  Then they had about 7x that number in negative training examples, which might be things like waves, or other actions that SVM can use to differentiate between random movement and an intentional punch. (Thanks to Claude for clarifying this)

Kinect and Kids: Pitfalls and Pleasantries – Deborah Hendersen

If you had asked me to make a Kinect game for kids (ages 3-6) before seeing this presentation I likely would’ve designed something with a dumb-me as the target audience. What I quickly realized is how wrong I would’ve been to make that assumption. At that stage of development kids are not capable of interacting with games I’m used to playing.

Something as simple as a menu of options is an impossibility since they are illiterate. How many games have you seen that you could play without knowing how to read?

When interacting with an onscreen character, the kids ignore social norms of waiting for the person to finish talking. They may just jump the gun if they already know what is expected and get frustrated if they can’t do it when they want to.

Kids are distracted very easily and will make their own games out of game behavior. Deborah mentioned one story where a kid stopped playing the game because he realized he could get the game to react to leaving the play area and Kinect could no longer detect him the game would do something. So he made up his own game of jumping in and out of the play area to activate this condition; utterly boring for adults, completely entertaining for this kid.

You almost have to design the game like passive experience like a children’s TV show. Where on TV because there is no feedback, the TV show host asks the kid, “Can you find _______?” and the kid at home says something, and expecting this the show simply pauses while he waits for the response. The game has to function in essentially the same way, regardless of the kid participating in the expected fashion the game has to move forward. If it functions like a state machine that requires proper actions to move forward the kid may become bored and simply want to move on. If the game refuses to let them move on, they’ll just walk away.

I really enjoyed this presentation because it was very clear how difficult the problem space is and it was interesting to hear how they tried to solve each one.

Kinect Hands: Finger Tracking and Voxel UI – Abdulwajid Mohamed and Tony Ambrus

This presentation was broken into two completely different parts, the first part was on finger tracking with Kinect. This is one area I’ve been playing around in for awhile so it was interesting to see someone else’s attempt to solve the problem. Because the Kinect is a structured light depth camera you don’t necessarily have depth at each pixel like you would on a time of flight depth camera. Structured light cameras build a topology of depth using the light pattern it projects into the scene, viewed from a different angle it can discern depth, but a single dot does not give you depth. It connects groups of them when determining the depth of a surface. This means that even though your hand can be seen by Kinect, the further you back away from the sensor, the more like a mitten your hand becomes. The gaps between your fingers disappear until they are just clumps on your wrist.

Because of this limitation you can’t go past 10 feet, there simply isn’t enough data. Ideally the user is at 6 feet or closer, past 6 feet the accuracy begins to break down.

The way Microsoft tackled the problem was to first capture lots of hand examples and then to train an SVM (Support Vector Machine) against a curvature analysis of the hands. So once you know all the pixels that make up a persons hand you find the points on the hand that result in the largest changes in curvature. On an open hand these curves are your fingers and if you’re close enough to the camera that it can see the gaps between fingers it’s a very large change in curvature. A closed hand has more or less a uniform curvature change viewed from any angle. By training the SVM against a set of closed hand curvature examples vs. open hand curvature examples they were able to get pretty accurate results at about the 6-8 foot range for an adult, 5-7 feet for kids.

Because the detector is instantaneous i.e. it can tell you in a single frame is the hand open or closed, you need some something to counteract a single/couple misinterpreted frame. So they trained an HMM (Hidden Markov Model) on examples of a flaky transition where the system is quickly switching between 2 states because the hand is at an odd orientation confusing the SVM; I thought it was an interesting solution to the problem. I’ve only ever tried something simple like requiring 3 contiguous frames of agreement to have a state change.

The second half of the presentation was on a 3D (not stereoscopic) UI for Kinect. One of the problems with navigating a ‘push to click’ interface is that it’s hard to correct for user drift. When a user pushes forward they may do several things,

• Push toward the TV
• Push toward the sensor
• Push forward (wherever forward happens to be at that moment in time)

Depending upon what you’re expecting them to do there’s going to be drift away from the thing on the screen they are trying to click. To attempt to correct this Abdulwajid presented a UI where the hands are visualized as voxelized clumps of boxes in a 3D environment with 3D buttons that could be mashed. Seeing the hand in the same space as the button appeared to make it much easier to perform the click.

One thing I noticed that was not called out was his use of 2 directional shadow casting lights. By having 2 directional lights facing each other both casting shadows, the resulting effect is a focal point. As the hand gets closer to a surface the eye perceives the two shadows heading towards each other and can see the point where they will meet. I thought that was and additional powerful indicator of where your hand was moving in the space and made it much easier to correct drift.

Trip Report: Gamefest 2011 – Seattle @ nickdarnell.com

I ended up coming up with a solution that I thought would be good to do a post on, since there doesn’t seem to be much out there on SWIG and these types of problems.

Some Background

For those unfamiliar with why it’s difficult, let me explain.  When your API returns a native pointer to be wrapped, of lets say class Foo, and your function returns Foo* but the instance it returns is actually the more derived version Bar.  Sadly SWIG has no way of knowing this, so the object it creates in C# would be a C# Foo class.  Therefore, if you tried to cast the C# Foo to the more derived C# Bar you’d be unable to because as far as .Net is concerned the instance of Foo is just a Foo and nothing more.

Alright now that you understand the problem, lets talk about how we can solve it.

How We Can Solve It

The solution to the problem can be solved in essentially 3 ways,

1 // Walk Away

Simply don’t do anything that would require it.  Don’t expose classes that users will ever need to downcast to for any reason.  If this option is available to you, take it.  This however means that your base class needs to support all the functionality a user will ever need, something not always possible or desirable.  But if you can, do it, because once you get to script land, down casting becomes a expensive process.

2 // Manual Cast Wrapper

Write or generate C++ functions for every downcast that you would like to perform and SWIG those functions.  This would allow you to take object Foo as a parameter, returns Bar, does the dynamic cast in C++ and returns the pointer to Bar.

This option is manpower intensive and is truly brute forcing a solution.  It also causes potential holes where nasty aliasing occurs and can become a real problem if the source object is ever deleted.  Imagine an API with a factory function that returns object Foo, but Foo is actually Bar due to the way the factory works.  You need to access a Bar specific only function on the object so you downcast Foo to Bar.

Now in C++ these objects are the same object, but SWIG doesn’t know this, so two completely different C# objects are created one wrapping the native pointer to Foo, and one wrapping the native pointer to Bar.  If the SWIG layer was properly authored, the factory function notes that the object returned was created with new memory and thus SWIG is responsible for deleting the object.

So we call our Bar specific function and we decide to hang onto Bar and we let Foo on the stack pass out of scope, because why hang onto it, it’s the same object as Bar…right?  But Foo was responsible for managing the memory for the common native object they both point to.  So the underlying native object is destroyed when the garbage collector runs next after Foo is no longer referenced and now our Bar object points to totally bogus memory.

3 // Custom Solution

The solution I ended up coming up with requires you to have a custom RTTI system that allows you to check if an object is a specific type by name.

The first step is to extend the C# wrapper for your base class with RTTI information.  If you don’t have a base RTTI object this will be a bit harder.  The reason we have a new m_castedSource member variable is to solve the aliasing problem.  After the cast we can store the source here so that we never have to worry about the garbage collector accidentally cleaning up memory we’re still using.

%typemap(cscode) YourBaseRTTIObject
%{
    internal YourBaseRTTIObject m_castedSource;
 
    public bool IsKindOf<T>()
    {
        // Use your C++ RTTI system to test if this wrapped C# object's
        // native object is a 'typeof(T).Name'
    }
%}

The next step is to create a function that can create the C# method needed to factory the C# object. We could use reflection, but that method is slow and we can do a far better job by emitting IL at runtime to specifically initialize one type and just cache those created methods.

private static Dictionary<Type, Func<IntPtr, bool, object>> constructorCache =
    new Dictionary<Type, Func<IntPtr, bool, object>>();
 
private static Func<IntPtr, bool, object> CreateConstructorDelegate<T>()
{
    // We need to first grab the reflection information for the more derrived type we're casting to.
    // SWIG has protected constructors we'll be able to call that take 2 parameters, the native pointer
    // to the type and whether or not SWIG owns the memory
    ConstructorInfo ctor = typeof(T).GetConstructor(
        BindingFlags.Instance | BindingFlags.NonPublic, null,
        CallingConventions.HasThis, new Type[] { typeof(IntPtr), typeof(bool) }, null);
 
    // Lets emit some IL that will allow us to construct our type much
    // faster than using reflection.
    DynamicMethod dm = new DynamicMethod("Create", typeof(object),
        new Type[] { typeof(IntPtr), typeof(bool) }, typeof(T), true);
    ILGenerator il = dm.GetILGenerator();
    il.Emit(OpCodes.Ldarg_0);
    il.Emit(OpCodes.Ldarg_1);
    il.Emit(OpCodes.Newobj, ctor);
    il.Emit(OpCodes.Ret);
 
    // Create the delegate for the dynamic method, which will further improve the calling speed.
    return (Func<IntPtr, bool, object>)dm.CreateDelegate(typeof(Func<IntPtr, bool, object>));
}

After that we need to add a function that can perform the down cast. This is just a matter of checking if the object is actually the desired casting type. If it is we get or create the factory delegate for the type. We construct the new wrapper object and hand it the pointer for the type. (NOTE: This is only viable if the new type isn’t offset in the vtable, as would be the case with multiple inheritance).

public static T CastTo<t>(YourBaseRTTIObject self) where T : YourBaseRTTIObject
{
    if (self == null)
        return null;
 
    // Check if the object is actually the type we're trying to cast to.
    if (self.IsKindOf<t>())
    {
        Type type = typeof(T);
 
        // Check if we've already cached the factory function.
        Func<IntPtr, bool, object> factory;
        if (!constructorCache.TryGetValue(type, out factory))
        {
            factory = CreateConstructorDelegate<t>();
            constructorCache.Add(type, factory);
        }
 
        // Call the factory function and set the casted source to the
        // current object.
        T castedObject = (T)factory(YourBaseRTTIObject.getCPtr(self).Handle, false);
        castedObject.m_castedSource = self;
 
        return castedObject;
    }
    return null;
}

SWIG: Casting Revisited @ nickdarnell.com

Sadly it never went past stage one.  It’s harder than you might think to hold a T-pose for 3 minutes while someone circles you twice snapping pictures at 10 degree intervals.

I don’t have any pictures of the results; I came out looking like the elephant man.  We’ll probably try again at some point, but in the meantime I made another scan of a pair of static objects that was turning out pretty good until I got to the back of the monkey.

All in all Autodesk’s Photofly software is pretty cool.  It’s still lacking in the area of iteration and debugging.  You can try manually tagging photo matchup points between images to give it a better idea on how the images fit together but it takes awhile for the data to be processed in the cloud.  It’s also unclear where some data comes from, or why portions of the background become part of the foreground mesh.  If it had better feedback for how that data became part of the mesh cleaning up the results would be a lot easier.

Also, if you own a camera with a sports video mode that captures at 60 FPS you can just slowly circle the subject and then dump all the frames using ImageGrab.  Which is way easier than snapping individual pictures.

I wonder if I could generate 3d art for a game jam…

Project Photofly Experiment @ nickdarnell.com

I couldn’t find a service online that did everything I wanted.  The only one I found with any promise was Google Docs, but they have all kinds of size restrictions and the PDFs uploaded aren’t searchable.

So I ended up cobbling together a solution that works for me and wanted to share the solution with everyone else.

The Cobbling

I started with a Media Wiki based web application because I want to be able to access this library anywhere.  I already had a bunch of domains lying around I could use and was already paying for hosting.  Media Wiki also has a huge number of extensions people have written for it that were invaluable in accomplishing my end goal.

The main page is a combination of two extensions,

• WikiCategoryTagCloud – Lets me display all my categories as a cloud on the main page, letting me quickly find something from a particular event.
• DynamicPageList – Displays the giant list of files I’ve collected.  They are organized by when they were uploaded, so finding something I uploaded recently is a breeze.

I also made some modifications to the default limits on my wiki’s upload so that I could upload the fairly large PDF files and PowerPoint slides.

One crucial goal I had was to make the text in PDFs and PowerPoints searchable.  For that I used the FileIndexer extension.

I could have stopped there but ended up finding a extension that would render the contents of the PDF directly to the browser window if for some reason I didn’t have a PDF reader or was on a slow connection and wanted to look at a single portion of a large PDF.

For that I was able to use the PdfHandler extension, which has a dependency on the WebStore extension.

One Year Later

A year after cobbling the pieces together I’m still pretty happy with the solution.  It hasn’t failed on me yet and I have constant access to my library no matter where I am.  I wish I could package up the relevant pieces so others could just setup their own whitepaper repository, but I haven’t checked the redistribution rights of all the relevant bits.

How I Organize Research @ nickdarnell.com

Almost a year ago I wrote a couple of posts on Hierarchical Z-Buffer Occlusion Culling (HZB-OC).

• Hierarchical Z-Buffer Occlusion Culling
• Hierarchical Z-Buffer Occlusion Culling – Shadows

One very glaring issue I left largely untouched was the workflow issue.  Artists currently have to author all or a large percentage of the volumes by hand in the implementations I’m aware exist.  Ideally these simplified occlusion volumes would be generated by a tool in the art pipeline instead.

A few weeks ago I started looking for a new side project and decided to try and flesh-out a very rough idea I had for automatically generating occlusion volumes.  This is still a work in progress but I wanted to share my current progress because I think it has some value even at this early stage.

The Problem

An ideal occlusion volume has some important features,

1. Conservativeness – Doesn’t extend beyond the surface of the mesh
2. Simplicity – The occlusion volume is made of very few triangles or is fast to render
3. Volume Conservation – Closely matches the original mesh’s volume
4. Dynamic – Some games have large movable occluders or destroyable walls

Normal methods of simplifying a mesh such as typical triangle simplification causes problems in both the area of conservativeness and volume conservation.  In some cases you can use the physics collision mesh if available.  One thing to be aware of is that when the physics volume is larger than the visual mesh it can cause false occlusions leading to things popping into view.  Also not every mesh needs a physics volume nor are physics volumes always meshes.  Will Vale’s talk from GDC 2011 covered using physics collision meshes pretty thoroughly for HZB-OC, you should check it out.

The Technique

Let me start by summarizing the technique I’ve been developing for generating the occlusion volumes before I go in-depth into each step in the process.

1. Find all the voxels completely inside a mesh
2. Find the voxel at the densest point in the volume
3. Expand a box from this point until all sides collide with the mesh surface or another box
4. Repeat 2-3 until you’ve consumed X% of the total volume
5. Filter small or useless boxes (Unimplemented)
6. Use a Constructive Solid Geometry (CSG) algorithm to merge the boxes you create

1. Voxelization

First you have to determine all the voxels completely inside the mesh.  That way we can have complete confidence that anything we generate contained inside these voxels will be conservative.  It also gives us a very easy way of quantifying the total volume and the volume remaining.

The voxelization algorithms I ended up using comes from this paper Fast Parallel Surface and Solid Voxelization on GPUs [Schwarz 2010].  I used the 3.1 section for surface voxelization and the 4.1 section for solid voxelization.  Both are required because a voxel is considered part of the solid volume if the center of the voxel is inside the mesh.  We want the entire voxel to be inside the mesh though.  So you have to remove the excess voxels from the solid set using the surface set.

My first implementation was in C++ and entirely on the CPU which took about 20 seconds to run on a mesh with ~10,000 triangles and a 50³ voxel volume.  In the version I’m presenting here I moved to C#, when I did that time went up to about 60 seconds.  So I ended up porting a similar version of the CPU implementation to the GPU using OpenCL (Cloo), on the same data it ran in about 6 seconds on my Nvidia 540M, with no attempt to optimize.

One unfortunate limitation of Schwarz’s solid voxelization algorithm it that it requires watertight meshes.  The reason for this is you’re shooting a ray down a column of voxels until you intersect with a triangle.  You’re then xor’ing the bit for each voxel signaling that this column of voxel’s is inside or outside.  Voxels that are inside a mesh will be Xor’ed an odd number of times, where as outside voxels will be xored an even number of times.  So if there is a hole in the mesh, you’d leave a incomplete trail of xor’s in that column, leaving every voxel from the last triangle to the bounding box marked as ‘inside’.

Two papers I’ve run across that might contain solutions to this problem are An Automatic Hole-Filling Algorithm for Polygon Meshes and Complete Polygonal Scene Voxelization.

2. Find The Highest Density Voxel

In this step you’ll find the voxel that is furthest away from any external area, and excluding any voxel that has already been enclosed by a box in step 3.  Because the number of empty voxels likely far exceeds the number of voxels touching an empty voxel, you’ll probably want to test against that list instead of determining the distance from the closest empty voxel.

For my project this was implemented on the CPU, but this is probably something you could do faster on the GPU if this ever became too slow for your needs.

3. Box Expansion

Once you’ve found the densest voxel you’re going to create a 1³ voxel sized box at that location.  You’ll then proceed to iteratively expand each side of the box in voxel space until you can’t expand any side of the box without entering an empty voxel or another box that has already been placed.

As you verify each expansion of the box you’ll mark the enclosed voxels in such a way so that the next time you choose the densest voxel you can exclude any already enclosed in a box.

Currently the expansion order is uniform, however, I suspect a better approach would be some small amount of prediction allowing me to grow one side more than another to maximize the potential volume of each box. 

The best implementation of this AABB expansion methodology would be to use an optimization solver, but that could potentially take quite awhile to run.

I’m still looking for alternatives to the AABB box expansion method of generating the simplified geometry. The axis aligned boxes aren’t the best for models at non-right angles.  Perhaps a better approach could be divined with OBBs but that’s just conjecture.

I briefly considered the Iso-Surface method Xi Wang mentioned in his Automated Level of Detail Generation for HALO: REACH talk at GDC 2011 but that generates a lot of triangles and a volume larger than the voxel volume you start with. So that just gets us right back to the problem we started with.

4. Repeat 2-3

I ended up having two different stopping conditions.  If an absolute percentage of the remaining volume is met I stop adding new boxes.  Alternatively if the last box created consumed too small a percentage of the total volume that also stops the box expansion process.

Some other things I didn’t implement but that could be tried are, setting a maximum number of boxes, or a box too small (in voxel space, instead of percentage) as a cutoff point.

5. Filter Boxes

This portion of the technique I haven’t tried but makes a lot of sense depending on your stopping condition.  If several small boxes are generated before one of the cutoff conditions is met during step 4, you’ll probably want to ignore them before moving to step 6 to reduce your occluder triangle count.

6. Merge Boxes With Constructive Solid Geometry

Drawing individual boxes even as a single draw call would cause lots of overdraw.  Instead we should take our collection of boxes and combine them into a single mesh before we finish.  To do this we’re going to use Constructive Solid Geometry (CSG).

Soon after I realized I was going to need CSG (or something akin to it) to solve this problem I was informed of a new book, Game Development Tools.  In that book there is an implementation for Real-Time Constructive Solid Geometry written by Sander van Rossen and Matthew Baranowski.  The implementation is pretty slick and I was able to get it up and running pretty quickly.

The Results: Time-Lapse

The box expansion runs in a fraction of a second, but I slowed it down and broke it up over several frames so that you could see how the occlusion shape evolves.

The Results: Different Models

To get an idea of the results on different models I’ve provided some screenshots.  The settings I used for these was to cutoff after 90% of the volume had been consumed by boxes.

Notes

In situations where the technique doesn’t work very well allow an artist to specify a custom mesh.  In some cases an option to use the same mesh for both the visual and the occluder would probably be a good idea (like for planes).

To reduce draw calls you will likely want to generate the occluders during export time.  Then during the cook stage in your level editor, merge some number of the occluders together to reduce draw calls.

Download

Voxelization Occlusion Generator.zip License: MIT

Thanks

Thanks to Michael Noland and Shaun Kime for letting me bounce ideas off them.  Also thanks to Stephen Hill for telling me about Rossen / Baranowski’s Constructive Solid Geometry implementation and for giving me feedback on this post.

Hierarchical Z-Buffer Occlusion Culling – Generating Occlusion Volumes

I don’t want to show you how to add a command button or toolbar to Visual Studio, there are plenty of examples out there for that. I want to provide you with an example showing how writing a Visual Studio extension could benefit your studio’s workflow.

Andy Firth wrote an article on #AltDevBlogADay a few days ago on Extending the Watch Window in Visual Studio via autoexp.dat. It’s a nice introduction article to modifying the autoexp.dat file to auto-expand and format variables in the watch window.

Now, if I take a bunch of time designing a nice formatter to speed up my debugging sessions how do I share it across the company?

Well, the only way is for you to manually merge other peoples additions to the AutoExp.dat file into your own. Which inevitably leads to varying levels of adoption of the formatters and debugging times suffer as a result.

Lets fix this!

We’re going to write an extension to patch the AutoExp.dat file with one collocated with the solution file.

Getting Started

After downloading the Visual Studio 2010 SP1 SDK  you’ll create a new solution using the Visual Studio Package template.  You could have also chosen the Add-in template.  Though the packages are basically just as easy to install in Visual Studio 2010 and they offer much more control should you end up needing it.

The code you’ll start out with will be a bit more verbose than this, but here’s the important bit to get started playing with different capabilities.  Visual Studio is based on a loose coupling service architecture, so to accomplish anything useful you’ll need to find the the service that handles it.  Luckily Microsoft has documented the List of Services.  If I had to pick the “main” service it would be the DTE service.  It manages the majority of the functionally you’ll be interested in playing with.

public sealed class MyPackage : Package
{
    private DTE m_dte;
 
    protected override void Initialize()
    {
        // Your package is also a service container
        IServiceContainer serviceContainer = this as IServiceContainer;
        // The DTE object contains most of the goodies you'll want to play with
        m_dte = serviceContainer.GetService(typeof(SDTE)) as DTE;
        // I could also add a service of my own if I wanted
        // serviceContainer.AddService(typeof(MyService), new MyService(), true);
    }
}

The AutoExp.dat Patcher

Because we’ve decided to look for an AutoExp.dat patcher file beside the solution, lets hook some events so that we know when the solution is opened and closed.

// Must hold a reference to the solution events object or the events wont 
// fire, garbage collection related
m_solutionEvents = m_dte.Events.SolutionEvents;
m_solutionEvents.Opened += SolutionOpened;
m_solutionEvents.AfterClosing += SolutionAfterClosing;
 
// Most of the events you'll find useful are catagoried under 
// DTE.Events...

Now when the solution is opened we’re going to scan for the patcher file in the same directory as the solution file and we’re going to use it to modify the AutoExp.dat file.

void SolutionOpened()
{
    string slnFile = m_dte.Solution.FullName;
    string solutionDirectory = Path.GetDirectoryName(slnFile);
    string pathfileName = Path.GetFileNameWithoutExtension(slnFile) + ".autoexp";
    string patchFile = Path.Combine(solutionDirectory, pathfileName);
    if (File.Exists(patchFile))
    {
        string autoExpPatch = File.ReadAllText(patchFile);
        string autoExpPath = Environment.ExpandEnvironmentVariables(
            "%VS100COMNTOOLS%\\..\\Packages\\Debugger\\autoexp.dat");
 
        string blockBegin = "; BEGIN AEP [AutoExpand] (" + pathfileName + ")";
        string blockEnd = "; END AEP [AutoExpand] (" + pathfileName + ")";
 
        // What follows is a bunch string parsing code that's not relevant to the article.
        // (1) We remove the previous auto expansion block we added, if we added one 
        //     in another session.
        // (2) We insert the new auto expansion patched section from our 
        //     SOLUTION_FILE_NAME.autoexp file.
        // (3) Save the new AutoExp.dat file.
    }
}

Because our extension isn’t triggered by a command and doesn’t have a UI we need to make sure Visual Studio loads our extension from the start.  You accomplish this by adding the ProvideAutoLoad attribute to the Package class, with the right GUID identifying the time to load the extension.

// --- Microsoft.VisualStudio.VSConstants.UICONTEXT_NoSolution
[ProvideAutoLoad("ADFC4E64-0397-11D1-9F4E-00A0C911004F")]
public sealed class AutoExpPatchPackage : Package
{
   //...
}

Results

In our test solution I’m going to start without any patch file and this is what I see when hovering over the test variable.

After we add the patch file though and reload the solution we see something completely different.

[AutoExpand]
 
TestStruct =x=<x,g>, y=<y,g>, z=<z,g>

Voilà we’ve created a simple extension to Visual Studio that could actually improve workflow.  We can now check-in our patcher file into source control, allowing us to share our rules with all of our fellow employees.

The extension I wrote only handles the [AutoExpand] section of the AutoExp.dat file.  With a little bit of work though you could easily handle the other sections.

You could also add some other improvements such as removing the patched portions after the solution is closed to make sure there are no conflicts between versions, if you work on multiple versions of the codebase.

Another improvement would be to automatically re-patch the AutoExp.dat file if you detect a change to the patcher file while the solution is open.

Download

You can download the source code for the AutoExp.dat Patcher here.

Resources

• Visual Studio 2010 SP1 SDK
• Visual Studio Extensibility Forum
• MZ-Tools: VSX Articles / Resources
• Visual Studio MSDN
  • Samples
  • List of Services

Extend Visual Studio! @ nickdarnell.com

I’m still debating if it’s worth investing time into my own cleaner/purpose built wrapper generator.  I’d like to hear some thoughts from others who have had to deal with this decision and what you chose.  If you went with writing your own, did you use any helpful libraries (like for parsing C++)?  Also, feel free to add to the this list in the comments if you’ve run into another notable problem you ran into with SWIG.

1 ) Consistent Generator Configuration

Using different build configurations for code generation is a bad idea.  Even though your generated code will compile using different build configurations, the #defines you set on the swig command line (-DMY_DEFINE) should be consistent in all build configurations.  Varying the generator configuration can lead to things like stale files (unless you rebuild every time), as well as an inconsistent wrapper interface.

2 ) Automatic Macro Expansion Is Evil

If you have a macro that does different things based upon Debug or Release SWIG will pick the one matching the -D command line define and expand the code to match that.  It won’t just insert your macro as you would if you were writing the code naturally.  The best way I’ve found to prevent this from happening is just block off the macros it’s trying to expand.

#if !defined(SWIG)
    #define MYNEW ...
    #define MYDELETE ...
#endif

3 ) Const Correctness

C# does not have a way of mimicking C++’s const’ness.  So objects returned in C++ from a function as const Type& are treated as a regular pointer.  Allowing your wrapper user to try and change the underlying data in the object even though it’s suppose to be const.  To avoid this, you should probably remove the function entirely.  Alternatively you can treat it as a new object and generate a new C++ object on the heap and copy it there so that any edits made to it wont be reflected in the actual object.  But that has many negative sides also, like not being able to see changes in the object the reference was for.  You also create a memory leak since SWIG has no idea it has created a new object, so you need to find a way to tell swig you created new memory.  To create a wrapper class for it you can use the %typemap(out) macro like this,

%typemap(out) const Vector3& %{ $result = new Vector3(*$1); %}

4 ) Return Reference

By default SWIG treats TYPE & as a pointer.  Which is in some sense good and in others horrible.  Like it makes perfect sense for a List class to have a function called Get(int index) that returns the TYPE& to the object in the list.  In a case like that, treating it as a pointer is fantastic.  However, suppose you have the following C++ class,

class Matrix
{
    Matrix();
    Matrix& Identity();
}

Then in C# you do the following,

Matrix4 wrappedMatrix = new Matrix4();
return wrappedMatrix.Identity();

Creating the Matrix4 from C# sets the flag that SWIG needs to delete the underlying native Matrix4 instance when this object goes out of scope.

Next we call Identity() which using the default C# generated code will return a pointer to the same native Matrix4 instance, but a new managed C# wrapper class will be used to wrap it.  This new C# class will have the flag set to NOT clean up the memory since it’s just holding onto a pointer and wasn’t responsible for creating it.

Finally we return the C# object created using the Identity() method, which is a different instance than the one wrappedMatrix points to, even though they both point to the same native class.  Then we return from the scope we are in, wrappedMatrix is garbage collected, the Identity() spawned matrix continues to live on in managed code, but is now pointing at a completely bogus location in memory since wrappedMatrix was responsible for deleting the native object and did so.

So how do you solve this one?  The easiest way is the same as #3, safe bet is just to not have those functions wrapped, instead provide functions that return void.  Ideally, SWIG would provide some way for me to markup the class and say,

SWIG_THIS Matrix& Identity(); // SWIG_THIS doesn't actually exist.

SWIG_THIS would indicate that I’m just returning the same object, so instead of creating a new wrapper C# class when you return from the wrapped Identity() function, just return "this".

5 ) Customizable Allocation

There’s no standardized way of changing the allocation mechanism SWIG uses.  It will create/destroy arrays using stock malloc and free.  It will create/destroy everything else using new/delete.  The way around this is to modify the SWIG script files it uses to generate the default code. (swig\Lib\*.swg,*.i).  In particular, carrays.i for changing array allocation.  For object creation, you’ll have to modify swigs scripts for your destination language of choice.  For C# you can find them here (swig\Lib\csharp\csharp.swg).

6 ) Alignment Fail

SWIG and aligned data types don’t get along together.  Presumably you have your own allocator and have either A) Have overridden global new/delete, B) Created a common base class with new/delete functions, or C) Created a macro to use on every class you want to override new/delete on.  If you have then in the case of object creation, you’re safe.  Though you may want to replace SWIGs use of new/delete anyway with your new/delete macros as I am sure you’d like to make sure you track exactly where those allocations came from.  If you have any arrays SWIG creates, you’ll definitely need to replace their usage of malloc and free with one going through your allocator so that you can handle alignment.

7 ) Nested Types

SWIG doesn’t handle nested data types.  I haven’t found a way around it, you just have to pull them out of the parent class if you want them wrapped.

8 ) Where Are The Out/Ref Parameter Flags

SWIG doesn’t map method parameters that are passed by (non-const) reference as "out" or "ref" parameters in C#.  The way around this is to do the following,

%include "typemaps.i"
 
%define %TypeOutParam(TYPE)
    %apply TYPE& OUTPUT { TYPE& };
%enddef
 
%TypeOutParam(bool)
%TypeOutParam(int)
%TypeOutParam(float)
...

That will make all instances of those types being used in the context void MyMethod(int& x, int& y) would be mapped to void MyMethod(out int x, out int y) in C#.  Alternatively if I had done,

%include "typemaps.i"
 
%define %TypeRefParam(TYPE)
    %apply TYPE& INOUT { TYPE& };
%enddef
 
%TypeRefParam(bool)
%TypeRefParam(int)
%TypeRefParam(float)
...

The method void MyMethod(int& x, int& y) would be mapped to void MyMethod(ref int x, ref int y) in C#.

9 ) IntPtr == void*

For some unknown reason SWIG does not map void* to anything useful in C#.  So I came up with the following to map it to the IntPtr struct in C#.

%typemap(ctype)  void * "void *"
%typemap(imtype) void * "IntPtr"
%typemap(cstype) void * "IntPtr"
%typemap(csin)   void * "$csinput"
%typemap(in)     void * %{ $1 = $input; %}
%typemap(out)    void * %{ $result = $1; %}
%typemap(csout)  void * { return $imcall; }

10 ) string[] –> char**

For C# string[] is not mapped to char**.  So here is how you map it,

%typemap(ctype) char** "char**"
%typemap(imtype) char** "string[]"
%typemap(cstype) char** "string[]"
 
%typemap(csin) char** "$csinput"
%typemap(csout, excode=SWIGEXCODE) char**, const char**& {
    int ret = $imcall;$excode
    return ret;
  }
%typemap(csvarin, excode=SWIGEXCODE2) char** %{
    set {
      $imcall;$excode
    } %}
%typemap(csvarout, excode=SWIGEXCODE2) char** %{
    get {
      int ret = $imcall;$excode
      return ret;
    } %}
 
%typemap(in) char** %{ $1 = $input; %}
%typemap(out) char** %{ $result = $1; %}

11 ) Operators

Operators are not wrapped by default.  There is probably a way to map them to operators in C#, but I went with another route. You can rename them so that they are, for example

%rename(Add) Vector3::operator +;

12 ) Memory Lifecycle Management

SWIG has this internal flag in generated classes that tells it if it needs to clean up the memory for the native object when the instance is garbage collected by .Net.  If you have pesky life-cycle management requirements in your system you’ll need to be able to control this flag on the fly.  The way I’ve found to do this is with the %typemap macro,

%typemap(cscode) YOUR_CLASS
%{
    public bool IsLifecycleManaged
    {
        get { return swigCMemOwn; }
        set { swigCMemOwn = value; }
    }
%}

Now you can control on a per instance basis whether or not .Net garbage collecting the object will clean it up.

13 ) Callbacks

If you need to do callbacks use the "%feature("director")".  The documentation explains it more in depth, but it wasn’t clear to me at first how to do it until I learned about directors.

14 ) Extension Macro

You can extend/add functionality to the C++ class prior to the wrapper code getting generated using the "%extend" macro.  The extension code is added as a C function though, so you might wonder, how do I access the class that this function is supposed to be part of.  The code SWIG generates refers to the ‘this’ pointer as the variable ‘self’ so you can access anything public on the class by just doing

self->...

15 ) Unfriendly Template Class Names

SWIG generates unfriendly names for templated classes.  The best way around this is to use the %template macro.  It works like this,

%template(ListFloat) List<float>;

16 ) Tracking New Memory

If a function creates a new object and returns it, you need to tell SWIG this, otherwise you’ll have a memory leak.  By default SWIG does not destroy any object it wraps unless it created a temporary object in the heap for a value type on the stack.  The way you inform swig a method returns a new object is by doing,

%newobject MyFactory::Create;

17 ) Casting

As it stands I don’t have a good solution to this one. I thought I should mention it though, because it has been a real thorn in my side. If you have a method in C++ that you wrap and you return a base class like BaseObject*. SWIG will wrap that inside a BaseObject C# class. Now all the other SWIG wrapped classes that derrived from BaseObject will exist and will similarly inherit from a C# BaseObject.

However, attempting to cast your new C# BaseObject into anything other than its parent classes will result in an exception. Because SWIG has no way of knowing the true type of the BaseObject* it returned from the function. So the C# object it creates is going to be BaseObject and not the actual C# wrapper class for the native types true type.

SWIG and a Miss @ nickdarnell.com

The first time I stepped in front of a depth camera was almost a year ago now. We had a reference version of a PrimeSense camera that is heavily related to the final hardware that went into Kinect. The first thing I got to do was make a stick figure guy move around on the screen. It was very captivating to see him match my movements, even with the occasional arm through my chest Kali-Ma style.

Those first days were filled with lots of experimentation because everything was new to us in this world of full body motion gaming. Which reminds me…if you ever want to see a cool effect, go grab a large mirror and hold it in front of a depth camera at an angle; now you’re really playing with portals!

Introduction

With all the time I’ve spent around these cameras I wanted to capture some thoughts on some problems developing games and software driven by full body motion input.

Unnatural User Input

Lately I’ve come to find the statement “Natural User Input” a bit of a misnomer. There are still many technological and human hurdles that have to be overcome with time and good ideas before the interaction is truly natural. The problem with natural is that it’s different for everyone, which generally forces you to make it unnatural for some group of people. Also with the limitations of the current technology you will often find yourself making unnatural concessions to make something work.

A great example of this is getting detected by the camera, often referred to in the office as “Doing the Truffle Shuffle”. Some skeleton SDKs require a pose or gesture to be detected as an active user. For example, OpenNI has the “Psi” pose. Some ask you to wave your hand. Some just work, like Kinect but even so many games have logic layered on top about when a user can join that is highly varied and currently unnatural because there isn’t one consistent way yet.

Another good example of this is turning. If I asked you to turn, how would you do it?

Q: Would you naturally turn, away from the TV?
A: No, then you couldn’t see the TV.

Q: So would you turn your whole body and continue to face the TV, or just your shoulders?
A: If you turn your body naturally you’ll occlude half of your body, making it harder to detect other actions simultaneously (Walk + Turn). Also many skeleton SDKs have varying levels of success tracking shoulder angle and occlusion of the shoulder usually causes them to move around.

Q: What about if we let the hands determine turning, moving them left to move left, right to move right?
A: Good in some contexts, like skiing and horseback riding. It’s very unnatural when walking around. It also prevents you from using the hands to do other things at the same time. It’s also very hard to hold for long periods of time if you have to keep them there.

Q: How about leaning left to turn left, leaning right to turn right?
A: It’s great from a technological standpoint. It won’t ever occlude any part of the body. Very easy to do for all users. Very easy to hold for long durations. Can be combined with many other actions. However, it’s completely unnatural.

The best advice I can give here is to get people to test out your ideas. I can’t tell you how many times I’ve thought to have solved a problem only to see a tester or coworker break it almost immediately. If you can help it, find new people to try out the game. We refer to them as untrained users around the office. For these systems you’ll find that over time the system trains you back. You learn just the right movements without thinking about it, which will lead to a false sense of improvement in your gesture detection code.

I haven’t seen it happen yet, but I suspect many motion games in the future will actually ship with multiple ways of handling the same input and users will select the one they prefer. In the same way we have inverted controls and different control schemes.

Noise

The cameras are not perfect and they’re mapping a physical space to some finite number of pixels. Surfaces that poorly reflect infrared, other infrared sources (like the sun), and even the manner in which the cameras define a contiguous surface can cause variations from frame to frame leading to lots of jaggy shifting edges on objects. This jitteriness influences the volume of an object and thus the calculated positions of bones in a skeleton are shifting too.

So you’ve got to find a way to smooth out the data without adding lag to the propagation of player movements onto the character. The best way we’ve found is with a predictive filter. They average in old frames with the current frames data, but are simultaneously predicting N (usually 1) frames forward in time. The only drawback is they end up over and undershooting the actual curve of motion because it’s predicting the motion is going to continue in the same direction. Luckily this largely goes unnoticed by users.

Generally Avoid

The amount you should avoid each of these varies across cameras and skeleton SDKs, but generally speaking this is my own list of things you should try to avoid.

• Small Motions – Detecting them is very difficult, they are very easy to confuse with noise.
• Holding hard poses – It’s hard to hold your arms out for extended periods of time.
• Motions near the body – Occlusion problems, bone loss.
• Fast motions – Most of the consumer grade depth cameras right now are running at 30 FPS. It’s very easy to move faster than the segmentation / skeleton prediction code is willing to bet you’ve moved and will happily ignore your motion.
• Extreme poses – Poses most people would have trouble making. Not just because people have trouble making them, but because most of the skeleton SDKs are not trained for unusual body positions.
• Sitting – It’s is generally not handled well across skeleton SDKs. The overall skeleton becomes a lot less trustworthy.

That’s Normal Right?

All the skeleton SDKs I’ve used so far don’t generally return you anything other than the rawest of the raw bone positions. Which is generally a good thing; you wouldn’t want them to hide the raw data from you. However, this will tends to result in moments when your hand will penetrate your chest, your knee will flip backwards and you’ll have your leg behind your back.

So it becomes important to try and avoid these events by using joint constraints. Even though the skeleton SDKs usually have bone confidence numbers, they’re not comparing confidence based upon how a normal human can move. It’s based on can they clearly see something they think is a body part. If so they will report things like, 100% confident your leg has driven itself up into your chest.

Time

Timing is very difficult. The user has to predict how long he is going to take to move, while at the same time accounting for how long the avatar will take to move, plus how long the gesture detection will take to detect his action. Making it very hard for him to predict when he has to jump or duck or move to the side.

In these situations feedback that he has done the right thing, as well as how long he has left to do the right thing can be important. One handy trick when compressing timespans to play back animation is Bullet-Time. Imagine a player running and jumping hurdles. There’s this unknown zone that once entered there will not be enough time to playback the animation to jump the hurdle without it looking sped up bizarrely fast. However with bullet time, if you detect the gesture just in time, you can slow down time long enough to play back the animation and also indicate to the user, “Hey, you almost missed that one”. Bullet-Time is also handy for just giving the user more time to make a split second decision, and then as soon as they’ve made it, speed back up.

Just a Little Bit Closer…

Depth perception is another frustrating problem. Users have really poor perception about how far away objects are from their avatar that they can interact with. Luckily there are many ways around this problem.

• Depth Cues – shadows do a great job of helping to show distance as you get closer to an object
• UI Visual Cues – Visual feedback that you can now interact with the object is important. If I’m playing a volleyball game, changing the halo around the ball from red to green to indicate I can now jump and hit it can be valuable feedback, because it’s hard estimating how high my character can jump, or when they can jump.
• Camera angle is everything. Having the right angle to the object can make it much easier to tell depth.
• Audio Cues – I don’t see these get used very often, but sound is a great way to indicate action is required, or success or failure on the user’s part.

Do the Truffle Shuffle to Start @ nickdarnell.com / activate3d.com

One thing shader authors routinely take advantage of is the fact that they can do some simple work to filter out data points in the vertex shader by placing them outside the frustum.  Giving them a cheap and effective way to remove all that data that would otherwise go to the pixel shader or show up as a branch if this were all happening in a single shader.  By being able to have multiple stages the shader cores are all able to stay busy instead of several of them idling from a discard branch.

Which brings me to the main question of this blog post,

How can I chain compute shaders to consume variable amounts of data to duplicate this same staging capability?

So here is our scenario, you have a FilterComputeShader (FilterCS) and a HardWorkComputeShader (HardWorkCS).  The FilterCS will use the AppendStructuredBuffer to output some amount of data less than or equal to some incoming amount of data.  Now, you place one or more branch statements in your FilterCS that way by the time you get to the HardWorkCS you’ve got all your shader cores being utilized and no branching there (hopefully)

After appending some amount of data onto this buffer, how will we know how many threads or thread groups to spawn after the FilterCS runs?

ID3D11DeviceContext::CopyStructureCount

You can use it to copy the count from the buffer into some other buffer on the GPU.  Now you could then read back that value from the GPU and use it in another call to Dispatch, but forcing a sync point between the CPU and GPU isn’t a good idea.  Which is why there is this.

ID3D11DeviceContext::DispatchIndirect

You can pass that same buffer you just copied that size of the append buffer into as input into this function to subsequently spawn some number of thread groups.

For this to work, your compute shaders would all need to be defined with [numthreads(1, 1, 1)].  That way the number of threads matched the number of thread groups.

However, part of utilizing our shader cores effectively will be making sure we aren’t defining our compute shaders with [numthreads(1, 1, 1)].  Ideally the number of threads matches or is some multiple of the size of the GPU’s wavefront.  If you don’t do this you’ll be vastly underutilizing the GPU.

Crap…

So how do we get around this without adding a sync point between the CPU and the GPU?

Another compute shader! :D

RWBuffer dispatchBuff : register(u0);
 
[numthreads(1, 1, 1)]
void UpdateIndirectDispatchBuffer()
{
    dispatchBuff[0] = (uint)ceil(dispatchBuff[0] / NumThreadGroupSize);
}

I know I said [numthreads(1, 1, 1)] was bad, but here it makes sense, we’re only going to spawn a single thread group and thread to do make this quick fixup to the buffer we’re going to use for DispatchIndrect so that instead of telling it the number of threads to dispatch, we tell it the number of thread groups to dispatch, based on the number of threads in our thread group for the x thread group dimension.

So now the code flow looks like this,

• Dispatch(FilterCS);
• CopyStructureCount():
• Dispatch(UpdateIndirectDispatchBuffer);
• DispatchIndirect(HardWorkCS);

I haven’t finished the side project this is part of so I don’t know if the overhead of using the AppendStructuredBuffer + 2 additional Dispatch calls is actually better than doing it all in a single compute shader with a branch.  My current assumption is that this method will be far superior in cases where you’re filtering.

If the branch were simply doing hard work X or hard work Y, there’s obviously no win.  But given the choice hard work X or no work.  Your best bet is to be able to handle each stage in a separate Dispatch, or so I think.  The performance results of the side project this is part of will probably make for a good post.

Chaining Compute Shaders @ nickdarnell.com

The valuable nugget from the talk is the idea that you can approximate object density by inverting the mesh normals and have your DCC tool generate a map of ambient occlusion.  Which is a pretty good approximation of internal surfaces proximity to each other.  Which gives you a map of thickness you can use in SSS calculations.

My graphics side project of late has been experimenting with Morgan McGuire’s Ambient Occlusion Volume paper.  Which generates AO by creating prisms around each polygon to accumulate AO where they overlap against an opposing surface.

So of course while reading the Approximating Translucency paper I thought I could take Ambient Occlusion Volumes, flip the normals the prisms are generated on so that I could generate AO for the purpose of calculating thickness for subsurface scattering.  voilà Subsurface Scattering Volumes.

So the awesome bit here is because the thickness is calculated based on AO generated from the geometry directly.  You could use this method if you wanted an approximation of subsurface scattering to work on something that was destroyable or morphable.  Now, I doubt the applicability of this method in any immediate sense, but it’s still a neat effect.

Enough words, bring on the moving pictures!

NOTE: This is cross-posted at my personal blog, nickdarnell.com

Welcome to part two of my series of posts on teaching a toaster to feel love, or introduction to machine learning for the rest of us.

If you missed part one, you can read it here.

Last time I thought in part two I would be able to start posting some code.  However after writing a good chunk of the code I wanted to include in part two, I realized it was way too much to cover in one post.  So I’m going use this post to introduce some function calls and classes, but focus mostly on the process.

Download

A fairly popular SVM library is LibSVM, which you can download on this page here, or directly here.  There are several variations of that library in many languages.  It’s what I will be referencing in this and in coming posts.

Features

The first and most important step when using an SVM is to figure out what constitutes a good feature vector.  If you remember from part one, a feature is just a floating point value with importance to the programmer but is just another number in a dimension for the SVM.  The feature vector is the collection of these features, constituting a single example.

The features that your feature vector is composed of is the real secret sauce, everything else in the process is more or less mechanical.  Sadly it also means I can’t tell you what to put into your feature vector because everyone’s problem needs a different vector.  However, there are lots of tactics that apply in all cases that you should remember.

Normalization

You need to remember to normalize your feature data.  I don’t mean just normalizing the feature vector like you would a float4. I mean, each feature should be normalized either from 0..1 or –1..1 (not both).  For two important reasons, numerical stability and weighting.

Numerical Stability – We don’t want to get into situations where we might be hitting up against the 32bit limit with very large numbers during the testing process.

Weighting – We also don’t want one dimension to greatly out weigh any other dimension accidentally.  Because an SVM is attempting to find the hyperplane with the greatest margin that divides the data, having one dimension in the feature vector run from 0..1000 while everything else runs from 0..1, would mean that almost all your data would be classified based on that one feature because it might find a hyperplane that gave it a margin of 500 in that one dimension while every other feature’s margin added together doesn’t even approach 500.  So that feature more than any of the others would be what determined classification.

Numericalization

Not all kinds of data that you may want to include in a feature vector exists naturally as a number.  Sometimes you need to transform a category into a feature.  Your first thought might be to divide the category number by the number of categories and voilà, a floating point number.  However, this is not the best approach.  The better approach is to represent each category like you would if you wanted to represent them as bit fields, and dedicate one dimension for each category.  So if there were 3 possible categories, you’d represent the first category with 3 features, 0, 0, 1, the second category as 0, 1, 0, and the third as 1, 0, 0.

Rephrasing

It’s important to remember that a machine doesn’t have insight.  If I gave you position and time data, you could tell me other information like velocity and acceleration.  However an SVM doesn’t know what the data is, so it can’t derive any sort of insightful observations about it.  So it’s important to remember to rephrase the same data in different ways that tell the machine new things about the same data.

Ordering

The order of the features does not influence the process, however you need to keep your ordering consistent.

Size

Your feature vectors should all have the same number of features.

Training

The second step in the SVM process is training the SVM model using the feature vectors you’ve built containing all the useful features you think will help distinguish one thing from another thing.

To train an SVM you need 2 sets of data; the training set and the validation set.  While you can use just a training dataset, it’s not advisable as it can lead to a problem known as ‘over fitting’.  Which is just a fancy way of saying, you’ve created an SVM that ONLY recognizes the training data and doesn’t handle variation very well.

The most common and easiest training methodology to use for beginners with an SVM is known as a grid search.  The process is pretty simple, to determine the best support vectors we’ll iterate over a series of parameter magic number combinations (Cost (C) and Gamma) until we find a pair that seeds the process of dividing of our data the best.  The kernel type we’ll be using to divide our data will be the Radial Basis Function (RBF) kernel.  You could devise any kind of kernel you wanted to divide the data, however there’s a bunch of existing kernel functions people tend to re-use because they work in a wide number of situations.

The Cost (C) parameter represents the penalty for miss-classification in the training process.  The higher the cost, the more rigid/strict the trained model will be for the training data.  Meaning that while you may see fewer miss classified items (for data that resembles closely the training set), the model will be much more likely ‘over fit’ the problem.

Gamma is a parameter that’s used directly in the RBF kernel function.

The first step in the training process is to create an svm_problem object.  It holds all the training data in the form of svm_nodes.  Then we define our svm_parameter, which represents the kernel type and all the other parameters used in the training and creation process of our svm_model.

To determine the best parameters in the grid search we’ll use the svm_cross_validation method that’s built into SVMLib.  Which allows us to take the ground truth training data and see how well it would be classified given the resulting model from the selected Gamma and C parameters.

Now, you don’t have to vary JUST the C and Gamma parameters you could try different kernel functions.  But the more things you vary in the grid search the longer the training process is going to take.  Which, depending on the size of your dataset may be hours.  It all depends on how many different combinations of parameters you are willing to try.

Side Note: Should you ever run into a training time problem, I highly recommend looking into ways to running the training process in distributed environment.  The training process for an SVM is VERY amenable to being parallelized.  There are also some GPU (CUDA) implementations of the SVM that I’ve read about but not yet tried.  You can always use one SVM implementation for finding the best parameters and another for runtime usage.

Once you’ve determined the best C and Gamma based on the miss-classifications from your svm_cross_validation grid search process.  You’ll just need to call svm_train to generate the final model that you can save out and re-use at runtime to classify datasets that you want the SVM to identify for you.

Once you have your model you’ll want to run your validation dataset against it to see how well it runs on data that wasn’t used to train it.  If you find that the model performs very poorly you’ve over-fit to the training data.  So you may want to save the N best C and Gamma parameters you found and actually generate N final svm_models and pick the one that performs the best on the validation dataset.

At the very least, having a validation dataset allows you to make sure there’s no regressions in the unit tests of your code.

Next Time

Next time I should be able to walk you through the code easier and just reference concepts introduced in this post.

Cross-posted to my personal blog.

As new devices are developed to interface with humans for games and other applications; accelerometers, gyroscopes, depth cameras with skeletal tracking and who knows what next.  Interpreting the input reliably has become a much more difficult task.

So I wanted to write a series of posts which hopefully will benefit anyone wanting to learn how to use one variety of machine learning; Support Vector Machines (SVM).

Introduction

An SVM makes a binary decision about a feature vector deciding if it is closer to the positive class or the negative class.  During the training process for an SVM it’s given many feature vectors that exist as examples of what kinds of data appears in the positive vs. negative classifications.

Here is a breakdown of a class,

Class
{
Feature Vector 1 { Feature 1, Feature 2, … },
Feature Vector 2 { Feature 1, Feature 2, … },
…
}

So what is a feature?  A feature is simply a floating point number that has some special meaning as far as the programmer is concerned.  To the SVM however, it’s just a number in a dimension.  So as you can see the inputs for a class are nothing more than a list of vectors filled with floats.

To determine the best best hyperplanes (support vectors) that best divide the data you have to train the SVM with example feature vectors for each class.  The trick is finding the support vectors that maximize the margin (distance) on either side of the hyperplane.  The easier it is to split the data, the less likely you are to see incorrectly classified data when you put the system to use.

So let’s talk examples, if I were to say the following feature vector { 0.25 } was the only feature vector in the positive class, and feature vector { 0.75 } was the only feature vector in the negative class, the ideal support vector would be { 0.50 } because it best divides the data, giving the greatest margin between both classification feature vectors.  Which is easy to see when you have only 1 feature and 1 feature vector in either class.  The problem becomes infinitely more difficult when you have classes of 10,000 feature vectors with 600 features in them each being used for training data.

There are several kernel types used to find the best hyperplane through the features. Linear, Radial Basis Function (RBF), Sigmoid, and Polynomial.  From the papers I’ve read, RBF seems to be quite popular.

Next Time

For the first post I just wanted to introduce the basic vocabulary.  Next time I’ll get into actually using them in practice.

Cross-posted from my personal blog.

• Specialized language
• Specialized framework (tied to language)
• Specialized operating system
• New versions would require manual deployment
• Non-commercial license

I wanted a solution that didn’t require a specialized framework or language because chances are I would find something I wanted to distribute that I didn’t want to completely rewrite. No specialized OS, I want to be able to slave any unused windows machine in the office (dedicated farms are really expensive). Also if I need to perform new operations or fix a bug, I don’t want to reinstall / deploy new versions to everyone’s machine.

So I decided to roll my own solution and share the design. I’ll share the finished version of the software once I’ve completed it.

Let’s start with a usecase. I have an executable and a pile of data on disk that I would like to distribute over a ton of machines. I was able to quickly modify the existing program to have a command line option that allowed it to process a range of the data on disk instead of all of it.  How do I distribute this work over multiple machines so that I process all the data on disk?

Each time we execute the program with some amount of data to be processed, let’s call that a task. Each task will be processed by some machine in the cloud. To submit the tasks, we will need some common / cross language mechanism of communication. I chose named pipes just because they are easy to use on windows.

To submit tasks to the named pipe, we could either write a simple reusable program that reads task descriptions from a file and submits them to the named pipe, or we could wrap the named pipe communication in a C++ library so that C++ and C# (P-Invoke) could both reuse the logic inside of many various tools (including the generic reusable program that reads task descriptions from files or std::in).

Each machine has a single server running on it. I chose to write the server in C#, but it’s a blackbox as far as your tasks are concerned so you could use something different if you wanted.

Once the server receives the task description via the named pipe it looks at the list of servers it knows about.  The serves are detected using a simple UDP broadcast.

Using .Net remoting I then connect to each one of these machines to see what it can offer me.

Now each of these machines could have who knows what installed on them, and you’re about to transfer and run an exe that may have requirements, like .Net 4.0. So each task needs to contain a list of requirements. For now, I’ve got them scoped to 3 things, defined environment variables, registry key/values and .Net version. You could probably drop the .Net version if you just always keep your server written using the latest version so it’s a prerequisite for any machine on your network.

Now I can try and reserve a slot on the machine, if I fail to reserve a slot because of a race condition I move onto the next server.

Having reserved a slot (some machines that are idle or dedicated may have multiple slots), I need to transfer the executable and all the data files listed inside the task description that are claimed to be needed by the task.

I tell the remote server to then begin running the task in a new thread and I continue looking for empty slots in the cloud to submit tasks to.

After I discover that a task has finished, everything that is different in the remote folder where the task was dumped and subsequently executed is then transferred back to the local machine that submitted the task.

The named pipe that was used to submit one or more tasks remains open the whole time and is notified after each task is finished and the data successfully transferred back to the host machine.

So that’s the design in a nutshell. It’s not a solution designed to solve every distributed computing problem, but I like that it solves a very common pattern of problems I see fairly often in a non-intrusive manner.

I haven’t nailed down yet the best way to prevent the system from being abused by a malicious user.  However, I suspect having a SQL server with the list of MD5 hashes of the executables that are approved for deployment is one idea I’ve been toying with.

Here’s a simple diagram to help explain the task submission process. Because who doesn’t love diagrams.

Also posted on my personal blog.

I’ve talked about AtomineerUtils before on my blog very briefly several months back when I first discovered it.  I’ve now been using it pretty much straight since then and I can’t live without it at this point.

So what does it do?

It takes an educated guess at what your method does based on the method name, return type and parameters and inserts a pretty good stub of documentation to get you started.

Think about it… if you write Doxygen style comments in your code, how much time do you lose to just formatting and tagging each parameter and then repeatedly copying and pasting the same description over and over for parameters that are basically self explanatory?

The current version lets your organization share the same preference files so you can all share the same rules to assist the tool generate the first stab at documenting your code.  The files that allow you to control the documentation generation are VERY customizable, my favorite feature is word expansion.  Which allows it to look for patterns in variables or method names like “MS” and expand them to “milliseconds” in the documentation.

I whipped together some examples to show you what it produces out of the box…

Before:

After:

It has a tool menu button you can use for generating the documentation, but I prefer to bind it to a key, makes it a lot easier to use. Here are the steps if you don’t know how.

Go to Tools > Customize

Click on Keyboard… in the bottom left and you can setup a key binding to the AddDocComment action, personally I use Ctrl+D.  All you need to do then to generate the doc block is have your cursor just above or on the method or class you want documented and just hit Ctrl+D.

Remember though, this is not a replacement for writing actual documentation. Don’t let yourself get into the habit of hitting Ctrl+D and moving on. AtomineerUtils will get you started, but you still need to verify and improve its results. If you find yourself correcting the same stuff over and over though, try creating a rule for it.

Happy documenting.

I’ve been party to five game jams over the last few years and I’ve made some observations of what works really well and what doesn’t.

1) The Organizer

Every Jam needs an organizer, the guy who aligns everyone and makes sure there’s a Jam Site and that that everything is ready to go long before anyone else shows up. There’s a fair amount of work that goes into a Jam, especially if you require internet access. A lot of businesses that might like or be open to hosting it tend to nix it because of the work, risk or liability issues. Depending on the size of the Jam, the organizer also needs to recruit helpers. People that have experience with Jams and can help people out that are new to the experience and possibly to the technology primarily being used in the Jam. Also, don’t forget to setup a source control server for your participants. Otherwise how will they collaborate?

2) Pick a Framework

We tend to use XNA for our Game Jams. While I’m sure you could use UDK or Unity for a Game Jam, those frameworks have a much higher barrier to entry. If you are unfamiliar with the technology, you’ll spend a lot of time chasing down tutorials and you just don’t have that kind of time in a jam. Doing an XNA game in a weekend for someone who has never touched XNA is far more achievable. But you should ask your participants, heck, I almost want to host a Jam focused around Little Big Planet 2.

3) The Schedule

Typically a Jam runs Friday to Sunday. Starting on Friday around 5-6 o’clock and ending at approximately the same time on Sunday to do presentations. We’ve found that the first night is usually not the time to start burning the candle at both ends; most people are already tired from work. You should recommend to the jammers to get into groups, talk about the game, go get some food and then do a little work on it. Go home and get plenty of rest and come in Saturday bright and early and ready to write a ton of code for 12+ hours.

4) Simple and Creative Theme

Your theme should be creative and simple. Something that allows for a lot of creativity but doesn’t completely open the players up to making whatever game they want. The global game jam themes tend to actually be my least favorite. The group organizing it tends to pick themes that are kind of vague, and actually a little hard to turn into concrete ideas for games. Last year’s theme was a little better than their first, “Deception – and you had to incorporate a net, a set or a pet”. The year before that it was “As long as we have each other, we’ll never run out of problems”. Personally, I find these kinds of themes a little annoying because they tend to turn into a challenge of “how do I morph my game idea into this vague theme”, which I don’t think is what you want.

You should really try and give your jammers something concrete to build from. For example, the local jam we do in Raleigh, NC has had themes like “Triangles” and “Blocks”. My personal favorite was Madlibs. Everyone comes to the jam with 5 nouns, 5 adjectives and 5 verbs. We then entered them into a simple website which randomized everyone’s words into game titles in the form of “Adjective Noun Verb”. So you might see game titles like “Meaty Nun Flyer”, “Musical Dragon Twirler” and “Pixelated Martini Roller” just to name a very small number of the ones generated. I like this method a lot because a title like “Meaty Nun Flyer” is evocative, immediately conjuring up possible game mechanics and a mental image. Something I find very important when I’m trying to crank out a game in 48 hours.

5) The Grouping

After the theme has been announced you should give everyone 15 minutes or so to come up with an elevator pitch for their game. If you are doing something like madlibs you skip this step. Then have each person with an idea stand up and pitch it. Have someone write down the title and a 3 word description. After everyone finishes, now comes the least fun part, the puppy killing phase. You need to reduce the number of game ideas to the ones people actually want to work on.

We tend to do this by first getting a general consensus of what people would even be interested in working on. So just go down each game title, remind them what it was and then have them raise their hand if they would want to work on it. People can vote multiple times in this phase. You should cut anything from the running that doesn’t at least have 3-4 people even interested in it (numbers vary based on the size of the jam).

After you’ve narrowed the list you should have the final grouping phase. Start by going down the titles left and asking people to vote once. Ask them to pick the title they want to work on, even if no one else wanted to work on it. People can skip this phase if they are unsure. This will give you the list of games that are going to be made. From here, anyone left undecided can just join the group they are most interested in.

6) Forget

Forget everything you ever learned about software engineering. It’s actually nice to remove yourself from the mindset of someone writing production code 50+ hours a week. Copy paste code, make everything public. I mean you’ve got 48 hours; it’s really not the time to worry about overhead, performance or maintenance.  Always take the quick and dirty path to getting the game done, you simply don’t have time to constantly refactor systems during this process.

7) Gameplay First

Get the mechanics and controls of the game working before anything else. You need to try and aim for getting the game done by Sunday at noon and leaving the next 4-5 hours to do the random bits like improve graphics (maybe add a neat shader effect) and add a title screen, all of the polish items basically. It also means you’ll be having fun on the last day instead of stressing over your game only being half done.

8) The End

Just some final thoughts I have on the subject. If you’re a student you should come to game jams. Industry folks do participate and it’s a good way to make contacts.  If not, at the very least you’ll have something cool to show a potential employer. If you’re already in the game industry, you should participate in game jams. The experience can be very invigorating to return to a simpler time when games or projects don’t stretch on for several months or years. It’s nice to be able to sit back and feel like you’ve accomplished something that is a whole instead of merely a part in such a short time.

Cross posted from my personal blog.