Motivation

This has entirely been a hobby project for me, because I do not foresee its application into any project I’m involved with. However, I’ve realized that Unity is rapidly approaching the point where someone may want to use a more complicated skin shading model than just basic wrap lighting (an example of which is provided in the documentation). With this in mind, I thought it might be of interest to do a post pulling together some of the better resources I’ve found on the topic as well as posting some of my more useful code so that perhaps it might help serve as a jumping off point for a more serious endeavor.

The one issue that is always at the core of skin shading is that of Subsurface Scattering, which is the effect of light bouncing around underneath the surface of the skin and re-exiting elsewhere. A simply using a Lambert model causes very harsh edges, because the scattering is what gives skin its softer appearance. Here’s what the built-in “Bumped Diffuse” shader looks like, the basis that is trying to be improved:

Background: Texture Space Diffusion

No discussion of skin rendering starts with anything other than mentioning the NVIDIA Human Head demo, which has a detailed description of its implementation in GPU Gems 3. The technique relies upon Texture Space Diffusion, where the lighting for the mesh is rendered into a texture and then blurred various amounts (based off of the scattering of light inside of human skin). The results of those blurs is combined together to form the actual diffuse lighting term. I actually played around with this technique in Unity around Christmas time last year, but this proved to be difficult given the nature of the TSD and the fact that Unity did not at the time easily support some nice features like Linear Space lighting.

There have been some very useful resources on skin shading in since the publication of GPU Gems 3. There are some excellent Siggraph slides from John Hable where he details what Naughty Dog did to attempt cheaper calculations than the NVIDIA techniques in Uncharted 2. There is also a technique by Jorge Jimenez detailed in GPU Pro that does subsurface scattering calculations in screen space, which removes the cost per mesh of TSD (a serious limitation for it’s use in an actual application). This seems to have garnered some adoption in game engines, but I understand that it is still a reasonably expensive technique (note: I’m less knowledgeable of Screen Space Subsurface Scattering, so take that comment with a grain of salt).

With regards to tools for doing Skin Rendering on your own time, a good quality head scan has entered into the public domain from Lee Perry-Smith and Infinite Realities. Also, Unity3d is much friendlier to doing some high quality rendering. There is now a public beta for Unity 3.5, which now features Linear Space lighting and HDR rendering conveniently built into the engine.

The Latest Hotness: Pre-Integrated Skin Shadding (PISS)

I’ve been impressed with the quality articles in GPU Pro 2 since I picked it up this past summer, one of which is Eric Penner’s article detailing his technique “Pre-Integrated Skin Shading.” He also gave a talk describing it at Siggraph, and the slides are available online. There are three main parts to it: scattering due to curvature, scattering on small details (i.e. the bump map), and scattering in shadow falloff. I did the first two, no shadows yet for me, but I’ll do a follow up post if I get around to it.

The basic motivation is to pre-calculate the diffuse falloff into a look-up texture. The key to making the effect look good is that instead of having a 1-dimensional texture for NdotL, it is a 2D texture that encompasses different falloffs for different thicknesses of geometry. This will allow the falloff at the nose to differ from that in the forehead.

I’ve adapted the sample code to precompute the lookup texture into a little editor wizard for Unity. Simply make a folder titled “Editor” in Unity’s assets and drop it in, and you’ll have it extended to the editor (the function is added to the menu “GameObject/Generate Lookup Textures”). I’ve included the full script at the bottom of this blog post, which computes both a falloff texture that appears to be about the same as the one that Penner shows in GPU Pro 2. Here’s what my lookup texture looked like when I was all said and done, this is to be sampled with 1/d in the y, and NdotL in the x:

An important note: if you activate Linear Rendering in Unity 3.5, know that the texture importer has an option to sample a texture in Linear Space. GPU Gems 3 has an entire article about the importance of being linear, I noticed that after I turned it on in Unity, the lookup texture had the effect of making the head look like it was still in Gamma Space. I then realized I needed to flip on that option in the importer. In fact you may notice that the lookup texture in GPU Pro 2 looks different than the one in Penner’s Siggraph slides, this is because the one in the slides is in linear space. The one in the book is not, which is also the case with the one pictured above.

The parameter 1/d is approximated with a calculation of curvature based on similar triangles and derivatives (there’s a great illustration of this in the slides), and I’ve included the snippet of GLSL from my shader. To see the output of curvature on my model (easily tuned with a uniform parameter in the shader to offset the calculation):

1
2

// Calculate curvature
float curvature = clamp(length(fwidth(localSurface2World[2])), 0.0, 1.0) / (length(fwidth(position)) * _TuneCurvature);

This curvature calculation is actually a serious problem with implementing the technique in Unity. This is because ddx and ddy (fwidth(x) is just: abs(ddx(x)) + abs(ddy(x))) are not supported by ARB, which means Unity can’t use the shader in OpenGL. Those of you familiar with Unity, will know that CG is the shading language of choice, and it is then compiled for each platform. A thorny issue, that I have been stumped with in the past, especially because I do many of my side projects in OSX on my laptop. Tim Cooper (@stramit) came to my rescue with a solution that works decently: to write an OpenGL only shader directly in GLSL, which will support fwidth no problem. This is a *little* painful, especially because Unity seems to have dropped the glsl version of AutoLight.cginc (my 3.4 build has AutoLight.glslinc, but unfortunatey 3.5 beta 5 does not), which is going to make things like using the built-in light attenuation methods much more of a headache. In fact, the reason my sample images use a directional light is because I don’t currently have point light falloffs matching *exactly* the same as Unity’s. That being said, it hasn’t been problematic enough to cause me to just abandon OpenGL support and move to a Windows computer. Furthermore, there’s a nice wiki that served as a good jumping off point for doing GLSL in Unity.

(NOTE: Aras from Unity posted a suggestion down in the comments that might cause my choice to use GLSL directly to be un-needed. I will properly update the post after I try it out myself)

Softening Finer Details

As I mentioned, I’m sharing my experience with trying out 2 of the 3 techniques detailed by Penner. The second part is smoothing of small detailed bumps. While the pre-integrated texture will account for scattering at a broader sense, the fine details contained in the normal map are still much too harsh. Here’s what my model looks like with just the pre-integrated scattering:

“Too Crispy” is my favorite way of describing the problem. Penner addresses this by proposing blending between a smoother normal map and the high detail normal map. The high detail is still used for specular calculations, but for the diffuse, you pretend that red, green, and blue all come from separate normals. This is very similar to what Hable details as the technique used in normal gameplay in Uncharted 2 in his Siggraph 2010 slides. By treating them separately the red channel can be made to be softer, Penner advises using the profile data from GPU Gems 3 to try to match real life.

In order to avoid the additional memory of having 2 normal maps, Penner uses a second sampler for the normal map that is clamped to a lower mip resolution (as opposed to using the surface normal like Uncharted 2). However, Unity does not allow easy access to samplers to my knowledge, so the only way to set up a sampler like that is to actually duplicate the texture asset, which won’t save any memory. So, my solution was to just instead apply a LOD bias to the texture lookup (an optional third parameter in tex2D/texture2D in case you’re not familiar). Here’s what the same shot from above looks like with the blended normals applied using a mip bias of 3.0:

Specular

While many articles on skin shading focus almost entirely on the scattering of diffuse, the GPU Gems 3 article provides a good treatment of using a specular model that is better than Blinn-Phong. The authors chose the Kelemen/Szirmay-Kalos Specular BRDF, which relies on a precomputed Beckmann lookup texture and takes into account Fresnel with the Schlick approximation. Being that I had played around with that model a year ago when I was experimenting with TSD in Unity, I simply rolled that code into Penner’s work. Here’s the resulting specular:

A shot before applying specular:

And finally the specular and diffuse combined together:

Concluding Thoughts / Future Work

This has been a fun little side project for me, and I think it turned out decently well. I’ve made and fixed a few stupid mistakes along the way, and I wonder if I’ll find a few more yet, don’t be afraid to point out anything terrible if you spot it. My two main goals left are tighter integration with Unity : proper point/spot calculations that match CG shaders in Unity, and shadow support. When it comes to shadows I’m at a but of a crossroads between trying to hook into Unity’s main shadow support vs. rolling some form of my own. I suspect rolling my own would be less useful in the effort to make the skin shader completely seamless with unity, but might be less of a headache to accomplish. Either way, if I do make make any drastic improvements, I promise I’ll do a follow up post :)

Finally, if you’re interested in the IGF project, Dust, that I mentioned as the root cause of my hiatus from AltDev, you can check out the project on it’s website at www.adventureclubgames.com/dust/, and the trailer for it if clicking a link is too much effort:

Source Code

As I promised, here is the source for my Unity script that generates my lookup textures:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125

// GenerateLookupTexturesWizard.cs
// Place this script in Editor folder
// Generated textures are placed inside of the Editor folder as well
using UnityEditor;
using UnityEngine;
 
using System.IO;
 
class GenerateLookupTexturesWizard : ScriptableWizard {
 
	public int width = 512;
	public int height = 512;
 
	public bool generateBeckmann = true;
	public bool generateDiffuseScattering = true;
 
    [MenuItem ("GameObject/Generate Lookup Textures")]
    static void CreateWizard () {
        ScriptableWizard.DisplayWizard<GenerateLookupTexturesWizard>("PreIntegrate Lookup Textures", "Create");
    }
 
	float PHBeckmann(float ndoth, float m)
	{
		float alpha = Mathf.Acos(ndoth);
		float ta = Mathf.Tan(alpha);
		float val = 1f/(m*m*Mathf.Pow(ndoth,4f)) * Mathf.Exp(-(ta * ta) / (m * m));
		return val;
	}
 
	Vector3 IntegrateDiffuseScatteringOnRing(float cosTheta, float skinRadius)
	{
		// Angle from lighting direction
		float theta = Mathf.Acos(cosTheta);
		Vector3 totalWeights = Vector3.zero;
		Vector3 totalLight = Vector3.zero;
 
		float a = -(Mathf.PI/2.0f);
 
		const float inc = 0.05f;
 
		while (a <= (Mathf.PI/2.0f))
		{
			float sampleAngle = theta + a;
			float diffuse = Mathf.Clamp01( Mathf.Cos(sampleAngle) );
 
			// Distance
			float sampleDist = Mathf.Abs( 2.0f * skinRadius * Mathf.Sin(a * 0.5f) );
 
			// Profile Weight
			Vector3 weights = Scatter(sampleDist);
 
			totalWeights += weights;
			totalLight += diffuse * weights;
			a+=inc;
		}
 
		Vector3 result = new Vector3(totalLight.x / totalWeights.x, totalLight.y / totalWeights.y, totalLight.z / totalWeights.z);
		return result;
	}
 
	float Gaussian (float v, float r)
	{
		return 1.0f / Mathf.Sqrt(2.0f * Mathf.PI * v) * Mathf.Exp(-(r * r) / (2 * v));
	}
 
	Vector3 Scatter (float r)
	{
		// Values from GPU Gems 3 "Advanced Skin Rendering"
		// Originally taken from real life samples
		return Gaussian(0.0064f * 1.414f, r) * new Vector3(0.233f, 0.455f, 0.649f)
			+ Gaussian(0.0484f * 1.414f, r) * new Vector3(0.100f, 0.336f, 0.344f)
			+ Gaussian(0.1870f * 1.414f, r) * new Vector3(0.118f, 0.198f, 0.000f)
			+ Gaussian(0.5670f * 1.414f, r) * new Vector3(0.113f, 0.007f, 0.007f)
			+ Gaussian(1.9900f * 1.414f, r) * new Vector3(0.358f, 0.004f, 0.00001f)
			+ Gaussian(7.4100f * 1.414f, r) * new Vector3(0.078f, 0.00001f, 0.00001f);
	}
 
    void OnWizardCreate () {
		// Beckmann Texture for specular
		if (generateBeckmann)
		{
	        Texture2D beckmann = new Texture2D(width, height, TextureFormat.ARGB32, false);
			for (int j = 0; j < height; ++j)
			{
				for (int i = 0; i < width; ++i)
				{
					float val = 0.5f * Mathf.Pow(PHBeckmann(i/(float) width, j/(float)height), 0.1f);
					beckmann.SetPixel(i, j, new Color(val,val,val,val));
				}
			}
			beckmann.Apply();
 
			byte[] bytes = beckmann.EncodeToPNG();
			DestroyImmediate(beckmann);
			File.WriteAllBytes(Application.dataPath + "/Editor/BeckmannTexture.png", bytes);
		}
 
		// Diffuse Scattering
		if (generateDiffuseScattering)
		{
	        Texture2D diffuseScattering = new Texture2D(width, height, TextureFormat.ARGB32, false);
			for (int j = 0; j < height; ++j)
			{
				for (int i = 0; i < width; ++i)
				{
					// Lookup by:
					// x: NDotL
					// y: 1 / r
					float y = 2.0f * 1f / ((j + 1) / (float) height);
					Vector3 val = IntegrateDiffuseScatteringOnRing(Mathf.Lerp(-1f, 1f, i/(float) width), y);
					diffuseScattering.SetPixel(i, j, new Color(val.x,val.y,val.z,1f));
				}
			}
			diffuseScattering.Apply();
 
			byte[] bytes = diffuseScattering.EncodeToPNG();
			DestroyImmediate(diffuseScattering);
			File.WriteAllBytes(Application.dataPath + "/Editor/DiffuseScatteringOnRing.png", bytes);
		}
    }
 
    void OnWizardUpdate () {
        helpString = "Press Create to calculate texture. Saved to editor folder";
    }
}

I haven’t included the actual shader because it’s a little messy still and currently only supports the OpenGL path. The numerous resources that I’ve mentioned should guide you in the right direction though.

The game that my friends made was called Yeti Tracks. The premise of the game is that your character has crashed his plane in a blizzard and you have to use your rifle to both hunt a rabbit for food and fend off a Yeti that is hunting you. You win by successfully hunting the rabbit and returning to your plane. It’s a small, unpolished student game:

What’s interesting is that indie developer Vlambeer showed off a new project of theirs at Fantastic Arcade recently. The game is titled Yeti Hunter… and it looks exactly like what most people would call a clone, but I don’t think that was Vlambeer’s intention. It’s worth noting that they say this game may never find it’s way to a formal release, but it’s certainly gotten more attention than the Yeti Tracks did. The key difference as far as I can tell, is that the point is to hunt yetis instead of staving them off, but the gameplay is eerily similar in this small, unpolished indie game:

You may have noticed Vlambeer in the news recently, their game Radical Fishing got cloned on iOS before they could release the actual iOS version of the game, Ridiculous Fishing. It seems relatively certain that the clone was intentional, and it made my blood boil when I read about the whole incident. But the Yet Tracks / Yeti Hunter similarities also made my jaw drop, and then I realized that it was highly unlikely that the developers found an obscure student game and decided to clone and polish the gameplay.

So what does that leave us? Does it make it alright? If Yeti Hunter’s existence is acceptable, then does that mean that we also have to find Ninja Fishing to be acceptable?

Or can we really have the criteria be based on what the intention of the developer was, so that the similarities in the Yeti games are alright, but an intentional fishing game rip-off is not? Using that distinction is incredibly hard to do. Consider the sciences and engineering: keeping a log of your work can be just as important to prove your work was developed originally, as it is to validate the methods used.

If we want game design to be an art form, wanting that sort of proof that an idea is original would be ludicrous, never mind that it would be infinitely harder to “verify” as well. The whole dilemma seems like a tangled mess, and the thing that really makes me sour is that the student that had the original idea for the game admitted that the existence of Yeti Hunter meant that he could never release a polished version of Yeti Tracks without people constantly making comparisons.

In fact, I wonder if he did a more public release of the game (say, on iPhone), if the community would attack him for “ripping off” Vlambeer, considering their unfortunate situation with Radical Fishing.

Olympus

After finishing out my second year of college, I spent my Summer working on what would become the longest running project I’ve ever been a part of. Olympus is a project from the MSU Games for Entertainment and Learning Lab developed to study the effectiveness of exercise in motion games in the context of a game intended primarily for entertainment.

The design was that of an action adventure game set in Greek mythology. It not only required us to support toggling everything from dialog options to whether the game was played with a 360 Controller or a Wii-mote/Dance pad combination for various research groups, but it also demanded that we deal with the problems of maintaining a codebase over the course of a long-term project.

The lessons here were invaluable. There were only ever two main programmers on Olympus at a time, with Shawn Henry Adams and myself moving into those roles as the original two graduated. It very quickly became apparent to us that the code base was riddled with quick fixes, lack of proper tools, and poor software engineering. In other words- it was made by students, and not bad students at that. The fact that students are rarely have to revisit class projects in the future is something that I think should be fully realized when doing a student based project.

I’m not sure when students are supposed to learn the life lessons of the dangers of ignoring an attempt at good software design, but I certainly learned it from Olympus. It was also the most fun I ever had making a game for the GEL lab, working with my coworkers to get through the challenges of developing hours of gameplay and adapting existing, and often messy, systems to new designs. So as a student if you have the chance to work on or take a class that involves a project that lasts at least a semester, don’t let it pass you by.

Becoming Nocturnal

When school actually started up, I was enrolled in two classes that changed me forever. One was the portfolio class for the game development track at MSU, and I threw myself into that class like there was no tomorrow. There was not an all-nighter that I wouldn’t pull for the sake of my games, and looking back now I’m pretty sure I know why I had that mentality. The number one thing that I’ve ever heard about getting a job in game development is that a stellar portfolio is essential. Worthwhile? Yes. Although, as I lamented in my last post, a passing grade is not enough incentive to have people create amazing games. For me, feeling like I’m the only one working on a project is a lot more troubling than a difficult technical challenge. I’ll always help and vouch for the kids that throw themselves at their work, because with a little direction, they’ll always do great things. But in a portfolio class, it was always frustrating to deal with kids that had already let their enthusiasm slide to the point of no longer pursuing game development seriously.

I also took Yiying Tong’s graduate level graphics programming class that Fall, and it was perhaps both the greatest and hardest class I’ve ever taken. There’s a different lesson to be learned here than working hard to make a good game: it’s worth it to go way over your head in a subject you enjoy. While a lot of it was beyond my comprehending at the time, I’ve come to learn that not being afraid of material you don’t fully understand can help you out in the long run. Maybe the first time or the second time it doesn’t make sense, but eventually, it all comes around.

On My Way Out

The Spring of junior year brought along my first serious attempt hunting for a job in the industry beyond Michigan State University’s game development research projects. It also brought about the beginning of my writing for AltDev, the beginnings of a stint writing tutorials for Shaders in Unity, and my first semester as a TA for the introductory Game Design course. This was a bit of a tipping point for me and draws my story to a close.

I never had a good gauge for whether or not I was on track to make games professionally after college. When I seriously started teaching others how to make games, and people actually seemed to be improving as a consequence of my effort, I began to realize I was probably doing alright. My hunt for a game development internship landed me at the mighty Iron Galaxy Studios, and maybe someday after what I was working on gets announced I’ll talk a little about my experiences as an intern there (spoiler: it was great!), but it certainly reaffirmed my suspicions that I had gone from someone wanting to learn game dev to someone that could competently contribute to a substantial game.

A final word on my efforts to contribute to helping others learn about game development: it’s very rewarding. If you find yourself in a position to help others out, seriously consider it, whether it’s teaching a class or writing a blog post or even giving a presentation at a local club or IGDA chapter meeting. I know that the reason I enjoy spending so much of my time trying to give back to the game development program at MSU and the development community as a whole is because a younger version of myself would have loved to be on the receiving end of any of the information I’ve shared. Many people helped me along the way to understanding what I’ve learned and experienced, it’s all I can do to continue that effort now that I’m in their shoes as one of the more experienced individuals at MSU.

Getting Paid to make a Game

During the academic semesters for my first two years of college, I got paid by a scholarship to work for the Games for Entertainment and Learning Lab (an incentive to have first and second year students involved in research labs on campus). However, part of the way through the summer leading up to my second year of college, I got pulled on to a project as a second programmer, and being paid outside of the academic year by the Lab’s actual budget was exciting for me. However, the project went anything but smoothly (and had been a mess well before I was brought on).

I’ve noticed that sometimes game devs talk about getting jaded or disillusioned after getting into game development professionally. If there was ever a project that did it for me it was this one, a serious game about power plant management. It was continually misdirected by a client that didn’t understand game design or development. The project got extended into the Fall semester as the development continued to be a mess. Eventually this culminated into the client asking us to add back as many of the features the Lab had attempted to add to make the game more fun, finally realizing that the direction they had been steering it was keeping it from being fun for the target audience.

For as unfortunate as that was, I think something really important came from this, the knowledge that programming, not game design, was what I loved no matter how badly a project was going. Level design in particular came hard for me, realizing that it really just couldn’t hold my interest like a programming challenge. I think most people can find a lot of glamour in more than one discipline of game development, and for me I thought that maybe game design was as an enjoyable as coding. Clearly I was wrong, and it proved to me that even a bad project can have a lot of value, giving me a better sense of direction of what I wanted to do moving forward.

Greener Pastures

While the power management game was wrapping up, two much more promising projects were just starting up. One was a competition hosted by Ford Credit between Michigan State University and University of Michigan’s game development clubs (Ford is headquartered in Michigan, thus the localness of the competition). The goal was to make an serious/adver game to teach potential Ford customers about car financing. The target audience was people in their early 20′s, so having college students make a Flash game seemed like a great way for Ford Credit to go about it and was probably a fun PR stunt at the same time.

While the content of the game might not sound amazing, Ford Credit was very hands off about the development, which was a breath of fresh air compared to the project I was coming off of. The real kicker though was the prize: all expenses covered for GDC with all-access passes. I threw myself into that project like there was no tomorrow, and it ended up being the first project I seriously crunched on as the team streaked towards our relatively aggressive deadline (I believe we had 3 or 4 months of development, and many of us had never done a Flash game before). It wasn’t uncommon for me to be up at 2 am rolling in features and artwork that probably should have been cut for scope reasons, and even then I was fixing bugs right up to the deadline. And when I say “right up to,” I mean that I did the submission build of the game on a laptop in the back of a van as we drove to present the game to the judges in Dearborn. You can check out the game here.

The result was that we won, and getting to go to sessions at GDC 2010 blew my mind. While I know many devs that have been in the industry go to GDC as much for the socializing as the sessions, as a student I’d say that the talks are infinitely valuable. Especially compared to the Career Pavilion, which I’d wager is what the majority of students attend GDC for. It was also then that I realized that the allure of rendering and engine code was ever so tantalizing, with John Hable’s talk about HDR lighting in Uncharted 2 convincing me to drown myself in graphics programming in my spare time. This was a big jump for someone that thought they might still want to be a game designer less than a year earlier.

Lesson here? Student competitions are important, teaching more about deadlines and quality game development than any class could, because to be honest, student projects often are only a fraction of a someone’s grade. The project for Ford Credit had that extra mile of polish that can only come from really wanting to make the best game possible. Getting in that extra stretch of polish and bug fixes before a class deadline lacks incentive because it probably won’t budge someone’s grade unless it’s worth at least half the student’s grade. I’ll be revisiting that theory of mine in my next article.

Enter: Olympus

I mentioned that their were two projects in the wake of the power plant management game. The second one was a motion controlled action adventure about Greek Mythology. The purpose of the game was to be used to study the effectiveness of aggressive motion controls in an entertainment game (as opposed to a game like Wii Fit, where exercise is the consumer’s intent).

I started as what I would probably refer to as a “junior programmer” handling basic gameplay tasks while I was still heavily involved in the Ford project and my first class about game design and development. However, I would inherit the role of leading up the player and motion control code when the original programmer graduated, continuing into the summer. I’ll pick up on that story in my next post about lessons from my third year of learning game dev.

I think very few people will disagree that it’s a lot harder to get into the games industry than it used to be, being that there are so many more people interested in getting into the industry these days. I’ve been happily interning for these past months at Iron Galaxy Studios where I do programming work commercial games, so I’d say that I at least got some important bits right. I think a lot of what I’ve done can manifest itself in a slightly different way for other prospective game developers, so hopefully these are helpful in some way to them.

Year One

When I started college, I knew I wanted to make video games for a living. However, like most college freshmen, I didn’t know how to make video games, but I did know that getting into the games industry was no cakewalk. One of my scholarships paid a stipend in exchange for working 10 hours a week under a professor, essentially a way for the university to get underclassmen involved in research without putting strain on a lab’s budget. Due to my interest in game development, I joined the MSU Games for Entertainment and Learning (GEL) Lab to work under Professor Brian Winn, but I suppose it was not the most opportune time to be a GEL Lab professorial assistant.

There was very little going on in the lab that year other than a small game design conference that we helped organize called Meaningful Play. However, beyond helping prepare for the conference, there was very little concrete game development work to hand me. Besides the lack of projects, I didn’t know a whole lot about game development. I can still remember not having a good answer about what part of developing games I actually liked doing when I first started in the lab. All I knew was that I liked programming in general from the few classes I had in High School, although game design still seemed like “the cool thing” at that point, and I thought that I would probably enjoy design more if I was given the chance (spoiler alert: programming is actually way cooler, but this won’t be discovered until year 2).

Contrary to what one might think, something very good came out of the lull of activity in the GEL. My commitment to GEL Lab was for 2 years, so Brian had me begin to teach myself how to use Unity with the hope that I’d be able to use it for future projects, being that the department had just adopted into the curriculum as its 3D engine of choice. As a result I had a conscious reason to teach myself game development, putting in at least 10 hours a week towards a small 3D project. I started with the standard tutorials, which are only really helpful for learning the menu flow. As with any first 3D game, the learning curve still felt steep even though I was using a fairly user-friendly engine like Unity. However, if you get a jolt of excitement from getting a cube to move back and forth across the screen for the first time in your life, then you know that game programming might actually be your thing. The project evolved into a small game that I presented at the end of my Spring semester.

It was an action-adventure game about a manatee. It was terrible, and my code base was even worse, but to this day I still love it (and amazingly its poster presentation won an award). What’s important here? I did everything, even the art, and I committed to spend at least a minimum amount of time on it each week. I learned so much, and I didn’t have things like fears of letting team members down, because I was the whole team.

Speaking of teams, I did get involved with Spartasoft, the student game development club, which was another important step toward being able to program a half-decent game. The club served a few primary purposes at that point, such as hosting a games party occasionally and getting alumni to come back and present to the club about their experiences in the games industry. However, the most important function for me was the 48 hour Game Jams that were hosted every few months. If you’re not familiar with the concept of a game jam, we basically split into small teams on a Friday evening, a theme is announced, and then each team makes a game about that theme over the course of 48 hours. It often results in a lot of terrible games, but inevitably there’s something new that’s learned, new game ideas explored, and a lot of friendships built with game developers that you might not otherwise get to know. I cleared my schedule for these as a Freshman, and participated in every single one.

It’s how I got connected with a couple of seniors, and I ended up putting in more than just 10 hours a week on one project in the Spring. I started meeting with them in between game jams to polish some of our better ideas. The fact that upperclassmen like Bert, a programmer that now works with me at Iron Galaxy, and Marie, an amazing artist that’s now a grad student at SCAD, wanted to work with some freshman was amazing. I had gotten past the hump for being able to contribute to a game at all. I could help make their games better, and because I was more than ready to step up to the task, I ended up learning a lot back from them.

Conclusion

So what can be learned from my first year making games? First, don’t be afraid to go it alone and force yourself to spend at least a minimum amount of time each week working on it. Secondly, game jams are great, especially if you don’t know very many people to collaborate with. Between the manatee game and the game jams, I had worked on 6 different games by the time I finished my first year of college. How many games had I worked on for class? Zero, and Michigan State even has a game development curriculum! If you have the opportunity to work on *any* game when you’re just starting out, even a game jam game, you’d better have a damn good reason if you pass on it. Failing any opportunities for collaboration, the only person keeping you from making your own game is yourself. Don’t be the asshole that’s keeping you from learning how to make video games. I got lucky that I didn’t do that, it’s easy to be lazy when you’re an 18 year old college freshman.

The “Simple” Answer is Misleading

Backface culling comes up all the time when people are first becoming acquanted with 3D game dev. All it takes is deleting half of a cylinder in Maya and dropping it inside of an easy to use engine like Unity. Then they’re suddenly be trying to figure out what got screwed up that’s causing the inside of it to be invisible. Inevitably the answer given is that those faces are backface culled to avoid rendering the inside of 3D models. That’s true to some extent, but in my opinion very misleading.

The other part of the common answers is that backface culling works by only drawing the side with outward facing normals. This part, is not actually right and it leads to a lot of misunderstandings. I think this answer comes up because people like artists and designers are used to thinking of meshes in 3D space, rarely do people think about the process that turns it into a 2D image unless they get into rendering programming.

The Triangles are not Drawn twice

First off, understand that you have a big mess of triangles that you are using to represent your piece of art. One of the simplest representations of this is to have all the vertices in a big long list, and then to have a big long list of triangles defined from those points. Your computer is taking those triangles and transforming them into the 2D space that is displayed to you. That triangle is so simply defined to expedite rendering that it has no concept of “front” or “back” (well it does, but I’m getting to that). The vertices may have normal information, but that’s used for lighting, not culling. Consider a camera pointed at a half-cylinder that renders properly in the first case, with the following badly drawn diagram taken from above. The green lines are normals:

Now consider the *exact same* list of triangles with the only change being to flip the normals at each vertex. If you try this on your own, it is probably best done by modifying your vertex list in the rendering code, for reasons that will be explained shortly.

With backface culling turned on, you will still see the cylinder in each image, except the lighting will be flipped in the second one. And if you turn off backface culling then the same images will be rendered, and it will be done so with the same number of triangles, because there are no backfacing triangles in the given example. Don’t think that the GPU is rendering the backside and then going over it a second time when you have backface culling turned off for the front side, unless you actually did specify two triangles in that big list of triangles with vertices in the exact same locations.

Assuming you understand that what I described is not what typically happens, let me now explain the point that I’m making. When you flip the normals in a 3D modeling package it’s doing more than just changing the vertex information, it’s also changing that triangle list. This is because the front and back face of a triangle are determined through the novel concept of winding order, which is quite simply that the front of a triangle is the side of the face where vertices are viewed counterclockwise. Suppose you have a mesh with no normal data, just positions in 3D space for vertices. You can still use backface culling just fine as long as your triangle list is specified in the proper order, a point that I think is often missed by designers and artists when trying to understand backface culling.

If you want the proof in the pudding, here’s the OpenGL call to specify culling:

glFrontFace(GLenum mode);

Where mode is either GL_CW or GL_CCW, standing for clockwise and counterclockwise, notice that this has nothing to do with vertex normals!

And here’s a diagram to illustrate which side is the front for a triangle by default (you could assume that triangles are front clockwise as your standard if you really want to, but the default is counterclockwise). If the points defining your triangle are listed in an order that would cause the diagram on the left to be true after transformed into screen-space, then that triangle is front facing:

So what are we culling again?

We’re culling triangles facing away from the camera, which are going to be obstructed by the forward facing polygons on a closed mesh. Those triangles that would be drawing the inside of that cylinder are just going to be covered up by the front-facing triangles that you would actually see. With the winding order in hand, the triangle’s face normal can be determined and checked against the camera. I made this little diagram (insipred by the great explanation for backface culling in Real-Time Rendering). The triangles on the backside of the full cylinder are facing away from the camera, so they are culled (indicated by the dotted part).

So on a closed mesh like a full cylinder, you can avoid doing the work of rendering all of those triangles that you already know are going to be obscured. This is why disabling backface culling is typically not the correct answer when you have triangles culled that you didn’t intend to, and it’s also bad because that usually means that the backfacing triangles are being shaded incorrectly with backwards normals.

If you do intentionally cut that cylinder in half, the easy fix is to add front facing triangles along the inside of the mesh. I believe there is functionality these days (DX10 maybe?) to figure out which side of the triangle is being rasterized. Theoretically you could have the shader flip the normals based off of that information to still have correct lighting, but if you just read a post on the basics of backface culling, I bet that’s not what you’re looking to do.

Because 3D modeling programs are automatically adjusting the winding order for you based off of what direction your normals are, it’s easy to mistakenly think of backface culled triangles as triangles with incorrect normal information, when really it’s winding order that determines it. This is why sometimes you can really end up in weird situations while using a 3D modeling package. I know that as a Freshman in college, there were many models at game jams that had those stray triangles that artists just couldn’t get to show up, and maybe if they did the lighting got all weird. Perhaps thinking about what’s actually happening can help alleviate those pains.

The Problem at Hand

For those of you that might be reading this and haven’t drank the graphics programming Kool-Aid, I’m going to fill you in a little bit as to why we care about anti-aliasing.

In our line of work, the end result of a rendered scene is typically a 2-dimensional array of colors that is displayed on the user’s monitor. Just like how audio (which is originally analog), is quantized when it is made digital, the image is built into a discrete number of pixels during the process of rasterization (i.e. filling in each triangle). The measure of how many pixels are used to display the image is of course known as the resolution, which is why things become increasingly blocky as you lower a game’s resolution, due to there being fewer dots per inch.

Even at high resolutions, individual pixels can still often be identified by the viewer, because the lines and edges of triangles form distracting “jaggies”. This is noticeable along the outline of an object being displayed due to rasterization, but can also occur on interior surfaces for other reasons such as shadows and texture map resolution. The user typically picks the resolution their monitors are set at, but most console games provide 720p or 1080p images to the TV, and monitors tend to follow suit (my laptop is set to ~720p). Other than relying on hardware to pack more pixels into the same physical area by increasing the dots per inch (which is something that Apple is claiming with it’s retina displays, but check out this post for an interesting look at that), we have to find ways to smooth the transition between pixels. Perhaps the simplest strategy to dealing with aliasing is to render into a texture twice as large as the target resolution and then downsampling it to actual output resolution. This allows you to take the average of every four pixels, softening out the boundaries by averaging the edge colors together. However, this also just straight up sucks for performance. You end up processing 4x as many fragments from the higher resolution, and you have 4x the memory usage for the buffer. This is way too high a cost to pay for some smooth edges.

The Hardware Option

There is a hardware accelerated method, MSAA (Multi-Sample Anti-Aliasing). This operates by computing additional samples when rendering the frame instead of rendering additional pixels and downsampling. The fragment shader is only run once for each group of samples. The problem is that it doesn’t work with the increasingly popular deferred rendering techniques, because you can’t really take more samples of a buffer you’ve already rendered. In short, the damage is already done, the data has been discretized to a particular resolution. Furthermore, I’ve always found that MSAA is still pretty expensive (but then again, the only way to make your rendering take 0 ms is to quite doing rendering and switch to a job in finance).

New Maps of AA-land

The desire to use deferred rendering has pushed alternate forms of anti-aliasing to get a lot of attention (I mean who doesn’t need more acronyms, right?). Perhaps the most prevalent that you may have heard of is MLAA, but most of the techniques that I’m referring to here are post-processing techniques that rely on detecting and softening edges. By doing anti-aliasing as a post-process, it will work seamlessly with deferred rendering, and pretty much anything else for that matter. If the technique only needs the color buffer access, then it can even be applied to video/screenshots/whatever of existing games, which I think researchers in this area have found as a great method of showing off how their work.

MLAA was originally a CPU based technique developed by Intel, that has since then been adopted to be done on the GPU. This has been outlined in GPU Pro 2, Game Developer Magazine, and around the net, but if you’re not familiar with it I’ll make a few brief points about it. It essentially boils down to storing edges in a texture with different colors indicating which side of the pixel the edge is located at, and an additional buffer is used to calculate the blending weights for the blurring.

Impressively, they found results falling somewhere between 4x and 8x MSAA, while being 11x faster than 8x MSAA. The memory footprint of the technique is 1.5x or 2x the size of the back buffer depending on the hardware (2x for the Xbox 360 for all you console devs who probably have already heard everything I’m saying). That’s pretty good for something that solves the problems encountered with deferred rendering at the same time. This is undoubtably why the technique has garnered so much attention, and I really recommend the article in GPU Pro 2 if you want a clear view of all of the details.

Enter: FXAA

As you may know from reading other posts, my big side-project/hobby/thing is that when I give a new technique a try, I do it in Unity because it a) is usually not straightforward and requires actually understanding the technique to get it working and b) can be evaluated using the many projects I’ve done in Unity previously. From looking over the details of MLAA, I knew it would probably take a full weekend to get it right, and I had been procrastinating quite a bit with getting around to doing it.

When the third iteration of FXAA rolled out I decided to take a look at what it entailed. I knew in the back of my mind that FXAA was a strictly luminosity based technique, which is interesting to me. The authors of MLAA recommend using depth to determine edges for best results and performance. However, luminosity based techniques offer the advantage/disadvantage of smoothing boundaries that exist in places other than depth, such as with aliasing on texture maps and with shadows. The downside is that this can produce results that are too blurry in places you don’t want blur, such as with text on a prop in 3D space. I once tried a very, very simple luminosity based AA filter, that resulted in too many cons (especially with blurry text) for me to seriously use. I was curious if FXAA would give me similar problems.

With the intention of just looking over the code briefly, I suddenly found myself staring at a very simple and easy to use code base offering a ton of well explained pre-processor options for target platform and quality. Being around midnight when I started looking, I quickly decided that porting the higher-quality PC version of the HLSL code to CG/Unity would be fun. There were two steps involved:

1) At the end of all the other post-processing, calculate Luminosity and slam it into the alpha channel (super easy to do).
2) Perform the FXAA pass. Porting mostly involved fixing texture look-up syntax.

No extra buffers involved. This cuts down on the extra memory needed for MLAA, and the code seemed simple enough that it would probably be pretty fast. I ported all the code within 2 hours and then went to sleep. The next morning I finished setting it up for use in Unity… and was blown away by the speed and the results. Here’s a breakdown of what I got running in the Unity editor at ~720p, using Dust, a previous project of mine, to test it. These are PNG’s cropped down at native resolution:

Shot 1: No Anti-aliasing

Shot 2: FXAA3

Shot 3: 6x MSAA

It takes FXAA3 only ~1 ms [Note: corrected from an erroneous order of magnitude type when I first posted this article] on my laptop (MacBook Pro with an NVIDIA GeForce 320M) to be completed (including the luminosity calculation), and as I mentioned, no additional memory either. I’ll pay that millisecond any day of the week for that quality of anti-aliasing. Furthermore, the blurriness on text was much more acceptable than the fast blur I had tried previously. Note that this text is not really meant to be read, but rather recognized to match the voice over so the player can understand that the voice is that of the journal’s author. I wish I could credit the sources where I put together the fast blur from, but it’s been more than a few months since I implemented/ditched it. Here’s a comparison:

Shot 1: No Anti-aliasing

Shot 2: FXAA3

Shot 3: Fast Blur

This is getting a bit more to the personal opinion end of things, but even though FXAA3 does blur the text, I feel like the fast blur makes it almost uncomfortable to look at. I get the sensation that I’m getting tested for a new prescription of glasses, and have to recognize out of focus letters. To me, that seems like a pretty good indication that your luminosity based AA technique isn’t up to snuff if you get that type of blurring. FXAA3 on the other hand seems acceptable enough that it’ll definitely be enabled in future builds of Dust that get pushed up onto my website.

What did we learn?

Hopefully we learned that FXAA is both fast and doesn’t require additional memory, and is trivially simple to implement or port. There are versions for the 360, PS3, and PC, as well as HLSL and GLSL variants. It’s a luminosity based solution, and comes with the associated pros/cons those techniques have, but I’ve found FXAA to minimize the cons. It should literally take you at most a few hours to get it up and running in your game and evaluate if it’s a good fit for what you’re doing. Furthermore, it is in the public domain (credit to the post by Eric Haines for this twitter snippet):

Although at this point, I’d the amount of quality you’re getting for the effort of implementation, it might as well be under the beer license. Finally, I would point out that Timothy Lottes is still improving on the code, and has already released version 3.9 in the time since I first touched it. You can find the latest source links and updates on his blog: http://timothylottes.blogspot.com/.

Motivation

Chances are if you’re an educator at a University looking to start a gamedev curriculum, you have two routes to go. One is to add game specific classes within your department, generally targeted at a specific discipline, so a computer science department adds a game programming class. The other, is to try to create inter-discipline courses, where designers, programmers, and artists can collaborate. At Michigan State University, we have both, and I’m the TA for the first class in our collaborative curriculum (which is set up a bit like a minor).

You might suspect that the collaborative environment is often the more desired, and I certainly am of that mindset. It adds much more value to an education to collaborate across disciplines on projects, because it offers opportunities that are much harder to get if you’re teaching yourself. I still value discipline specific classes teaching skills like graphics or mobile game design, but like I said, those things can be self-taught much more readily than collaborative game development.

However, one issue that I know we suffer from at MSU is a lack of balance among the disciplines. The four course collaborative track has served as a foundation for our game development curriculum, and additional programming and design specific classes have sprouted up to further grow the quality of gamedev education at MSU. However, our art program has not grown at nearly the same rate. This is largely due to reluctance of the Studio Art department to encourage non-traditional mediums. The 3D art/animation classes are taught through the same department that is the home of most designers, Telecommunications. That department is contains game design, film, audio production, television, and 3D art, so you can understand why it can be difficult for a program to flourish without strong backing from tenured faculty encouraging growth. However, we know that this is an issue and are encouraging improvement, I’m just going to take the opportunity in this post to explain why I think it’s so important to have a balanced curriculum.

This is something that was non-obvious to me when I was deciding where I would go to college, but if anyone asks me for advice on what to look for in a game development curriculum, this is the first thing I mention these days. I’d go so far as to say an opportunity to be in a program with at least mediocre programs for the other disciplines should have as much weight as the strength your specific discipline’s department.

Distracting Flaws

Projects are an important by-product of a collaborative curriculum. However, if one discipline is weak, it can be detrimental in all sorts of ways. It’s hard for students to showcase their amazing artwork if the game crashes constantly from poor programming, or for programmers to showcase their work when poor design causes players to lose constantly. The fact of the matter is that students are expected to invest a considerable amount of time into these projects, it’s unfortunate if they get they collapse quickly from a missing support. Not to say that working around limitations is not an important skill, but being able to create a well rounded game is important. The University should see this as a problem as well, because even if students in the stronger disciplines are succeeding, there is a vacuum of publicity potential. If educators expect to be able to attract highly motivated and talented students, they need to show that their program empowers students to make quality work. The proof is in the pudding (yes I just used that phrase).

A Lack of General Balance discourages Specialization

I encourage people to specialize their focus as soon as possible, even if they change their mind about what they want to do. This complements small team collaboration, which often creates a more generalized environment. On a small team, there might only be one or two programmers. It’s entirely feasible for them to do an alright job on all of the core systems to make the game work, but when are they going to find the time to try out that cool pathfinding technique they learned about if they also have to find solutions for the fact that their artist hardly knows how to model let alone texture or animate?

Consistently weak team members creates a stressful work environment an can even shut off paths entirely for students. Why is someone going to learn to do quality game animations if the programmers repeatedly botch the animation control code? To me the best students are the ones that want to do the things that interest them most in addition to their core work. I hate to think that some might end up impeded from that if part of their team is failing to get its core work done.

Concluding Thoughts

I’m looking at this from the perspective of a student that has worked on some terribly balanced teams, but have also had the pleasure of working with some very talented artists at MSU, most notably the incredibly talented Marie Lazar (http://www.pixelbutterfly.com/). To be honest, the projects that I worked on at MSU that essentially had no designers or no artists, never even came close to making it into my portfolio. To me, that’s more than enough evidence to emphasize the importance of finding a place to learn where you have access to collaborate with decent individuals from disciplines from beyond your own. Missing links don’t go unnoticed.

Even after I decided to write about this, I spotted this part of a fitting conversation between Emil Persson, John Carmack, and Stephen Hill:

This is actually a dilemma that is difficult to answer, and bears in mind questions about what other disciplines and groups have in mind. To me, the problem is that both approaches could be perfectly acceptable. In fact, I would go so far to argue that as broad generalization, that game players almost *want* to be put into a movie when they play a video game, often demanding increasingly cinematic experiences. On the other hand, I think a lot of gamers imagine the future as being “like you’re actually in the game,”  which strays more toward the argument that we want to mimic the human eye over the camera lens. Maybe thinking about what players want isn’t the right way to go about it, players want to have fun, and they don’t always want to understand why. I have a feeling designers would care even less about this issue, caring more about effectively immersing the player in their levels and plots, which is not entirely relevant here because though because this is more heavily dependent on the actual content of the game. We don’t care about content so much, we care about what the “eye” is.

The argument here is very much a discussion of whether or not rendering programmers are trying to simulate a film camera or a human eye, with a very significant part of that being the lens. If you’re not familiar, some effects used in video games such as depth of field and lens flares are largely due to issues with capturing images by using a camera, instead of directly with a human eye. Consider that these are some of the things that gamers rave about over every time a big title puts out a new trailer or demo, yet if it seems a little crazy that these things we sometimes try very hard to reproduce, are actually artifacts. I believe there are a couple factors at play here, a big one being that filmmakers have employed the classic “it’s not a bug, it’s a feature!” mentality and have often used these “flaws” to great effect, which ties directly into what Stephen Hill is arguing in that Twitter conversation. Depth of field in particular jumps to mind as an effect useful for storytelling, and even things like the poor film quality simulated in Tarantino and Rodriguez’s Grindhouse movies play a role in setting the mood, and I’m sure that such an effect could play a noticeable role in the right game. On the other hand, there’s this amusing little comparison between video games and photography that shows just how ridiculous this copying of camera flaws can get:

Yet one has to admit that HDR bloom tends to get a good “wow” factor from players, which leads to programmers running around on the internet sarcastically shouting for “MOAR BLOOM.” After several experiences with team members boosting effects way too high (not to say I haven’t done this myself, the key is to expect to adjust it back down), I’ve begun to understand why so many graphics programmers hold the physical basis of what they are creating to be of upmost importance. That being said, is it a physical basis of a camera or a human eye? I’ve read papers that discuss the physical basis of how an effect occurs in a camera, and I’ve read papers that discuss how an effect occurs in an eye, so really either could apply. However, that reminds me of a similar discussion…

Part 2: Reality or Art?

A related and perhaps more interesting discussion, is whether what looks “good” or what is actually “correct” is more important. This is something that when I started making games, I wasn’t familiar with how technical this issue could get, but I think it stems from more than a few core deviations between artists and programmers. When it comes to visuals, everyone wants to have the “best,” but just like the debate of whether or not that means duplicating camera artifacts among rendering programmers, there is the question of whether or not the best result is the one that is most realistic or the one that the artists think is best.

In a simple world the answer is easy when both parties agree. However, as technology capabilities grow, there are many more chances for rendering programmers to implement systems that do a much better job of modeling how these effect happen in real life. As our graphics improve, art content may start to look different in fundamental ways. Shifting our techniques to draw from a more accurate physical basis can allow us to create even more stunning visuals, but when I was reading through the course notes for Naty Hoffman’s “Crafting Physically Motivated Shading Models for Game Development,” I wondered if suddenly changing the way the specular highlight behaves might have a traumatizing effects on artists. I can easily recall several artists and designers that I’ve worked with that would take the side that a physical basis is not nearly as important as achieving a “good look.” It doesn’t help that programmers are not artists, which I know can make a lot of artists doubt their credibility if they suddenly think an “improvement” makes their art look worse. And speaking as a programmer myself, it doesn’t help that sometimes we can be pretty stubborn ourselves, especially when the math has been proven out that the change makes for a more accurate simulation of real life.

This issue is something that I’ve become increasingly aware of from the excellent work of Naughty Dog’s John Hable, from which I’d like to draw a couple examples of where this debate can come into play. In Hable’s 2010 GDC presentation about some of the rendering techniques used in Uncharted 2, he discussed the importance of doing calculations in linear space. One consequence of this is that the diffuse falloff will be much sharper when calculations are done in the correct space. This is something that artists might initially might be turned off by, because have soft falloffs is supposed to be a good thing, right? But this is really just a case of not knowing what we really want, and Hable points out the lack of soft falloffs in the film Avatar, which as you may recall, was highly praised for its visual quality. In the realm of skin rendering, something that Hable is also involved in, it is often noted that as we account for subsurface scattering, we need to use detailed textures as input. Artists may have had the inclination to blur details in the textures themselves from wanting to account for the subsurface scattering. However, once programmers account for the scattering in code, they need to become aware that they no longer need to soften texture details to achieve the best results. If you’re interested in checking out some of Hable’s work for yourself, I highly recommend you visit his website www.filmicgames.com.

There will always be a great deal of approximations and fudge work involved in real-time rendering, but where the fudge work happens is a constantly moving target as graphics hardware continues expand the possibilities for what can be properly simulated. As this target moves, it also creates an environment that reminds me of something a graphic designer once told me about people that are truly pro status at using Photoshop: the people who really know what they’re doing can make a dozen smaller tweaks that are hard to notice on their own, but when put together the end result is stunning. I think the same can be said about the direction that rendering is going. Just remember that improvements are not always improvements in everyone’s eyes, let alone easy to notice, and it doesn’t help that we all mistakes and sometimes our “improvements” are actually in the wrong. It’s all fun and games until someone gets stabbed over chromatic aberration.

Context

An important thing to keep in mind when you’re working on a feature is the context of why you’re doing it. This includes everything from how much time you can allocate for working on it to how much it impacts the user experience in your final product. In my case I was trying out a potential optimizations to hopefully allow an early prototype from a designer become feasible, which was not due to his heavy use of particles to create a dust storm, causing performance issues from heavy overdraw. This in itself is an important thing to keep in mind when doing an optimization, is whether or not what you’re optimizing is actually a bottleneck in your game.

In my case, I knew the overdraw was becoming an issue because I: 1) reviewed possible reasons why our tons of particles could hurt our performance (to be fair, I was already a little familiar with the issue: knowledge is power!), 2) tested the effects of changing things like particle size, count, screen resolution, and 3) attempted to kill the designer for not using Unity’ nifty overdraw visualization in the first place (Side note: attempting to kill a designer should only happen if you work in a strictly unprofessional environment, like a University lab, and will rarely have a direct impact on perf).

After identifying the problem, the next thing to do was decide how I could address it. Because the particles had only a minor impact on gameplay, the team cut it at the time and focused our efforts elsewhere for our deadlines at the time. However, I knew that being able to ramp up the particle density would really enhance the look and feel of the game, so I kept the problem as a side project.

Ramping Up

The project was slow to get started, due to it being on the back burners for some time. I started to really make progress on it when we were transitioning to hopefully ramp up our work on the game once again after many other projects that had been distracting team members for some months began to wind down (this is a continual problem in a University setting, but that’s a story for a different day). I knew that the feature would be important if the proposed design changes happened, and that the work would be wasted if the designers went ahead and designed new zones without the feature at their disposal early on. With this in mind, the project quickly rose to a much higher level of significance. I spent a lot of time when I wasn’t with the team cracking away at my work, and putting extra hours in at this point can often pay off.

Putting in that extra effort at the right time can really pay off. I quickly began to see measurable results and progress. The implementation that I discussed last time does have a substantial impact on a scene I brewed that’s mostly particles at a density similar to what design wanted, resulting in a greater than 10 ms performance increase in my test scene at a 1280×720 screen resolution when rendering particles to a buffer smaller by 2x. I could do even better than that when using a 4x smaller buffer. That clearly identified two things: 1) that the feature warranted further refinement and 2) that I was right about the bottleneck, as the increase in perf proved that I was not wasting time going down the wrong path. However, in my first post I detailed artifacts that are noticeable in our particular game, and that I intended to explore the mixed resolution rendering as a solution to those issues.

Spotting the Wall

I knew that going to mixed resolution would require substantially more resources for the rendering and compositing of multiple passes, but would also allow me to get away with using a 4x decrease in our smaller buffer. So I started out with the intention of laying it out as simply and as efficiently as possible, and then tune from there.

After a little more work, I had it: mixed resolution off-screen particles. They had some room for quality improvements, but they definitely did a lot to fix the artifacts from before. I had also gained something else: my offscreen particles rendered with almost no increase in perf. I had lost all of my savings from my earlier work, which also meant that I may have moved my bottleneck as well.

I had a couple options at this point: 1) tediously work to increase perf and quality at the same time, 2) scrap mixed resolution and try to find another solution, or 3) kill it. The first option bears the weight of what would most likely be a lot of work on a project that I only have a limited amount of time to work on, and is especially difficult to do without source level access. I did try a couple quick and dirty thoughts on the second option, but going back to the drawing board has its own costs associated with it.

So I Killed It

It’s not easy to kill something, especially when early work shows promise. However, it’s an important thing to be able to do. I have other things to be poking around at in my spare time, and a feature that the game can live without is certainly not worth it. I had invested a reasonable amount of time into the feature already, but that didn’t mean it was worth further pursuit. I suspect further work on it would be roughly equivalent to doubling the scope of the feature. Your time is always one of your most valuable resources, especially when code familiarity in itself makes you a valuable asset to your team. Getting stuck with your wheels spinning on a feature that isn’t worth the benefits for the work isn’t going to help anything, no matter what discipline you’re in.

The Hunt

I started my hunt for getting professional development experience by testing the water by sending out a few intern applications during my second year of college. The first step to getting any job is to actually apply for it. Typically this happens online these days, especially for intern positions. This brings me to my first point: looking good on paper counts for something. No matter what, you have to look appealing enough for a company to think about investing the time to actually talk to you. This was evident because, even though I was completely unqualified at the time, I scored a programming test for the Insomniac core team. I’m sure the only reason that happened was because I didn’t let my grades slip during my first two years of college (proving that even the most painful “general education” class in existence still counts for something), and I also had experience working on several game development projects around the University. In short, I looked good on paper. Having a well thought out resume and portfolio are crucial, and the evidence adds up both from responsiveness to online applications and responsiveness from studio representatives at job fairs, such as the annual career pavilion at GDC.

After making it past the first hurdle, there are typically a two things that happen next. You will almost always need to do a technical interview and a programming test, although the order that they happen can vary. The Insomniac test was proof that I wasn’t qualified yet, but at least it showed me what I needed to do to get past that initial challenge of getting noticed in the ever growing pool of fresh applicants looking for work in game development. So what exactly is in a programming test and how should you prepare?

The Test

For an intern, most tests are emailed to you, and you typically have a specified period of time to send it back. I’ve had several tests structured as a small programming project, in the style of typical work required to make a video game from the ground up. Anything that requires a substantial amount of work for a single problem is usually very generic in nature, so don’t expect heavy AI or Graphics problems if the position isn’t specifically for that discipline. Some of the more “small project” style questions that I’ve had have included memory management, binary file I/O, basic 2D collision detection, and string parsing. Almost every test requires C/C++ to be used, so make sure you’ve had practice! Also, take Jaymin’s advice about showing a smart answer instead of a correct answer. Most problems have many solutions, is yours clean and optimized, and does it handle a good spread of use cases?

Some tests are limited to just a few hours, and I tend to find those to be more towards the stressful end. Perhaps the hardest test that I ever took was a 17 page short answer onslaught crammed into the span of 3 hours. It had required sections on bug finding and solving (things like what to do if the game crashes in release only, in a soak test with no memory leaks, etc), short answer code writing (think bit twiddling, list reversing, basic assembly writing, etc), and math problems (involving vectors, matrices, and trig) together forming the first 12 pages, and then 4 optional sections on gameplay, graphics, core, and networking programming (the remaining 5 pages). For me, it highlighted just how different a test could be from the “small project” style test.

Interviews

A technical interview tends to feel different in nature than a programming test, which was surprising to me the first time I had one. One thing that you should know before any interview is the difference between a struct and a class in C++, this is a question that gets asked over and over again, partly to break the ice. More than three quarters of my interviews have started with that question. Be prepared to know about inheritance, virtual functions, dot products, cross products, const correctness, cache use, and game engine design. Also make sure that you’re prepared to talk about game projects you’ve worked on, especially those you have on your resume or portfolio. Typical questions about your work include asking about your individual contributions to group work, what kind of difficult problems have you overcome, and if what is something (maybe just part of a project) you’ve worked on that you wish you could go back and redo.

Something telling of being a programmer is that no one has ever asked me if I have played games made by their company, where as I know designers that have had tons of questions about redesigning games made by the company they were interviewing with. Don’t sweat it if you haven’t had a chance to pick up a company’s latest game, spend your time refreshing yourself about the details of multiple inheritance and less commonly used C++ keywords instead. If anything, you might be asked what games you’ve been playing recently, just to make sure you play some games.

Rejection

Be prepared to not have everything go smoothly the first couple jobs you interview for. I’ve botched tests and interviews alike, sometimes from things completely out of my control, but the important thing was that I kept working to improve myself as a programmer afterwards. Have you read Jason Gregory’s excellent Game Engine Architecture? Do you have projects where you can show how you specifically contributed? Can you get involved with a mod group or a student club or a something to get experience working with others? Have you done projects completely on your own?

I’ve heard for a long time that game industry jobs require personal connections and luck, but to be honest you just have to be at a certain level of experience as programmer and be able to prove it. I’ve always maintained that the key is to get a feel for what that level is, and then set your goals accordingly. Make sure you feel like you’re continually getting closer to that end goal of landing a job (and in a work environment that suits you best!). If it seems hard, that’s because it’s certainly no cake walk to get there in 3 or 4 years time. This is especially because until you have a game development job, you usually have to spend time doing things like going to class or working a non-games related job. I attended a talk at my university last Fall about interviews and tests for general software engineering jobs, and I left knowing that non-game industry interviews are laughably easy in comparison, even for “top-tier” companies. It’s not easy, so keep at it!

Particles

Particle systems are cheap, flexible, and easy to set up, especially if you’re using Unity3D, as is often the case with the projects happening at my University. This is why on a lot of my projects that I’ve been on (small student projects with tight deadlines), a designer often picks up the task to reduce the workload of other team members. This was the case with one such project that I was working on the fall. That project was Dust.

As you might guess from the name, there’s a lot of dust in Dust. So much so that it takes place in a desert. Some of the designers on the project were tasked with helping to build the ambience for the game with some particle systems. An innocent enough task, but when it came time to integrate the work into our initial prototype, it was clear something was awry.

The problem with video game content creation is that people often have to learn the hard way how their work can impact performance. In this case, we wanted substantially more particles than anything other than a top of the line computer could accommodate (like the one we had been working on). And to be fair, a game like Dust should have as many particles as we can manage to have without blowing perf. One of the biggest impacts on performance from large numbers of particles is overdraw, especially when the particles are filling large sections of the screen:

Massive overdraw from dust particles

This is a shot of the Unity editor’s visualizer for overdraw. The bright areas are spots where many pixels are being drawn over and over again. This is particularly an issue with particles because the system contains many overlapping quads, where pixels will be drawn over many times. Keep in mind that the given shot is from the current version of Dust, his systems were originally much, much heavier on overdraw.

Doing it Offscreen

A solution to this particular consequence of particle effects is presented in the fabulous GPU Gems 3 by Iaian Cantlay. The technique boils down to reducing the number of pixels being rasterized by rendering the particles to a texture that is then composited back into the main image. The rendering is done after a depth buffer has been formed, so that the pixels in the particles that fail the depth test can be discarded properly as you render them. This means that the color can be applied directly back into the scene, which is especially easy if your particles are additive like ours were.

I started considering it as a possible solution to allow for thicker particles in future iterations of the project when I read through Shawn White’s post on his implementation in Unity for Off Road Raptor Safari HD. It didn’t take too long for me to adapt my own implementation a week or so ago, but there are definitely several issues that come to light very quickly (many of which are discussed in the GPU Gems 3).

Problems / Solutions

Zoomed in view of cracks from point sampled depth

Some of the issues that I immediately encountered were visible halos between solid objects (notably the boat that the player controls in Dust) and particles after compositing. One solution to this problem is to have the depth buffer, which is being down-sampled due to the low-res target, take the minimum or maximum depth to be used when downsampling. As noted in the gems article, this is really just a rule of thumb, but it did indeed fix issues with cracks.

Zoomed in view of depth farthest heuristic

However, it made it apparent that the much more serious issue is the aliasing that occurs due to the buffer being a lower resolution. It’s very obvious along the edge of the sail in Dust. My goal was to get the particles to render at quarter res, but right now only half res comes close to an acceptable quality. One thing I could do is render the edges at full res in yet another pass, but the big question in my mind is whether or not two passes for particles would still result in a performance increase. We already have to create a depth buffer specifically for the purpose of the off-screen particles, because we don’t do deferred rendering, it’s not used for much else.

Next Time

Hopefully by the time my next post rolls around (and I’m done with finals), I’ll have some more results (and perf numbers) and give some insight onto whether or not we decide to actually use it in the game. On the plus side, we modified Dust’s design recently to be more separated, so potentially we can split it so that it’s only in effect where the particles are prevalent enough to warrant its use. Also, I’ll try to set up some fresh test scenes that show the effects better, considering that upsampling the in game shots went horribly.

I decided I wanted to be a game developer when I was in 7^th grade, which is now 8 years in the past (I’m a junior in college now). This decision was met with resistance from most people that I knew, but I was really serious about it. Most people thought I’d grow out of it and choose to do something that “made a difference”, especially those who knew that up until that point I was a kid that wanted to make use of my love of math and science at NASA working as a researcher.

Clearly, I didn’t grow out of it. I suppose that it didn’t help that I was already very aware that it was “nearly impossible” to get into the games industry. With this in mind, I managed to get a bunch of scholarship money at a private school on the other side of town for one reason: computer science classes. I didn’t make games in high school (and I’m glad I didn’t – I needed to be a kid), but getting serious about programming years before college was one of the best things to happen to me. Exposure to code before college helps so much farther down the line (by the way check out Brett Douville’s awesome post about teaching his son about programming). But still, I knew that I had a great deal of work was ahead of me because I knew how hard it was to get a job.

Not that long ago, this video surfaced about game development:

To put my thoughts at the time into perspective, I acted a bit like I had that video running on repeat in my head (even though that particular video didn’t exist yet). I made the decision to attend Michigan State University largely for financial reasons, but also because of job opportunities with the game development lab I currently work for. Attending a large university was a bit worrisome to me, given that I didn’t think a diverse program could compete with a specialized school like Digipen or Full Sail. I felt like I had to work my ass off to make up to what I would probably be missing from class. I got involved with Spartasoft, MSU’s game development club, and vigilantly attended every game jam that year. By the time the year closed, I had worked on no fewer than six games of varying sizes, and was starting to get interested in writing shader code. To say the least, I was working hard and was perhaps overreacting a bit to my worries that I might be getting a “lesser” education.

Freshman year was also the year that I watched many of the seniors that mentored me fail to get jobs. The industry was starting to feel the pains of the recession and things were rough, and it scared the shit out of me. No matter how hard I was working that year, I pushed myself into overdrive the next year. I was involved with a team in a game development competition that ended up winning a trip to GDC paid for by Ford Credit. Being an underclassmen on the team as well as the primary programmer, it would be the first project I seriously crunched for, but I’m glad I did because going to GDC that Spring changed my perspective about everything.

Up until that year, I suffered from something that I suspect afflicts many young programmers. I thought I could be both programmer and designer. I mean design gets all of the glory right? Every kid wants to be the next Miyamoto or the next Ken Levine, and I still reveled in the thought. But that year I would finally get enough experience to realize that design is really hard and you might kill yourself if you try to be both a good programmer and a good designer. Still, I thought maybe gameplay code was the place for me, or a scripting heavy design position. And then GDC hit me like a train hitting a chicken, blowing apart all my thoughts about game development as a career. The fact of the matter is that engine code is really cool, tool development is incredibly important, and hand optimizing assembly code makes you a badass. I had started to become interested in graphics and rendering, and John Hable’s presentation about HD rendering in Uncharted 2 convinced me that I wanted to do graphics code professionally. If you get the chance, check out some of his presentations and his website, there’s some great stuff there.

Going along with these newfound desires to work on lower level systems, I once again decided I wasn’t working hard enough and hurled myself further into my work for the next year. I had the portfolio development class for MSU’s game specialization that fall, and I began pulling tons of late nighters and all nighters for my games. I look back on college and realize having a laptop and being able to take my work everywhere with me was both a blessing and a curse. I got to the point where I got a feature implemented into a game while in the back seat of a car on a coffee run. I was out of control, wanting to learn and accomplish so much, so fast. So was it worth it? Now that you’ve read through several paragraphs of me admitting to crunching increasingly more throughout college, I’m going to finally get to my point.

For the first Spring in a long time, I don’t feel like college is a time bomb with only so much time left. Three years of increasingly stepping up my dedication to learning the art of game development has finally lead me to become the programmer I want to be. Maybe I worked too hard at times, but it’s no small task to become a programmer cut out for game development in just four years time. The way I viewed life, if you want to get a job in the games industry, you have to become a good game developer, and the only way that that will happen is if you love making games and learning how to make even better games. I went back to GDC this year, and I’ll be back again next year, because I love learning about all the crazy techniques people are developing and trying. People talk about how you have to have connections to make it as a game dev (what are we, film?), but I honestly think that’s bull shit. I don’t want your card because I want a job, I want your card because I want to be friends with people riding the edge of what games can do (speaking of which, props to the Battlefield 3 team). If you want to make games throw yourself at it, because only you can make yourself a crack game developer.

When I’m walking to the lab and I feel like I’d rather go home and sleep through the afternoon, I listen to one song consistently. It’s called “Ali vs. Frazier 1” by a Massachusetts hardcore band named Bane. I think it summarizes my view’s on what it takes to become successful at anything worthwhile, including game development:

(rumble, young man, rumble)
how many more days will you sit
and talk about your ambitions
all that you can be
the person you are dying to be
the place you want to get to
but always out of reach
before that fury swells inside of you
grows so big that it forever quiets you
stand up to your demons
make a run at your goliath
find the best, find the worst
waiting in both of you
it’s not the who or the what that is lasting
but how you fight
that is the fight
the only mark that will not leave you
and I will feel my heart drum its final beat
if it meant that I have given this my all
there’s nothing left for me to believe in
if not your, if not this…
what else is there but death?
(it’s your call…it’s all on you)
give more
give everything
give blood

Tools and Motivation

As a developer of a real-time graphics technology (video games), you are almost certainly going to be making use of a GPU to accelerate your graphics processing. The GPU exploits high parallelization of rendering to speed up its work, and is responsible for transforming your 3D world into a 2D plane that is displayed to the player on their computer scene. However, not all effects are easy to simulate in a 3 dimensional space. Thats where leveraging further work on the GPU to perform additional image space calculations can come in handy. Some of these are quite obvious, such as depth of field calculations, being that depth of field is an artifact of lenses. Other popular image space effects include motion blur, color correction, and anti-aliasing. Image space calculations are also the foundations of deferred shading and lighting, rendering techniques that are becoming increasingly popular to handle a large number of lights in a scene.

The Actual Technique

Image space calculations can be performed on the CPU, but as with most things in graphics that would slow and booorrrriiinng (unless you’re playing with SPUs, in which case carry on). The main point is that iterating over an image in the main thread while performing per-pixel calculations on the CPU would most likely be in bad taste. Instead you might consider the following GPU based solution:

1. Render your scene normally, except to a texture if you are not already.
2. Render a quad that covers the entire screen, with the texture from the previous step as a texture applied across the quad.
3. Perform your calculations in the shader code used to render that full screen quad, modifying the how the texture is applied.

There are several catches with this that you have to keep in mind. First and foremost, each fragment’s calculations cannot be too dependent on other locations on the screen. If you have to sample your frame many times, then those texture accesses will quickly add up, which is one of the big considerations that comes into play when performing screen space blurs. Secondly, while this might provide an effect for much cheaper than trying to model a similar effect in 3 dimensional space, keep in mind the actual performance is dependent on the resolution of the screen, which may be less than desirable. As resolution increases, so does the number of fragments being processed. In the end, it’s all about picking when and where to perform different calculations in your game.

An Example

Here’s a sample of a very simple post-processing fragment shader, written in CG. It does a simple screen space distortion based off of the x and y channels of a texture. The vertex shader doesn’t do anything particularly special other than set up uvs that are interpolated from 0 to 1 across the quad. In general, most of the action happens in the  fragment shader when doing post-processing.

1
2
3
4
5
6
7
8
9
10
11
12

uniform sampler2D _MainTex;
uniform sampler2D _DistortionMap;
uniform float _Distortion;
 
float4 frag (v2f i) : COLOR
{
    float2 distortedOffset = (tex2D(_DistortionMap, i.uv).xy * 2 - 1);
    distortedOffset *= _Distortion;
 
    float2 distortedUV = i.uv + distortedOffset;
    return tex2D(_MainTex, distortedUV);
}

The shader itself is really only a few lines of code! So easy! Here’s what is happening a little more in depth. The float2 “distortedOffset” is simply the interpolated uv coordinate plus a lookup into the normal map which is then unpacked to fit into the range [-1,1] instead of the [0,1] range returned by tex2D(sampler2D, float2), which is then finally multiplied by _Distortion to control the strength of the distortion. Finally, a lookup into sampler2D MainTex is performed, where _MainTex is the previously rendered image. If there is no distortion, then the call would be equivalent to tex2D(_MainTex, i.uv), which would just copy the source image’s color to the new target. Speaking of targets, you might consider rendering this post-processing pass  into a texture as well, besides just your initial rendering of your 3D scene. This is so that you can pump the output of this post-processing into another post process you are implementing to be able to stack effects on top of each other.

Here is a sample of this particular distortion shader in action.
The original rendered scene:

Before Post-Processing

Texture that the distortion is calculated from in the shader:

Distortion Map

The final result:

After Post-Processing

Conclusion

Great success! The important question here should always be: how hard/expensive would it be to achieve the same effect in a different space? What do you gain/lose by doing it in image space? And also important is the question of whether or not this actually makes your game look any better. In the end, I personally think that post processing is great fun, especially used in terrible crazy ways on personal projects. You never know what you’ll come up with when you play around with ideas in a different space, for example here’s one paper exploring the possibility of moving skin rendering into screen space: http://jbit.net/~sparky/subsurf/cbf_skin.pdf. Finally, fun fact of the day: I modeled and textured that fish in the sample images way back when I was a Freshman, which is why it’s so shoddy and terrible.

Lerp(t) = (1 – t) * StartPoint + t * EndPoint

And before you know it, those beginner programmers are Lerping all over the place, using it to help overcome all sorts of challenged in game development from moving an object in 3D space to fading between two audio clips. In reality, this is actually just accomplishing one thing- it is just a straight line segment, reworked into a form that makes it easy to sample any point on that line. Anyone that’s taken high school level math knows that there is more to life than just straight lines. What if we need a curve?

So let’s consider how we might go about making this curve. An approach that I’ve seen many beginner programmers, including myself, to try to get around the problem by stringing several lines together as an approximation. This might be alright in some situations, but it fails in many. For one, it requires the placement of a lot of points and then storage of that data, which can be painful. Secondly, and perhaps more importantly, it will suffer from being jagged. Maybe at a certain distance we will have enough points that it will appear smooth enough, but when examined closely enough the lack of smoothness will always become apparent (unless the number of points approach infinity).

Enter: Bezier Curves

Moving to a more complex equation can allow us to overcome these difficulties. We know we can create curved functions with polynomial equations, but does this actually help us? We can write these as parametric equations much like the linear interpolation equation shown above, but we don’t want to write a new equation each time we want to generate a curve. We want to generate different curves between points, not just the same curve between them each time. Before this was not an issue, as we only ever used a straight line. What we can do is store additional data that factors into our equation with additional data points beyond just the start and end points of the curve. This is where quadratic and cubic bezier curves come into play.

If you’re new to bezier curves, you may find them a little intimidating due to the implication that there is some more complex math happening.  I used to think they were outside my league of math just because Adobe has released entire applications based around drawing with bezier curves.  Little did I know that I already used the 1st-degree, linear bezier equation all the time.  Here it is:

LinearBezier(t) = (1 – t) * StartPoint + t * EndPoint

As you can probably tell, the linear bezier is actually just the linear interpolation equation.  Higher degree bezier curves introduce control points into their equations and the math only gets a little bit worse, but the similarities are still very visible to our friend linear interpolation.

These 2nd and 3rd degree bezier curves can be defined as:

QuadraticBezier(t) = (1 – t)^2 * StartPoint + 2(1 – t) * t * ControlPoint + t^2 * EndPoint

CubicBezier(t) = (1 – t)^3 * StartPoint + 3(1 – t)^2 * t * ControlPoint1

+ 3(1 – t) * t^2 * ControlPoint2 + t^3 * EndPoint

Let’s look at what happened. Our computations are a bit more expensive, but can now craft more intricate curves. These equations allow for the manipulation of continuously defined curves between the start point and end point by adjusting our control points. Take a look at a couple of shots showing examples of these:

A Quadratic Bezier Curve

A Cubic Bezier Curve

The interface for modifying these curves is quite simply the same as the start and end points of a line, being that a control point is just another point in the same vector space as our start and end points.  Increasingly complex curves can be generated with higher degree bezier curves, but we don’t actually want that because the computation becomes increasingly expensive and the increases in control points can become cumbersome.  This is where turning to piecewise equations becomes a good solution.

B-Splines

Quadratic and cubic bezier curves do offer a lot more flexibility, but they also offer something else, the ability to be chained together without losing continuity, forming a piecewise function for your curve.  So instead of using higher degree equations to create more intricate curves, we can create a series of cubic bezier curves and restrict them a bit to meet continuity requirements.  Just as a refresher, a function meets different levels of continuity based depending on how many derivatives of the function are continuously defined.  The previously mentioned chain of straight lines only has a continuity of C0, meaning that all points along the spline are defined, but because it lacks C1 continuity, the first derivative is not continuously defined. The more derivatives that are continuously defined, the smoother our curve will be, but it will also result in less and less control over the spline as a consequence.

Now that we have moved to a chain of cubic bezier curves, we can easily obtain C1 continuity by requiring that the line formed between an end point and the second control point, is the equal and opposite of the start control point on the curve that is being transitioned to.  This looks like this:

Piecewise Bezier with C1 Continuity

However, even with that restriction we can still have relatively sharp corners occur.  If we require a smoother curve, we can convert our piecewise curve to a B-spline. This will allow us to achieve C2 continuity, meaning that the second derivative is continuously defined. Our bezier control points are now moved to being controlled by a new set of points known as deBoor points. A cubic bezier curve that always has C2 continuity will be defined by these deBoor points.  Perhaps the biggest drawback is that the start and endpoints of a segment are no longer directly controllable either, but it is still relatively easy to work with from just the control points. Here are the equations for the generating the cubic bezier from the deBoor points, note that I name the variables based off of examining a particular segment in the spline:

CubicControlPoint1 = (2 * deBoorStartPoint + deBoorEndPoint) / 3

CubicControlPoint2 = (2 * deBoorEndPoint + deBoorStartPoint) / 3

CubicStartPoint = (deBoorPreviousStartPoint + 4 * deBoorStartPoint + deBoorEndPoint) / 6

CubicEndPoint = (deBoorStartPoint + 4 * deBoorEndPoint + deBoorNextEndPoint) / 6

We can then build our cubic bezier with these values.  These relationships are a lot easier to visualize in a picture:

A B-spline defined with deBoor Points

As you can probably see, these are a bit more mathematically involved than just a bezier curve. They are still relatively easy to manipulate through the deBoor points, and they have the added benefit of not requiring any additional data beyond that of a bezier curve and any bezier code that you currently have can be fit to work as a B-spline. This is exactly what I did when I most recently used B-splines to obtain C2 continuity on a project, I took advantage of the fact that the deBoor points could control the cubic curve entirely tool-side, while my in-game bezier code functioned exactly the same.

Conclusion

As you can see, math is fun and can lead to better better solutions to the engineering challenges in game development. Bezier curves naturally extend into many areas, and many useful uses have been cooked up for them. For a really great example of bezier curves at work, check out this article from GPU Gems 3 about rendering vector graphics on the GPU: http://http.developer.nvidia.com/GPUGems3/gpugems3_ch25.html.  Keep in mind that all this awesomeness comes with the cautionary wisdom that moving to higher degree curves is not always worth the increased computational complexity.  As always, make sure you evaluate what the right tool is to get the job done.

Shortly after we arrived the conversation turned to quality of life, something that Mike has been talking about quite a bit as of late, including an excellent presentation at Game Forum Germany. One of the points brought up was that the huge influx of students interested in game development allows companies to basically burn through their employees and easily refresh themselves from a deep pool of people desperately looking for work. I didn’t end up contributing my thoughts on the subject from a student perspective, but I later realized that maybe I did have a relevant perspective to add to the discussion. Especially because I’m currently in the weird spot in my life where I’ve found myself fitting into many different sides of the fence dividing developers and those aspiring to become developers, which as a warning, might make me sound like I jump perspectives a few times throughout this. I might be wrong about some of this, so feel free to flame me in the comments in that case :)

When Start-ups are “Safe”

I attended GDC with a handful of other Michigan State students this week, many of which are actively in the hunt for jobs. However, Shawn and a couple others weren’t looking for work because as their current projects wrap up and they begin to graduate, they’ve decided to form Adventure Club Games, a start-up, doing a combination of contract work and smaller independent projects. They’ve already landed a few serious games related contracts through the professors with grants at the university, so they may be well on their way to surviving the first year as a start-up.

What worries me is that some of the students looking for jobs have told me this week that they feel that the guys involved in Shawn’s company have the “safe bet.” This is quite honestly one of the most terrifying things that I have ever heard students say. I’m intentionally not letting myself get roped into Shawn’s company, even though they’re all great guys and I know they’d be excited to have me on board. I’ve been involved in two start-ups in the past (not all game related), and I think that the stress that comes along with that could quite honestly kill me at this point in my life. Going into an industry where a start-up feels like a safer bet than finding a job is just… unbelievable.

Let’s circle back real quick to what I mentioned earlier, this conversation started because we were talking about crunch, and the thought that there is an endless pool of fresh talent ready to get killed in a vicious cycle of burnout is quite troubling. These kids would work well over 80 hours a week and throw themselves under a bus for the opportunity to work on “game X,” and they still feel like it would be easier to get involved in a start-up. In my experience, being in a start-up is a whole different devil, it’s a world where the crunch is endless because the sole determinant of success is yourself. The amount of time you spend on your work is a bottomless pit of stress, business, and development, but the reward is that you get ownership and control. People at Mike’s get together mentioned staying with a death march development because of loyalty to the studio or loyalty to the project, but think about it, in a start-up, you are the entire show. You either rise or die, and it’s a very difficult thing to do. To see students thinking starting a business as a more secure temptation than sticking through the hurdles of landing a job with an established studio- it shows just how crazy this all is.

I feel like students might be getting the wrong messages from the industry as to what should be expected of them. After spending time in the career pavilion, several students told me that they basically felt like 4 years of college had absolutely no relevance to them getting a job. Something is not right if people feel like that, especially if they’re part of a program that is apparently ranked among the top in North America. We don’t want people to drop out of school and try to make games in their basements. We want people to rise above their degrees and push their programs to new heights, even if the piece of paper you get at the end isn’t as important as what you learned along the way. Game development education is still in its infancy, we need to be encouraging programs to grow and mature.

To be a Student

This all really gets to me because I think that this perception finding work as a student game developer being impossible might be having a negative impact on the the student culture as a whole. This ties back into how uncomfortable it makes me when design students want to make games that are marketable instead of games that are art. Are we suffocating creativity out of students? I always thought design students would be stubbornly optimistic about games as art, wanting to experiment and create as they find themselves suddenly empowered to create virtual worlds. Shouldn’t students be encouraged to make use of their time in college to experiment and take risks? They’re getting an opportunity to make the games they want to make, as they exist in an environment free of the weight of money and the reponsibilities associated with a formal job. This is why I think the student showcase is such an important part of the Independent Games Festival- it encourages students to take a risk and do something creative.

This applies to disciplines beyond design as well. Programmers and artists should be encouraged to try new or unconventional techniques. In school, it’s alright to fail at an implementation and learn from our mistakes. We need to encourage people to experiment with ways to make their games interesting, optimized, beautiful, and every other thing that we want to achieve as developers when we work on professional projects.

From A to B

When I started college, it was also the beginning of me truly becoming serious about game development. I took a long look at myself and realized where I thought I might need to be as a developer by the time I graduate. I imagined the quality of game I thought that I might need to be capable of producing to “break in,” those devilish words that give an aspiring student the chills.

At the time, it seemed impossible. There was so much to learn, and improving the quality of my work was such a tedious process, I wondered if I would be heading to grad school at the end of my four years to continue my education. Now that I’m three years in, I feel a little foolish for thinking that during my earlier years. However, there’s something important there- that envisioning of where I wanted to be at the end has been vitally important for my growth as a developer. It was something that I did on my own that I really wish more people had been saying because I think it

In all honesty, I feel like we need to be telling students a bit more than just “you have to make games,” which is something that I’ve heard over and over again. That’s not enough. There’s so much more to becoming a good developer.

What I think we should be saying:

Based off of my own experiences over the past years, here’s a short list of what I think we should actually be telling students:

1. Set your goals. How long do you have until you will be looking for a job? The clock is ticking. But the harder you work early on, the easier it will get. Make sure you include getting team based experience in that road plan if you don’t intend to be a one man show your whole life.
2. You have to be good at what you want to do. Can you produce work of the quality of studios that you want to work at? Realize that basic skills across your broader discipline is important (art, programming, etc), but specialize in one or two areas is important as well (animation, rendering, etc).
3. Speed counts too. Artists hear this sometimes, but its important for everyone. If you are given a programming test, it will almost certainly be timed. If you are asked a design question in an interview, you have to be quick on your feet. This comes naturally with practice. If things aren’t slowly becoming second nature to you, you need to be committing more time to your work.
4. Live beyond your tools. Getting stuck inside a bubble will result in sloppy work and messy solutions to problems. Consider an animator that spends all their time playing in Maya or Max but doesn’t study natural motion, or a character artist that hasn’t studied anatomy, or a rendering programmer that isn’t familiar with linear algebra. You can cripple yourself if you’re not careful.
5. Try new and unconventional things in some of the free time you might have (if you’re like me this time is usually in between midnight and 4 am). Read white papers about the latest research, try new art tools, prototype an unconventional design, the list goes on and on. Like I said earlier, trying new things is especially important in college.
6. Your professors don’t tell the whole story. You’d better learn what they have to teach you, but quite often realize that they don’t always have the same goals in mind. This hearkens back to my feelings about how little my current roommate (@krismicinski) needs the capstone class that my school touts for it’s Engineering programs, being that he studies compilers and just got accepted into half dozen PhD programs. The class is designed to help people get jobs as the more generic breed of software engineers, not as game developers, and most certainly not as academics. You’ll have to learn a lot on your own no matter where you are, and this trend will continue into professional work.
7. Your free time is a commodity. One of my closest friends is part of my school’s brutal Biochemistry curriculum, which has been compared to doing graduate level work straight out of high school. He has a saying that “in college you can work, socialize, and sleep. We only get to pick two.” Playing games is, in my opinion, a dangerous culprit here. Playing through an entire AAA game is a time consuming process, so pick your games wisely.
8. Save up to attend GDC. Get at least a main conference level pass. I personally equate a week of cutting edge development talks to be worth an entire semester of college. Sound like a lot of money? How many 60 dollar games do you buy in a year that you don’t actually have enough time to play?
9. Keep a development blog. It’ll help you be coherent when people ask about what type of work you do.
10. Make a Twitter and follow developers. It’s an easy way to keep up to date on the latest research, techniques, and thoughts of some of the brightest people in the games industry.

Concluding Thoughts

So here’s where the hypocrisy comes in a bit. Even though I act like I’ve done a good job growing as a game developer, I still don’t know if I could resist accepting a position at a studio that has abusive dev cycles. I feel like I could at this point in my life, but at the same time I wonder what a dozen rejected applications would do to me after I graduate next year. I used to keep myself up wondering if I could actually cut it as a game developer. After three years of doing the ten things I just mentioned, I feel confident that I’ll be at the level I want to be in a year’s time for graduation. What keeps me up now is whether or not I’ll be able to live a healthy life style after college. There’s too many stories of developers who have let parts of their lives be wrecked by abusive work environments, too many developers who get up to accept an award and thank their wives for putting up with endless hours of overtime. So what are your opinions? I’d love to hear what advice you have for students, whether or not an endless pool of applicants enables a burnout dev cycle, and why I’m just straight up wrong. Flame on?

As such, I’m going to talk about some basic mathematical aspects that apply to our friend, the 3D mesh. This is the type of thing that will probably cause many rendering programmers to have violent flashbacks to “Intro to Graphics” courses, but I think people not as intimately involved in graphics might find this interesting. The entire motivation for this is because my group for a class project where we did some terrain generation with a procedurally raised race track, had no idea what our professor was talking about when he told us that our track generation always formed a star from our seed node. We thought he was talking about a star shape (you know, like the sun), and not the mathematical operation, but I’ll get back to that later.

Lesson 1: The Euler Characteristic

Unless you live in a 2D only world, I’m going to bet that most of the people reading this have been living in a forest of 3D meshes for years. And I know that a lot of artists have little rules of thumb that they use for what the ratio of faces to vertices their meshes should have to figure out if there are hidden faces or holes floating around, but theres are actually some hard math behind the number of vertices, faces, and edges in a given mesh (triangular or not).

This is the Euler Characteristic:

χ = V – E + F

Less Formally:

Euler Characteristic = Vertices – Edges + Faces

Whoah, what does that mean? Back as a newbie game developer that didn’t know the difference between a heap and a memory pool, I would’ve thought that was the least useful thing ever. However, what’s interesting is that the Euler Characteristic is actually constrained based off the topology of the mesh, and not by vertex or face count as it first appears.

This is Euler’s Formula:

2 = V – E + F

This is the Euler Characteristic for a sphere mesh, and more importantly it holds true for any mesh that can be smoothly deformed to a sphere has this characteristic, as does any convex polyhedra. You can think of the Euler Characteristic as being controlled by the number of “holes” in the mesh, with the number of vertices, edges, and faces dependent on the characteristic.

χ = 2 * (1 – G)

Where G is the number of holes in the polyhedron. So a sphere is a closed mesh with no holes, meaning it has a G of 0, which results in an Euler Characteristic of 2. Consider the tetrahedron:

A Tetrahedron in Maya

A tetrahedron has 4 faces, 4 vertices, and 6 edges. It also has a G value of zero for having no holes. So:

χ = V – E + F = 2 * (1 – G)

χ = 4 – 6 + 4 = 2 * (1 – 0) = 2

χ = 2 = 2 =2

= Win!

This math holds up with other meshes and also can be proved in a variety of ways. Cool right? Direct application? – probably not, but at least you’re a better person for knowing a little more math from our friend Leonhard Euler. Now go impress your friends at parties by knowing yet another Euler Formula.

Lesson 2: Closure, Link, and Star

So now I’m going to change gears a little bit and talk about some math terms that apply to meshes, but really could apply to other sets as well. Specifically, these are operations that apply to a simplicial complex. A simplicial complex is a set of simplices that follow two conditions:

1. A simplex is made up of faces, and each face of a simplex is also part of the simplicial complex.
2. The intersection of any two simplices in the the simplicial complex is a face of each of those simplices.

Triangular meshes, the simplicial complexes we care about, are made up of points, lines, and triangles, but other simplicial complexes can contain higher dimensional components, such as tetrahedra. So there’s a mathematical concept correlating to our lists of vertices and triangles, and theres also some mathematical operators that go along with a that structure. Also note that a face of a given simplex is a lower dimensinal component that forms it. So the face of a triangle is an edge. That edge is in turn a simplex that has vertices for faces.

The Closure

The closure of a set of simplices in a simplicial complex is the smallest subcomplex that contains each simplex in the set. Now say that ten times fast. Yeah, if I read that description in a paper I would probably stop reading. But this where fancy diagrams come to the rescue! Here’s a relatively basic simplicial complex made of triangles, vertices, and edges:

Consider a single edge being our entire set of simplices we are performing the closure operation on:

S = {one edge}

Any subcomplex that contains this edge becomes part of the closure. An edge cannot exist without the two vertices so they become part of the set to form the smallest containing subcomplex:

Cl S

The Star

The star is another simple operation on a set of simplices in a simplicial complex. And yes, my previous lack of knowledge about it last Spring led me to have a very misled conversation about what a star was during a presentation.

A star is the set of all simplices in the complex have a face in the set of simplices that the star is being computed for. Once again, taking the star of a set of simplices is also relatively straightforward once you get past all the wordage and see a picture of it in action.  We’ll use the same edge for set S as we did before:

St S

The Link

Computing the link is a little bit more involved than bit more involved than computing the star or the closure alone. That is because for a set of simplices S, the link is the closure of the star of S minus the star of the closure of S. This is much easier to comprehend when visualized.  We’ll use the center vertex for the set S this time:

S = {one vertex}

First consider the closure of the star of S, where the star of S will be the vertex, the 4 edges that contain it, and the 4 triangles that contain it, and then the closure will expand that to be the complete subcomplex that contains those faces, so the 4 edges and 4 vertices around the edge are added as well:

Cl St S

And then the star of the closure of S, where the closure of S will just be the vertex again, and the star of that will include the 4 edges and 4 triangles that include the vertex:

St Cl S

Finally, by taking the difference of the closure of the star of S and the star of the closure of S, we get the link, which in this case resembles a boundary around the triangles in containing our initial vertex:

Lk S = Cl St S - St (Cl S - ø)

So that covers the link, star, and closure.  Applications?  Just like with the Euler Characteristic you could probably go on living your life without knowing them… but that would be no fun!  In the context that I learned about the link, star, and closure it had relevance to types of information that might be stored in a data structure defining a 3D mesh.  For instance, you might want to store the star and the link of every vertex in the mesh to be able to find the triangles and edges that surround each vertex.

You Made It!  Rejoice and be silly!

Hopefully if you made it this far you actually found some of this to be interesting.  If not, well you probably won’t be any worse off for it.  But because I’m a college student and just wrote about a whole bunch of mathematical concepts, even though college students are supposed to spend their time doing silly things… I leave you with this: a video of Kris, my roommate, convincing the chemical engineer that lives across the hall from us that “Wumbopixels” are the metric equivalent to Megapixels.  Luckily this conversation was captured by another guy at the the table learning how to use the video feature of his fancy new phone.

He believed that “Wumbopixels” were real for several days before we finally decided that we should inform him of the truth before the damage could become permanent :)

This was a cross-post with my personal blog at jonmanatee.blogspot.com and you can always find me kicking around @JonManatee in the Twitterverse.

If I were to ask any random developer as to why developing serious games is challenging, I think I would typically get a response about the game design challenges associated with serious games. While it’s no cakewalk to try to make a game that is both fun AND teaches the player some other skill (such as financing a car or managing a power plant), there are some very likely pitfalls that will inevitably impact the entire team beyond the designers.

You can’t go it Alone

The issue that I think often goes overlooked is that because an average game developer is just an expert at making games, a serious game developer will have to consult an external source that is an expert in the subject matter of the game. More often than not, that source of expertise is also the source of funding for a serious games project. An example is a company contracts a developer to make a game about workplace safety, and the developer will have to work with them to understand what that actually entails.

“The Client”

This places people outside of the development team, who typically know very little about game or even software development, in a very powerful position. The delicate relationship that arises out of this is, in my opinion, potentially more intricate than that of the developer-publisher relationship. Personally I suspect it is more like working with an IP holder on licensed work because they care much more about how their content appears in the game. Often if the client wants to be too hands on, the game ends up not being fun, but if the developer just runs wild, the game could end up not teaching the players what it’s supposed to, which is just as much of a failure in serious game development. It has to be a symbiotic relationship to result in a successful game, and I’ve compiled a couple of pointers from my personal experiences from the past couple of years.

1. Figure out if the client understands what a fun game is, even if they don’t know how to make one. If they do, make playable versions of the game a high priority. If you’re on the right track with your game , make sure your client can play it and be on the same page as you, or else they might demand you scrap your work and change directions without giving your hard work a chance.

2. Be ready for change. If the game isn’t meeting its goals, the design team is going to want a change, and that change is necessary. Huge design changes are in, my opinion, much more likely when you have to teach the player about something typically less than fun. Convincing the client that the change is needed can be hard though if they are worried about meeting deadlines, especially if your code base won’t easily adapt to the new direction. Or the change could come at the demand of the client, which could be even more drastic, as one meeting can leave you commenting out half of your code base.

3. Be afraid if your designers and client don’t get along. You might notice that the client is portrayed in polar opposites in the past two points (wanting change and not wanting change), yet both can happen on the same project. Anything less than a good relationship between your designers and your client is an early sign of approaching trouble. As I said before, a successful serious game needs a balance between the entertainment and the game’s subject matter.

4. Keep in mind that sales is not the goal of the project. From a client’s perspective, a failure to meet the goals doesn’t always result in financial trouble. It can result in the client looking to fund a “do-over” of the project. If you didn’t obey point #2 is reworking the entire game going to be a nightmare? This is especially possible if the project was funded by grants. A failure of one project just means that it is the basis of the next grant proposal to fund the fixing of the first game’s shortcomings.

So there, a little peek into the world of serious game development. I’ve begun to wonder if the challenges of separated expertise is one of the reasons why it seems that there is a high concentration of serious games that are quite good at what they do because their purpose is to teach the player about game design.

This was a cross-post with my personal blog at http://jonmanatee.blogspot.com/2011/02/serious-developments.html and you can always find me kicking around @JonManatee in the Twitterverse. I promise I’ll write about something more technical next time around :)