In this post I briefly explain what these special numbers are, why they exist, and what to watch out for.

This article is the last of my series on floating-point. The complete list of articles in the series is:

• 1: Tricks With the Floating-Point Format – an overview of the float format
• 2: Stupid Float Tricks – incrementing the integer representation of floats
• 3: Don’t Store That in a Float – a cautionary tale about time
• 3b: They sure look equal… – special bonus post (not on altdevblogaday)
• 4: Comparing Floating Point Numbers, 2012 Edition – tricky but important
• 5: Float Precision—From Zero to 100+ Digits – what does precision mean, really?
• 5b: C++ 11 std::async for Fast Float Format Finding – special bonus post (not on altdevblogaday) on fast scanning of all floats
• 6: Intermediate Precision – their effect on performance and results
• 7.0000001: Floating-Point Complexities – a lightning tour of all that is weird about floating point
• 8: Exception Floating point – using floating-point exceptions to find bugs
• 9: That’s Not Normal–the Performance of Odd Floats

The special float values include:

Infinities

Positive and negative infinity round out the number line and are used to represent overflow and divide-by-zero. There are two of them.

NaNs

NaN stands for Not a Number and these encodings have no numerical value. They can be used to represent uninitialized data, and they are produced by operations that have no meaningful result, like infinity minus infinity or sqrt(-1). There are about sixteen million of them, they can be signaling and quiet, but there is otherwise usually no meaningful distinction between them.

Denormals

Most IEEE floating-point numbers are normalized – they have an implied leading one at the beginning of the mantissa. However this doesn’t work for zero so the float format specifies that when the exponent field is all zeroes there is no implied leading one. This also allows for other non-normalized numbers, evenly spread out between the smallest normalized float (FLT_MIN) and zero. There are about sixteen million of them and they can be quite important.

If you start at 1.0 and walk through the floats towards zero then initially the gap between numbers will be 0.5^24, or about 5.96e-8. After stepping through about eight million floats the gap will halve – adjacent floats will be closer together. This cycle repeats about every eight million floats until you reach FLT_MIN. At this point what happens depends on whether denormal numbers are supported.

If denormal numbers are supported then the gap does not change. The next eight million numbers have the same gap as the previous eight million numbers, and then zero is reached. It looks something like the diagram below, which is simplified by assuming floats with a four-bit mantissa:

With denormals supported the gap doesn’t get any smaller when you go below FLT_MIN, but at least it doesn’t get larger.

If denormal numbers are not supported then the last gap is the distance from FLT_MIN to zero. That final gap is then about 8 million times larger than the previous gaps, and it defies the expectation of intervals getting smaller as numbers get smaller. In the not-to-scale diagram below you can see what this would look like for floats with a four-bit mantissa. In this case the final gap, between FLT_MIN and zero, is sixteen times larger than the previous gaps. With real floats the discrepancy is much larger:

If we have denormals then the gap is filled, and floats behave sensibly. If we don’t have denormals then the gap is empty and floats behave oddly near zero.

The need for denormals

One easy example of when denormals are useful is the code below. Without denormals it is possible for this code to trigger a divide-by-zero exception:

float GetInverseOfDiff(float a, float b)
{
    if (a != b)
        return 1.0f / (a - b);
    return 0.0f;
}

This can happen because only with denormals are we guaranteed that subtracting two floats with different values will give a non-zero result.

To make the above example more concrete lets imagine that ‘a’ equals FLT_MIN * 1.125 and ‘b’ equals FLT_MIN. These numbers are both normalized floats, but their difference (.125 * FLT_MIN) is a denormal number. If denormals are supported then the result can be represented (exactly, as it turns out) but the result is a denormal that only has twenty-one bits of precision. The result has no implied leading one, and has two leading zeroes. So, even with denormals we are starting to run on reduced precision, which is not great. This is called gradual underflow.

Without denormals the situation is much worse and the result of the subtraction is zero. This can lead to unpredictable results, such as divide-by-zero or other bad results.

Even if denormals are supported it is best to avoid doing a lot of math at this range, because of reduced precision, but without denormals it can be catastrophic.

Performance implications on the x87 FPU

The performance of Intel’s x87 units on these NaNs and infinites is pretty bad. Doing floating-point math with the x87 FPU on NaNs or infinities numbers caused a 900 times slowdown on Pentium 4 processors. Yes, the same code would run 900 times slower if passed these special numbers. That’s impressive, and it makes many legitimate uses of NaNs and infinities problematic.

Even today, on a SandyBridge processor, the x87 FPU causes a slowdown of about 370 to one. I’ve been told that this is because Intel really doesn’t care about x87 and would like you to not use it. I’m not sure if they realize that the Windows 32-bit ABI actually mandates use of the x87 FPU (for returning values from functions).

The x87 FPU also has some slowdowns related to denormals, typically when loading and storing them.

Historically AMD has handled these special numbers much faster on their x87 FPUs, often with no penalty. However I have not tested this recently.

Performance implications on SSE

Intel handles NaNs and infinities much better on their SSE FPUs than on their x87 FPUs. NaNs and infinities have long been handled at full speed on this floating-point unit. However denormals are still a problem.

On Core 2 processors the worst-case I have measured is a 175 times slowdown, on SSE addition and multiplication.

On SandyBridge Intel has fixed this for addition – I was unable to produce any slowdown on ‘addps’ instructions. However SSE multiplication (‘mulps’) on Sandybridge has about a 140 cycle penalty if one of the inputs or results is a denormal.

Denormal slowdown – is it a real problem?

For some workloads – especially those with poorly chosen ranges – the performance cost of denormals can be a huge problem. But how do you know? By temporarily turning off denormal support in the SSE and SSE2 FPUs with _controlfp_s:

#include <float.h>
// Flush denormals to zero, both operands and results 
_controlfp_s( NULL, _DN_FLUSH, _MCW_DN ); 
… 
// Put denormal handling back to normal. 
_controlfp_s( NULL, _DN_SAVE, _MCW_DN );

This code does not affect the x87 FPU which has no flag for suppressing denormals. Note that 32-bit x86 code on Windows always uses the x87 FPU for some math, especially with VC++ 2010 and earlier. Therefore, running this test on a 64-bit process may provide more useful results.

If your performance increases noticeably when denormals are flushed to zero then you are inadvertently creating or consuming denormals to an unhealthy degree.

If you want to find out exactly where you are generating denormals you could try enabling the underflow exception, which triggers whenever one is produced. To do this in a useful way you would need to record a call stack and then continue the calculation, in order to gather statistics about where the majority of the denormals are produced. Alternately you could monitor the underflow bit to find out which functions set it. See Exceptional Floating Point for some thoughts on this, or read this paper.

Don’t disable denormals

Once you prove that denormals are a performance problem you might be tempted to leave denormals disabled – after all, it’s faster. But if it gives you a speedup means that you are using denormals a lot, which means that if you disable them you are going to change your results – your math is going to get a lot less accurate. So, while disabling denormals is tempting, you might want to consider investigating to find out why so many of your numbers are so close to zero. Even with denormals in play the accuracy near zero is poor, and you’d be better off staying farther away from zero. You should fix the root cause rather than just addressing the symptoms.

Well, you have to make sure that the video is encoded correctly, that the library works on all platforms and plays nice with your memory, file, sound and streaming abstractions, and that the audio and video doesn’t desynchronize, which for some inexplicable reason seems to be a huge problem.

But this is just technical stuff. We can deal with that. What is worse is that video playback is also a legal morass.

There are literally thousands of broad patents covering different aspects of video decompression. If you want to do some video coding experiments of your own you will have to read, understand and memorize all these patents so that you can carefully tip-toe your code and algorithms around them.

Of course, if you had a big enough pool of patents of your own you might not have to care as much, since if someone sued you, you could sue them right back with something from your own stockpile. Mutually assured destruction through lawyers. Ah, the wonderful world of software patents.

So, creating your own solution is pretty much out of the question. You have to pick one of the existing alternatives and do the best you can with it. In this article I’m going to look at some different options and discuss the advantages and drawbacks of each one:

• Just say no

• Bink

• Platform specific

• H.264

• WebM

There are other alternatives that didn’t make it to this list, such as Dirac, Theora, and DivX. I’ve decided to focus on these five, since in my view H.264 is the best of the commercial formats and WebM the most promising of the “free” ones.

An initial idea might be: Why not just do whatever it is VLC does? Everybody’s favorite video player plays pretty much whatever you throw at it and is open source software.

Unfortunately that doesn’t work, for two reasons. First, VLC:s code is a mix of GPL and LGPL stuff. Even if you just use the LGPL parts you will run into trouble on platforms that don’t support dynamic linking. Second, the VLC team doesn’t really care about patents and just infringe away. You can probably not afford to do the same. (As a result, there is a very real threat that VLC might be sued out of existence.)

A quick introduction

Before we start looking at the alternatives I want to say something short about what a video file is, since there is some confusion in the matter, even among educated people.

A video file has three main parts:

• Video data (H.264, DivX, Theora, VP8, …)

• Audio data (MP3, AAC, Vorbis, …)

• A container format (Avi, Mkv, MP4, Ogg, …)

The container format is just a way of packing together the audio and video data in a single file, together with some additional information.

The simplest possible container format would be to just concatenate the audio data to the video data and be done with it. But typically we want more functionality. We want to be able to stream the content, i. e. start playing it before we have downloaded the whole file, which means that audio and video data must be multiplexed. We also want to be able to quickly seek to specific time codes, so we may need an index for that. We might also want things like audio tracks in different languages, subtitling, commentary, DVD menus, etc. Container formats can become quite intricate once you start to add all this stuff.

A common source of confusion is that the extension of a video file (.avi, .mkv, .mp4, .ogg) only tells you the container format, not the codecs used for the audio and video data in the container. So a video player may fail to play a file even though it understands the container format (because it doesn’t understand what’s inside it).

Option 1: Just say no

Who says there has to be video in a game? The alternative is to do all cut scenes, splash screens, logos, etc in-game and use the regular renderer for everything. As technology advances and real-time visuals come closer and closer in quality to offline renders, this becomes an increasingly attractive option. It also has a number of advantages:

• You can re-use the in-game content.

• Production is simpler. If you change something you don’t have to re-render the entire movie.

• You don’t have to decide on resolution and framerate, everything is rendered at the user’s settings.

• You can dynamically adapt the content, for example dress the players in their customized gear.

• Having everything be “in-game visuals” is good marketing.

If I was making a game I would do everything in-game. But I’m not, I’m making an engine. And I can’t really tell my customers what they can and cannot do. The fact is that there are a number of legitimate reasons for using video:

• Some scenes are too complex to be rendered in-game.

• Producing videos can be simpler than making in-game content, since it is easier to outsource. Anybody can make a video, but only the core team can make in-game content and they may not have much time left on their hands.

• Playing a video while streaming in content can be used to hide loading times. An in-game scene could be used in the same way, but a high-fidelity in-game scene might require too much memory, not leaving enough for the content that is streaming in.

As engine developers it seems we should at least provide some way of playing video, even if we recommend to our customers to do their cutscenes in-game.

Option 2: Bink

Bink from RAD game tools is as close as you can get to a de facto standard in the games industry, being used in more than 5800 games on 14 different platforms.

The main drawback of Bink is the pricing. At $ 8500 per platform per game it is not exactly expensive, but for a smaller game targeting multiple platforms that is still a noticeable sum.

Many games have quite modest video needs. Perhaps they will just use the video player for a 30 second splash screen at the start of the game and nothing more. Paying $ 34 000 to get that on four platforms seems excessive.

At Bitsquid our goal has always been to develop an engine that works for both big budget and small budget titles. This means that all the essential functionality of an engine (animation, sound, gui, video, etc) should be available to the licensees without any additional licensing costs (above what they are already paying for an engine). Licensees who have special interest in one particular area may very well choose to integrate a special middleware package to fulfill their needs, but we don’t want to force everybody to do that.

So, in terms of video, this means that we want to include a basic video player without the $ 8500 price tag of Bink. That video player may not be as performant as Bink in terms of memory and processor use, but it should work well enough for anyone who just wants to play a full screen cutscene or splash screen when the CPU isn’t doing much else. People who want to play a lot of video in CPU taxing situations can still choose to integrate Bink. For them, the price and effort will be worth it.

Option 3: Platform specific

One approach to video playing is to not develop a platform-independent library but instead use the video playing capabilities inherent in each platform. For example, Windows has Windows Media Foundation, MacOS has QuickTime, etc.

Using the platform’s own library has several advantages. It is free to use, even for proprietary formats, because the platform manufacturers have already payed the license fees for the codecs. (Note though, that for some formats you need a license not just for the player, but for the distribution of content as well.) The implementation is already there, even if the APIs are not the easiest to use.

The biggest advantage is that on low-end platforms, using the built-in platform libraries can give you access to special video decoding hardware. For example, many phones have built-in H.264 decoding hardware. This means you can play video nearly for free, something that otherwise would be very costly on a low-end CPU.

But going platform specific also has a lot of drawbacks. If you target many platforms you have your work cut out for you in integrating all their different video playing backends. It adds an additional chunk of work that you need to do whenever you want to add a new platform. Furthermore, it may be tricky to support the same capabilities on all different platforms. Do they all support the same codecs, or do you have to encode the videos specifically for each platform? Do all platforms support “play to texture” or can you only play the videos full screen? What about the sound? Can you extract that from the video and position it as a regular source that reverbs through your 3D sound world? Some platforms (i.e. Vista) have almost no codecs installed by default, forcing you to distribute codecs together with your content.

Since we are developing a generic engine we want to cover as many platforms as possible and minimize the effort required to move a project from one platform to another. For that reason, we need a platform independent library as the primary implementation. But we might want to complement it with platform specific libraries for low end platforms that have built-in decoding hardware.

Option 4: H.264 (MPEG-4, AVC)

Over the last few years H.264 has emerged as the most popular commercial codec. It is used in Blu-ray players, video cameras, on iTunes, YouTube, etc. If you want a codec with good tool support and high quality, H.264 is the best choice.

However, H.264 is covered by patents. Patents that need to be licensed if you want to use H.264 without risking a lawsuit.

The H.264 patents are managed by an entity known as MPEG LA. They have gathered all the patents that they believe pertain to H.264 in “patent pool” that you can license all at once, with a single agreement. That patent pool contains 1700 patents. Yes, you read that right. The act of encoding/decoding a H.264 file is covered by 1700 patents. You can find the list in all its 97 page glory at http://www.mpegla.com/main/programs/avc/Documents/avc-att1.pdf.

I am not a lawyer, as they say on Slashdot, but this is my best understanding of how this patent game works:

• Buying a license from MPEG LA gives you the right to use the 1700 patents in the pool.

• This doesn’t mean you can’t be sued for patent infringement. Anyone that holds a patent which is not one of the 1700 in the pool could claim that H.264 infringes on it and sue you. That seems unlikely, MPEG LA has made an effort to gather all relevant patents, but there is no way to be certain.

• MPEG LA doesn’t by itself go after people who use H.264 without a license, that is up to the holders of the 1700 patents in the pool.

The licensing terms of H.264 are irritating, but not necessarily a big financial burden:

• If you distribute an encoder or decoder you can distribute 100 000 copies for free, then you have to pay $ 0.20 per unit.

• If you distribute a H.264 encoded movie, it is free if it is shorter than 12 minutes, then you have to pay $ 0.02 per copy.

Note that unlike the case with other popular codecs such as MP3, it is not just the decoder/encoder that you need to license, you also need a license just for distributing H.264 content.

From what I’ve been able to discern, but don’t take my word for it, a game that only plays a fixed set of movies/cutscenes would not be regarded as a general decoder (even though it contains decoding software), but rather as content, which means you would pay $0.02 per copy sold if you had more than 12 minutes of video in your game and nothing otherwise (you would still need to obtain a license though).

Of course, if you support H.264, you may also want to support AAC, the standard audio format that accompanies it. AAC is covered by a separate licensing body (Via Licensing) that has its own licensing terms. I haven’t investigated them in any great detail.

You have to decide for yourself how well these terms sit with you. At Bitsquid we finally decided that if we should have a standard video playing facility, it should be one that people could use without thinking too much about patents and licensing (to the extent that is possible).

Option 5: VP8 (WebM)

VP8 is a “free” video codec owned by Google. It is covered by patents, but Google has granted free use of those patents and also provides a BSD licensed library libvpx for encoding and decoding video files. The format is also endorsed by the Free Software Foundation.

It is generally acknowledged that when it comes to quality, VP8 is not quite as good as H.264, though the differences are not enormous. So you are trading some visual quality for the convenience of a license free format.

There has been some discussion (most of it about a year ago) about whether VP8 really is unencumbered by patents. MPEG LA claimed that it knew several patents that VP8 was infringing and showed interest in creating a “patent pool” for VP8 similar to the one it holds for H.264. Nothing has come of that yet and MPEG LA has not disclosed which patents it thinks VP8 is infringing, which means that Google cannot really respond to the allegations. It is hard to know how much stock to put in this and whether anything will come of it.

You could argue that with this potential “threat” against VP8, it would be better to use another “free” alternative such as Dirac or Theora. However, there is not much evidence that they would fare better. Everyone who makes a “free” codec tries their best to make sure that they don’t infringe on any patents. But with thousands of these patents around, each open to legal interpretation, there is just no way to be sure.

This is just the sad state of affairs of software patents. And you are not safe with the commercial formats either. Even if you have licensed the 1700 patents in the MPEG LA patent pool, someone can still sue you for violating patent number 1701. No one in this business offers indemnification against patent suits. Not Google. Not MPEG LA. Not Bink (I think).

It all becomes a question of risk. Bink has been around a long time without being sued, which is reassuring. VP8 hasn’t been around that long.

Will there be patent claims made against VP8? Maybe. Who knows. Similar threats were made against Theora, but nothing happened there. If it does happen, Google will most likely fight back and the whole thing will drag on in the courts. Will there be patent claims against you for using VP8? Seems extremely unlikely. Games are not interesting enough targets. Video decoding is not our main business, and we can easily switch technology if needed. Phone manufacturers, YouTube and TV companies are more interesting targets.

Do you have to care about any of this at all? Up to you to decide.

Our conclusion

None of these alternatives are really attractive, it is more about picking the least worst than finding the best, which is frustrating for a perfectionist with self-worth issues. Good cases can be made for all of them. This is what we have decided:

• We will provide VP8 decoding on all platforms through libvpx. All other things equal, the “most free” format will give us most flexibility in the long run.

• We will not (at least not right now) support matroska or other advanced container formats. Instead, we will play the videos from simple IVF streams. Sound will be played as Vorbis files through our regular sound system so we get positioning, reverb, etc.

• If needed, we will complement this basic approach with platform-specific libraries that take advantage of the hardware decoders on low-end platforms (phones and tablets).

• Customers that need to play a lot of movies while doing other CPU intense tasks and that aren’t happy with the performance of libvpx are recommended to look into Bink.

• Customers that are worried about “patent risk” with VP8 are recommended to do whatever their lawyers tell them to do. (Use Bink, a platform specific library, obtain a H.264 license or avoid video all together.)

This has also been posted to The Bitsquid blog.

Gaming culture has a problem and that problem has a lot to do with gamers themselves.  To be clear, I’m not talking about all gamers but rather a subset of gamers whose antisocial behaviour and habits drive people away from gaming.  Analysts at Piper Jaffray recently conducted a survey that found nearly 66% of high school students surveyed across the US claimed they were losing interest in traditional videogames with slightly over 66% stating they were interested in social, mobile games which was an increase from 34% who answered the same question the year prior.  Gaming as we know it is changing for a variety of reasons and one of those reasons is gamers have chosen to turn on each other as well as the people who make the videogames they play.  While gaming culture tries to evolve and leave the primordial seas, certain gamers are busy running along the shore with sharpened sticks trying to force us all back in.

The problem lies with the internet and the anonymity it affords its users.  This effect certainly isn’t limited to just gaming circles but as gamers tend to be a largely wired group of individuals the impact is pronounced.  Gaming has always had a social side but over the decades that’s changed, and you could certainly argue, not for the better.  Back in the 80’s and 90’s, gamers would flock to arcades or journey to friend’s houses to partake in the hottest, latest releases.  To illustrate how it’s changed, imagine four friends over for an afternoon session of GoldenEye sitting in their family den.  Now imagine one of those children lets loose with a barrage of profanity laced, racist, homophobic rants aimed at his fellow gamers.  Or imagine someone’s little sister is also invited to play and subjected to a stream of masturbation and rape jokes.  There’s a very good chance that the child would simply never be invited back for another GoldenEye marathon.  There’s also a chance that little Jimmy’s mother, having overheard the obscene rant would never allow that child in her house ever again and would make a quick phone call to inform the offending child’s parents of their unacceptable behaviour.

This type of antisocial behaviour infects network gaming and social interactions across the internet and therein lies the difference between gaming culture now and gaming culture then.  There are few, if any, social repercussions in gaming today and the impact of these behaviours eats at the fun factor of gaming for a large number of gamers, children and adult alike.  It doesn’t matter if you’re looking to game socially on your console or if you’re looking to partake in a general gaming discussion on the internet, odds are your experience will be sullied by another gamer hiding behind their internet pseudonym.

Researchers refer to this as “toxic disinhibition”.  The anonymity the internet and online gaming networks offers often results in the complete abandonment of social restrictions that would generally be present in face to face interactions such as in the days when we gamed locally with other people in the room.  The result of this “trolling” is we see more and more gamers being turned off of gaming, or seeing their enjoyment of games lessened, thus inhibiting the growth of gaming culture.  The impact of this online disinhibition also affects developers who can find themselves loathe to engage their own fan base for fear of fanboy backlash through internet flaming.

Recently, gamers made headlines for their disproportionate backlash against Mass Effect 3 developer Bioware.  The actions of certain gamers painted all gamers as whiney, entitled children prone to screaming fits when denied their pacifier.  Bioware found itself facing a FTC complaint while its writers and staff were targets of hate campaigns and death threats as some found the end of their latest game offering to be unsatisfactory.  Other gamers and non-gamers alike shook their collective heads in disbelief.  Blizzard also suffered a ridiculous backlash from gamers when screenshots of their now released Diablo 3 title were deemed “too bright” by some prior to the game’s launch, prompting Blizzard to mock the users by releasing screenshots containing unicorns shooting rainbows from their posteriors.  While Blizzard used the experience to have a bit of fun, the example illustrates a growing trend among gamers to instantaneously and viciously attack developers and other gamers alike for even perceived slights and as a whole, the gaming community becomes a less inviting place.

The online disinhibition effect certainly isn’t limited to gaming forums either as the development community itself isn’t immune from unprofessional behaviour.  I’ve seen my own personal blog postings regarding my development experiences targeted by other developers leveling harsh and often unfair criticisms.  For example, I’ve had a developer lambast the simply inclusion of our company logo on a splash page because “no one cares about your company”.  I’ve even had a local developer I didn’t know and had never met criticize my company online for the slight of not consulting with their group prior to launching our first game.  This type of challenging behaviour is far more likely to be witnessed online than in face to face interactions or official business communication and unfortunately it is becoming more prevalent.

They say in general you need a thick skin to blog but backlash I received from a recent blog post made me question the value of blogging my own experiences as a developer.  I posted a personal blog listing some of the complete, all in one game engines available that may be of interest to independent developers.  While researching game engines for my company I would have found such a blog useful as I looked for a game engine that offered features I required, such as Android and iOS porting and clearly the blog was not meant as an in depth review piece.  As I had not the opportunity to try each engine I listed, I made sure to note that where I was unfamiliar with the engine I was simply relaying information and opinions from various reviews I had come across and I provided links to each product so users could conduct further research.  Rather than promote a thoughtful discussion on the merits of various game engines as I had intended or to provide a starting point for further research, the resulting comments were almost all attacks against myself personally and my attempt to inform other indie developers.

The comments included people calling me a liar, posters comparing my blog to vulgar activities, writers incensed that I didn’t whole heartedly endorse their particular favorite engine.  I even read claims that I was intentionally trying to harm product reputations despite the fact I noted these opinions were sometimes not even mine but were simply being passed along when I lacked particular knowledge of the product being discussed.  I was frankly shocked by the lack of decorum I witnessed in response to a personal blog intended to simply inform and facilitate further research, and if other developers hadn’t contacted me directly to offer their support (with one commenting he would have done it publicly if not afraid of being “flamed” himself) I most likely would have never written another blog regarding my game development experiences.  This general lack of professionalism in a workplace environment would never be tolerated.  Indeed, when gamers and developers are afraid to share ideas due to fear of reprisal it’s time to take a hard look at the current situation and what repercussions this could have to our industry as a whole.  As a community, this type of behaviour should not be allowed to propagate.  I’ve been subjected to all manner of hate mail and threats from casual gamers and stalking fanboys alike over the years writing opinion pieces regarding the games industry, however the lack of professionalism I’ve witnessed since becoming a developer myself truly surprised me.

Gaming culture is suffering due to experiences like this, due to experiences like those Bioware recently endured and due to the ongoing profane, racist and homophobic behaviour tolerated every day in online gaming matches and in internet gaming forums.  The anonymity of the internet mixed with complacency among gamers and developers has led to this situation and the associated cyberbullying that goes along with it but as the genie is out of the bottle with regards to the internet, there is little that we can do to curb its impact.  The removal of anonymity in online gaming by the companies that operate these networks could potentially result in fewer incidents as people are less inclined to act in socially unacceptable manners when their real names and locations are attached to their actions, however this system would still rely on reporting tools that already exist are underutilized by the majority of gamers.  Most prefer to simply ignore the problem, and this does nothing to stem the rise of anti-social behaviour in the gaming community.

As more teens are turning towards social gaming where they can exercise more direct control over their social interactions through use of things like Facebook friend lists, as more potential gamers are turned off by what they see of gamers in the news and more core gamers turn away from online gaming and game forums based on the sliding social environment, today’s gaming culture must change or it will face decline.  We’re already seeing traditional game sales slide as gamers look elsewhere and a shift towards mobile games is evident.  Partial blame falls on gamers themselves for creating and tolerating an increasingly toxic game culture that runs contrary to the social spirit that videogames created for many of us while gaming in the 80’s and 90’s, and even some developers themselves are letting professionalism standards slump in their online communications which itself is the start of a slippery slope.  We can never go back to the way gaming was, but we can shape the future of gaming culture for the better by being conscious of where it went wrong and why.

The traditional web model is request/response based. The client requests a resource from the server, the server responds with some data. That’s the end of the transaction. In this model there is no way for the server to update the client without first receiving a request from that client. If you want to build a multiplayer game then this is likely to be very important.

It can be simulated either by regular polling from the client (does not scale well), by some fairly hacky techniques falling under the umbrella of Comet, or by the new WebSockets technology available recently. Google App Engine does not support WebSockets, but it does support Comet under the guise of the Channel API.

Test App Here!
http://channelapistresstest.appspot.com/

I was interested in the performance of the Channel API so I wrote a small test app. I’m a bit of a noob when it comes to Python and HTML/Javascript but if you want to take a look I uploaded my code here.

Overall I was disappointed with the results, and indicates to me that if you want any kind of fast response game then you may have to look elsewhere.

• Very unpredictable latency (200-1000ms)
• Big packet overhead of HTTP headers etc
• No peer to peer support at all (limitation of current web technologies)
• No support for broadcast messages (server must send to each client in separate call)
• Sending several messages at once often results in a massive increase in latency

That said I do think the technology is quite promising and certainly opens up the possibility of doing whole classes of games with two way communication. Strategy games, or other games where one player does not directly affect another. Adding chat for instance is a perfect use case for this. If you are thinking about using Channel then I found out a few things that might be useful to you.

• Reliable – in all my tests all messages eventually arrived
• Out of order – messages are not guaranteed to be in sequence so if this is important you need to handle your own sequence checks
• Multiple instances – don’t forget the service may spin up multiple instances to handle the requests
• Development server is much worse than the live server (on Windows at least)! It seems to serialize all messages sent with approximately 600ms between each one.

I checked the latency from my office in the UK and found it to be very unpredictable and often slow. If anyone from other parts of the world could test and report back your average latency that would be awesome! It’s entirely possible that I’ve screwed something up which is causing the poor performance, if anyone has any comments on this it would be great to hear them.

Addendum

After writing this I became interested in comparing the Channel API with Websockets. I took this opportunity to play with node.js and hack up a quick server. I found a great host in nodejitsu who provided one free app and a fantastic tool to deploy your app. Unsurprisingly results are much better than with Channel. My average latency tends to hang around 200ms, but there are no spikes or degradation when sending a lot of messages.

You can test my quick and dirty app at http://echo_test.jit.su/ (sorry it only works in Chrome/Firefox!). You can see the source here.

It is not really a fair comparison, because the back ends are designed for different purposes. The GAE server uses persistent storage to support multiple instances, whereas the node.js version will not scale. The Channel API is also fully supported on all browsers/platforms whereas support for WebSockets is still fairly thin on the ground.

This is a cross post from my blog.

Other articles in this series:

Storing Data In SQL Server

Let’s imagine that we’re building the data layer for a multiplayer game. The data layer probably contains tons of different persistent storage devices – user logs, game activity diagnostic data, auction houses, persistent user data, and so on. For today’s lesson we’ll create a database of item drops from monster kills.

To start I will assume you’re not using an ORM system like Hibernate or Entity Framework. Many people like ORMs since they abstract away most of the database work. But doing so means you also lose the ability to fine tune your performance requirements, and we’re just going to skip that part. There’s lots of good debate out there about why ORM systems aren’t always the right idea, so proceed there at your own risk.

Revision One: A Bad Design

I’ll present, first of all, the kind of database design we may all have seen at one time or another. Here’s a table that’s not really optimized for massive data storage; it’s ugly but workable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

CREATE TABLE item_drops_rev1 (
 
    -- This is the reference to the Items system ID number; it's not a foreign key 
    -- because the "Items" table is on a different database server
    item_id BIGINT, 
 
    -- This is the internal monster class name, e.g. "Creature.Wolf" or "Player.NPC"
    monster_class VARCHAR(50), 
 
    -- The Zone ID refers to a database on the same server, so we built it as a 
    -- foreign key
    zone_id INT FOREIGN KEY REFERENCES world_zones(zone_id),
 
    -- The position and time where and when the kill happened
    xpos REAL, 
    ypos REAL,
    kill_time datetime DEFAULT GETDATE()
)

We can probably live with this table structure. It’s not normalized, but it gets the job done. I’ll explain in a moment what we can do to improve on it.

On the other hand, for the client code, I’m going to show you the wrong way to get things done: I’ll create a SQL injection vulnerability. Any web-facing application or service layer is at risk for SQL injection, and frankly it’s easy to overlook something and get lazy and only discover your server hacked a month later. If you ever see this kind of code, this is by far the worst thing you can ever do:

1
2
3
4
5
6
7
8
9
10
11
12

SqlConnection conn = new SqlConnection(my_connection_string);
conn.Open();
// This is a massive SQL injection vulnerability - don't ever write your own SQL statements with string formatting!
String sql = String.Format(@"INSERT INTO item_drops 
    (item_id, monster_class, zone_id, xpos, ypos) 
    VALUES 
    ({0}, '{1}', {2}, {3}, {4}, {5})", item_id, monster_class, zone_id, xpos, ypos);
SqlCommand cmd = new SqlCommand(sql, conn);
cmd.ExecuteNonQuery();
// Because this call to Close() is not wrapped in a try/catch/finally clause, it could be missed if an 
// exception occurs above.  Don't do this!
conn.Close();

What’s bad about it? Do you see how the SQL statement is being assembled by a string parser? SQL statements are basically dynamic code. You would never let a random user type code into a web browser and execute it on your server, so don’t do the same thing with your database. Because SQL as a language has lots of features for combining multiple statements, someone could write a malicious monster_class string that would have terrible side effects. A simple attack might be “‘;UPDATE players SET gold=gold+10000;–”, but there are lots of other more subtle attacks too. SQL Server even has a feature called xp_cmdshell() that can allow a SQL injection vulnerability to turn into a sitewide security breach.

So don’t ever write your own SQL statements using string parsers; you instead want to use SQL Parameters. There’s also a little improvement we can make to avoid leaving dangling connections – the using statement. When the .NET execution environment exits a “using() { }” clause, it automatically disposes all resources that were obtained in the initialization. So those two items said, here’s what good code looks like:

1
2
3
4
5
6
7
8
9
10
11
12

using (SqlConnection conn = new SqlConnection(my_connection_string)) {
    conn.Open();
    string sql = "INSERT INTO item_drops_rev1 (item_id, monster_class, zone_id, xpos, ypos) VALUES (@item_id, @monster_class, @zone_id, @xpos, @ypos)";
    using (SqlCommand cmd = new SqlCommand(sql, conn)) {
        cmd.Parameters.Add("@item_id", item_id);
        cmd.Parameters.Add("@monster_class", monster_class);
        cmd.Parameters.Add("@zone_id", zone_id);
        cmd.Parameters.Add("@xpos", xpos);
        cmd.Parameters.Add("@ypos", ypos);
        cmd.ExecuteNonQuery();
    }
}

Besides being easier to read, this approach guarantees that SQL injection will not be a problem. So this is where I’ll start.

This code, using the SQLCommand and parameters, gets the job done and it’s fast. Running a test on an old laptop I have around, I was able to insert 10,000 records in 7.768 seconds. This kind of performance is good enough for most companies; running on a proper server it would be 3-4 times faster and everyone would be happy. For a business application this code is fine, easy to maintain, and reliable.

That said, we’re a gaming company, and what works for us is a bit different than the code used by an accounts payable system that processes a few tens of thousands of records per day. You’re probably going to put this code up in front of a web service that gets called every time a zone server in your game records a kill, so maybe 10,000 kills in 7-8 seconds isn’t good enough. Let’s see what we can find to make improvements.

Performance Limitations Of Revision One

In order to make this database insert process fast, let’s start looking at what happens when you execute this command. I’ll overlook all the internal stuff that we don’t have control over, and show you instead a list of a few problems this code suffers from:

• Connection time. When you execute “conn.Open()”, the client computer must parse the connection string, identify the SQL server, open a channel, and begin a dialogue; this can take up to a second or two. Fortunately, if you’re using C#, the .NET environment will automatically use connection pooling to retrieve a previously opened connection with an identical connection string. If you’re not using C# for your client library, make certain that your environment provides pooling!
• Network latency. Your data layer server has to communicate with your SQL server, and each packet it sends has a round trip time. It’s possible to run SQL server on the same physical machine as your data layer, but it’s not recommended. Ideally, you’d create a dedicated SQL network for reduced overhead and increased security. With modern switches it’s very easy to configure a VLAN to do this, and most servers nowadays have lots of spare ethernet ports.
• Parser overhead. SQL Server sees the string “INSERT INTO …” and has to evaluate it into actual work. The parser is the bit of code that turns the sql string into a “plan” – basically a list of function calls that the server will execute internally. The SQL Server Parser is so fast its timing is often negligible; but we can still help cut down on its work by using stored procedures instead of straight SQL.
• Database contention. To provide ACID compliance, SQL Server has to ensure that all database changes are serialized. It does this by using locks to ensure that database inserts don’t conflict with each other. Most SQL Servers are multi-user machines: in addition to your “INSERT” statement, the SQL server is probably handling dozens or hundreds of other clients at the same time. Maybe client A is inserting item drop records, while client B is running reports on these records, and client C is sending stats to the community manager in realtime.

  Well, when SQL Server is doing all of these things, it generates locks to ensure that A, B, and C each receive sensible results. SQL Server guarantees that each statement is executed while the database is in a fully predictable state, and managing those locks takes time. Additionally, if SQL Server is just overloaded with too many clients, it doesn’t matter how well written your query is – it will be slow! You need to take a holistic look at the server’s load levels, and continue to monitor each and every client that connects to the server to see where optimizations can take place.

• Constraint processing. When inserting data, each constraint (a foreign key, or a default, or a “check” statement) takes a non-zero amount of time to test. SQL server guarantees that each record inserted, updated, or deleted will continue to satisfy all constraints on the table. Do you really need foreign keys for large, high volume tables?
• Page faults. While executing the query, SQL Server needs to pull all the required information into memory. If SQL is under memory pressure, it will have to discard some pages and reload them from disk. Make sure to give it as much RAM as you can afford, and design your tables to reduce each record’s size. Be careful when designing your indexes; too many indexes means lots of paging. But of course, buying RAM is far cheaper than anything else you can do, and it’s a one-time cost, unlike that Hadoop cluster on Amazon EC2 that you keep paying for each month.
• Varchars. Varchars are essential tools, but they can also lead to lots of unexpected overhead. Each time you store a variable length column, SQL Server has to do more memory management to pack records together. Strings can easily consume hundreds of bytes of memory each. If you put a VARCHAR column in an index, SQL Server has to execute tons of O(string length) comparisons as it searches through the B-tree, whereas integer compare instructions are limited only by memory latency and processor frequency.
• Data storage. You can also help to optimize the amount of work SQL Server must perform by shrinking your data size. When inserting data, normalizing your database helps reduce the amount of data written to disk; but when querying, denormalizing helps reduce the number of tables that need to be joined. You should ideally wage constant war between normalizing and denormalizing; don’t ever let yourself get stuck on one side of the fence or the other.

After all this overhead, your query will return. In most cases, the query is so fast you won’t even realize all this overhead happened. However, there’s one more thing to keep in mind:

• Disk IO. SQL Server will eventually write the data you’ve just inserted to disk. In SQL Server, data is first written to a transaction log, then the transaction log is merged with the permanent database file when a backup is executed. This happens in the background, so it won’t delay your query directly; but each transaction that occurs must have its own lump of disk IO. You can reduce this overhead by giving SQL server physically separate disks for transaction logs and main DB files. Of course, the best solution is to actually reduce the number of transactions you execute.

As you can see, there’s a lot to consider when optimizing SQL. The good news is that these limitations are manageable, provided you pay attention to them.

Revision Two: Free Improvements

Given what we’ve learned, let’s rewrite the SQL Insert command and see what benefit we can get. First, we can normalize our database by moving “monster_class” into its own table. We can then load into our client application the list of monster classes and use a hashtable to find the ID number to insert into the database directly. That reduces the amount of data stored in each record, and it makes each record identical in size. It offloads some of the work that SQL server has to perform and distributes it to the application tier.

With these improvements, we’ll see a reduction in data storage size per record:

The old vs new record layouts

These changes cut down the size of the record noticeably. The original record used to take between 41 and 91 bytes, probably averaging 60 or so bytes per record; the new record takes exactly 35 bytes. Since SQL server stores records in 8K pages, that means that you can now store over 200 records per page, whereas the old system could only hold about 130. This increases the number of rows you can insert before SQL Server succumbs to memory pressure, it speeds up your inserts, and since each record is a fixed length, SQL Server reduces the amount of offset calculation work that’s done to rapidly scan through records in memory.

Next, let’s eliminate our constraints. This should be done when you’ve tested your code and know that removing the constraint won’t make your data integrity suffer. In the initial table, we had two constraints: a foreign key and a default value. If we’re not really worried about ensuring that each zone ID is rigidly correct, we can forego the foreign key. That means for each insert we no longer need to test the zone database to ensure each ID exists. For a business application, that kind of foreign ID testing is valuable; but for my game I’ll just write a test script to check once per day that no bad zone IDs are being written.

Finally, I’ll replace the custom SQL text with a stored procedure. This can potentially reduce parser overhead, although our program is simple enough that the parser overhead is pretty low and highly likely to be cached anyways.

The revised table, stored procedure, and client program is now capable of inserting 10,000 records into my ancient laptop in about 6.552 seconds, for a performance boost of ~15%. The code now looks like these snippets:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

CREATE TABLE item_drops_rev2 (
    item_id BIGINT,
 
    -- This is now a pointer to a lookup table
    monster_class_id INT, 
 
    -- Zone no longer has a FOREIGN KEY constraint.  SQL Server will allow bad
    -- values to be loaded, and it's our responsibility to handle them.
    zone_id INT, 
    xpos REAL, 
    ypos REAL,
 
    -- No longer has a DEFAULT constraint; this means we have to insert the date
    -- ourselves, but it reduces the work on SQL server
    kill_time datetime 
)
 
-- This procedure allows SQL server to avoid virtually all parser work
CREATE PROCEDURE insert_item_drops_rev2
     @item_id BIGINT, @monster_class_id INT, @zone_id INT, @xpos REAL, @ypos REAL 
AS
 
INSERT INTO item_drops_rev2 
    (item_id, monster_class_id, zone_id, xpos, ypos, kill_time) 
VALUES 
    (@item_id, @monster_class_id, @zone_id, @xpos, @ypos, GETDATE())

1
2
3
4
5
6
7
8
9
10
11
12
13
14

using (SqlConnection conn = new SqlConnection(my_connection_string)) {
    conn.Open();
    using (SqlCommand cmd = new SqlCommand("insert_item_drops_rev2", conn)) {
 
        // Setting the command type ensures that SQL server doesn’t need to do any complex parsing
        cmd.CommandType = CommandType.StoredProcedure;
        cmd.Parameters.Add("@item_id", item_id);
        cmd.Parameters.Add("@monster_class_id", monster_class_id);
        cmd.Parameters.Add("@zone_id", zone_id);
        cmd.Parameters.Add("@xpos", xpos);
        cmd.Parameters.Add("@ypos", ypos);
        cmd.ExecuteNonQuery();
    }
}

This is a great set of simple improvements. However, there’s one more change you can make, and that’s the big one.

Revision Three: Lazy Writes

Think about your data requirements. Is it essential that every record be written to the database immediately? Can it wait a minute? Five minutes? What would the penalty be if a record was lost? How many records could you lose before your business failed?

For a bank, losing any monetary transactions can be life or death; but for a game you can define your own risk level. Let’s imagine that, for our monster drop table, we only want to risk losing at most five minutes’ worth of data. This gives us an incredible opportunity for performance improvement: we’ll keep a list of records to insert and batch-insert them once every five minutes! SQL Server offers us a great way to reduce overhead when we batch up work like this: the Transaction.

A transaction is basically an atomic lump of work. SQL Server guarantees that each transaction will either succeed and be written to the database permanently, or fail completely and no data will be written to the database. This sounds complicated, but keep in mind that transactions also allow SQL Server to forego lots of extra overhead. If you execute ten statements all by themselves, you get ten lock-based overheads; but if you wrap them in a transaction you only have to do your lock-based overhead once.

The only downside to batching is that we can’t use GETDATE() anymore, since records may be inserted a few minutes after they were generated. Instead we must preserve all the data for all records in memory while waiting for the time to insert a batch. The new code looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

using (SqlConnection conn = new SqlConnection(my_connection_string)) {
    conn.Open();
 
    // This transaction tells SQL server to obtain locks once and keep them on hand until the transaction is committed
    SqlTransaction trans = conn.BeginTransaction();
    for (int i = 0; i < INSERT_SIZE; i++) {
        using (SqlCommand cmd = new SqlCommand("insert_item_drops_rev3", conn)) {
 
            // Setting the command type ensures that SQL server doesn’t need to do any complex parsing
            cmd.CommandType = CommandType.StoredProcedure;
 
            // Joining a transaction means that SQL Server doesn't have to close and re-open locks
            cmd.Transaction = trans;
            cmd.Parameters.Add("@item_id", item_id[i]);
            cmd.Parameters.Add("@monster_class_id", monster_class_id[i]);
            cmd.Parameters.Add("@zone_id", zone_id[i]);
            cmd.Parameters.Add("@xpos", xpos[i]);
            cmd.Parameters.Add("@ypos", ypos[i]);
            cmd.Parameters.Add("@kill_time", kill_time[i]);
            cmd.ExecuteNonQuery();
        }
    }
 
    // This statement tells SQL server to release all its locks and write the data to disk.
    // If your code had thrown an exception above this statement, SQL Server would instead
    // do a "rollback", which would undo all the work since you began this transaction.
    trans.Commit();
}

With this change, we’re now able to insert 10,000 records in 2.605 seconds. That’s about a 66% performance improvement – pretty major! Even more importantly, wrapping sensible objects into a transaction significantly reduces database contention and can really alleviate concurrency bottlenecks. Of course, if we weren’t batching these inserts together, adding a transaction is a slight performance penalty; but whenever you’re grouping lots of commands into one transaction you’ll cut down overhead.

Revision Four: Table Parameters

You can see that the above approach really saved us a ton of time. However, to insert 10,000 records into the database we’re still contacting the database 10,000 times. In fact, each call to “cmd.ExecuteNonQuery” generates a roundtrip message from your client application to the database. What if there was a way that we could insert all ten thousand records, but only contact the database server once?

The good news is that SQL Server 2008 introduced an incredible new capability called “Table Parameters”. Table Parameters work by grouping tons of records together into a single parameter into a stored procedure or SQL statement. This essentially converts overhead performance penalties from O(number of records) to O(number of times you batch insert). Additionally, by reducing the number of SQL commands being executed, you dramatically reduce database contention and improve performance for other programs.

Here’s the final insert code including table parameters. You may notice that I’ve removed the BeginTransaction() and Commit() calls – those only boost our performance when we’re doing more than one ExecuteNonQuery() command at a time. So here goes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

CREATE TYPE item_drop_bulk_table_rev4 AS TABLE (
    item_id BIGINT,
    monster_class_id INT,
    zone_id INT,
    xpos REAL,
    ypos REAL,
    kill_time datetime
)
 
CREATE PROCEDURE insert_item_drops_rev4
    @mytable item_drop_bulk_table_rev4 READONLY
AS
 
INSERT INTO item_drops_rev4 
    (item_id, monster_class_id, zone_id, xpos, ypos, kill_time)
SELECT 
    item_id, monster_class_id, zone_id, xpos, ypos, kill_time 
FROM 
    @mytable

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

DataTable dt = new DataTable();
dt.Columns.Add(new DataColumn("item_id", typeof(Int64)));
dt.Columns.Add(new DataColumn("monster_class_id", typeof(int)));
dt.Columns.Add(new DataColumn("zone_id", typeof(int)));
dt.Columns.Add(new DataColumn("xpos", typeof(float)));
dt.Columns.Add(new DataColumn("ypos", typeof(float)));
dt.Columns.Add(new DataColumn("timestamp", typeof(DateTime)));
 
for (int i = 0; i < MY_INSERT_SIZE; i++) {
    dt.Rows.Add(new object[] { item_id, monster_class_id, zone_id, xpos, ypos, DateTime.Now });
}
 
// Now we&#039;re going to do all the work with one connection!
using (SqlConnection conn = new SqlConnection(my_connection_string)) {
    conn.Open();
    using (SqlCommand cmd = new SqlCommand("insert_item_drops_rev4", conn)) {
        cmd.CommandType = CommandType.StoredProcedure;
 
        // Adding a "structured" parameter allows you to insert tons of data with low overhead
        SqlParameter param = new SqlParameter("@mytable", SqlDbType.Structured);
        param.Value = dt;
        cmd.Parameters.Add(param);
        cmd.ExecuteNonQuery();
    }
}

And what does the performance of this logic look like for inserting 10,000 records? We’re now down to 0.218 seconds, an improvement of over 97% from our initial attempt.

Conclusions

Although this calculation isn’t perfect, I’m going to suggest the following estimates for the overhead we encountered in this test. Out of a total 97% speedup we obtained, I’m guessing the breakdown goes roughly as follows:

• First priority: Reducing contention and removing unnecessary locks (a 51 percentage point gain)
• Second priority: Reducing the general SQL statement processing overhead by combining multiple queries together (a 31 percentage point gain)
• Third priority: Removing unnecessary constraints, VARCHARs, and SQL parser overhead (a 15 percentage point gain)

As you can see, you don’t need to abandon SQL Server to get massive performance improvements. However, it’s easy to forget about SQL Server performance because the technology is so effective. We use database wrappers and database abstraction layers that speed our development time, but separate us from the fine details. If you want to get lots of high performance work out of SQL server, use the same process you’d do if you were tuning C++ code: profile it, track the most frequently called functions, optimize the heck out of them, and finally rethink your work so you call them less frequently.

Finally let me pass some credit to Tom Gaulton and Rich Skorski who provided valuable feedback in the editing of this article.

A Brief History of Freemium

Take away the freemium model and you’re left with users having to pay for games up front. With traditional disk-based media that worked, the barrier to entry meant that there were relatively few titles on sale so it was possible to get a game noticed given a decent advertising push and by word of mouth – and that meant publishers could afford decent budgets and take a few risks. The business model wasn’t perfect, but it sufficed. Innovation wasn’t always top priority, but the quality bar was fairly consistent.

Fast forward to the age of downloads, in particular the AppStore, and the barrier to entry for developers was suddenly much lower – almost anyone with a computer and a compiler could release a game. Initially this seemed like a good thing, there was much excitement about “indie gaming” and a few notable success stories, but soon the market was flooded and developers began undercutting each other on price in order to get sales.

This process, commonly dubbed “the race to the bottom”, now seems to have run its course, with the majority of games now in the $1-2 bracket, and that has warped the general perception of value. Paying $5 for a game may have seemed cheap once but now looks expensive against the backdrop of bargain bucket titles – despite the fact that people will happily fork out more than that for a latte and a sandwich. As a result it has become incredibly risky to invest in making a high quality game for the mobile and tablet market. Even if your $5+ game is awesome, the chances of enough people finding it and paying for it are slim. They’re more likely to download half a dozen separate dollar titles and conclude that all mobile games are rubbish.

That’s where freemium enters the frame. If you can give your game away for free there’s nothing (bar the initial download) to stop people trying it out – sure, the market is still flooded with choice, but you’ve improved your chances dramatically. Once you’ve got people hooked on your game, then you can start billing them for items, and make your money that way instead – and because you’ve already hooked people in you can charge them a lot more than the minimum app price that people have come to expect for paid apps. Recently we’ve seen a few games exploit this system to make large sums of money.

So what’s the problem?

Notice I used the word “exploit” in the last sentence? Well, that’s the problem with freemium at the moment. Games don’t simply draw you in and then ask for a single payment to continue playing; they attempt to sell you a whole plethora of in-game items, often using devious means that encourage addictive behaviour and disguise the true cost. A while back I played a game called Paradise Island on Android that was a perfect example; you could buy special buildings for your island, but the way you did so involved so many different forms of currency along the way that it wasn’t immediately obvious what the cost would be. I once sat down and calculated that the true cost of a single building in their Halloween pack was a staggering $50.

It’s this addiction-feeding style of game which I believe has given freemium such a bad name. After all, if you look back to the era when magazine cover-disks were the norm you realise that freemium has been with us in spirit for decades, even if the buzzword hasn’t.

Can the problem be fixed?

I think we’re beginning to see light at the end of the tunnel. Some recent games have started to implement less aggressive forms of in-app purchasing – for example Triple Town works by limiting the free version to a fixed number of ‘turns’ per day, and gives you the option to purchase unlimited turns for a fixed fee. It still has the option to buy items to essentially cheat your way to victory, but at least you can opt out of that and make a one off payment. It’s a step in the right direction.

Perhaps more importantly though, the Japanese courts have just stepped in to ban the “kompu gacha” mechanic, whereby players pay a small fee for the chance of unlocking a rare item (in essence, a lottery), but can only do so after unlocking a whole set of other items more common items through the same mechanic. It’s only one specific mechanic that they’ve banned, and it’s only the tip of the iceberg, but the case has drawn a lot of attention from the press over the last few days, and looks like it might be the tipping point that wakes regulatory bodies, and players themselves, up to the devious way in which they’ve been manipulated.

Nice Freemium

Much as the makers of yoghurt products keep telling us there “bad bacteria” and “good bacteria”, I believe there can also be “evil freemium” and “nice freemium”. The evil freemium games are built purely to extort money from their players, while nice freemium are good honest fun, that just happens to come in a freemium package.

I doubt we’ve seen the back of evil freemium just yet, but rather than despair at the state of the freemium market, why not try making your own game freemium – just make sure it’s nice freemium.

I also really love having a cerabal cortex, so I left the studio life and learned biz magic.

I make a really good living in biz, I LOVE doing what I do now; I get to work with amazing people, I get to talk to amazing people. For the love of god, I spoke to the creator of Fruit Ninja tonight to have “chat”. How cool is that? Dude totally rocks btw.

My gig allows me to talk to my heroes. Seriously, I love playing video games that much.

The most common thing people say to me is “….wow, you are a business guy and you love playing video games?”

When I stepped out of the studio, I made myself the pledge that I would only work on titles that touched me deeply. I would only work on games that I personally wanted to play. I would be way richer if I just kissed ass and decided to suck it up for the lord, god, almighty dollar!

If I do say so myself, I was pretty decent on the creative side of things too, but to be brutally honest; I was running a studio that sucked at business.

So one day I stopped. One day I decided that I would step out of the studio. I put a 22 year old in charge of it and for 7 mobile games -feature phones- he was shit. Then he just flipped the page and was brilliant.

Recently, I met a guy called Jason Brice. He sent me @ messages constantly, he was really cool and then one day I seen the maps he made in an FPS.

OMFG

They were great!

I looked at them -I hated the crane on the harbor map- BUT I loved the game itself.

Jason was creating the game that I wanted to play.

His view was that there were way too many layers between the player and AK47′ing another guy in the face.

He had me at “hello”.

So as I play this game, I talk about it, I reveal things about it and then I talk about it again.

Simply put, THIS IS GAMING!

I listen to an average of 14 game pitches a week, this was one of the first game pitches that I have seen that melted my resolve. Here I thought that I knew everything and these noobz are teaching me how to love video gaming again.

I want to be fair, I want to be honest and I do not want to be a prick.

BUT I am sick of the modern day FPS titles or as I like to call them “We just came up with another feature that allows us to get you to buy another sequel”………….. ok that is a long title.

I am sick that on NPD day that we all shiver with anticipation at how many people bought our game. Here is a novel fucking idea, I am sick with anticipation at the thought of how many people enjoyed our game.

The game is ReKoil BTW. BUY IT PLUX

I am my most vulnerable when I am sitting in front of a metrics screen looking at the numbers, asking, wanting, no begging video gamers to like my title.

That is weak sauce, I want them to like me. I want them, no I crave that they like me!

It is the vulnerable essence of every video game maker. It is that vulnerability that allows me to exist.

This very insecurity, allows me to make games that I want to play, this insecurity allows Jason and the team to jump off a cliff and trust the fact that someone on the team will catch him.

Guess what? When someone that you consider a blood friend jumps off a cliff; you always catch them.

I am proud of what we are making today, I love that we are participating in the conversation! Will we beat Black-Op’s 2? God no! Their budget is 60X what people say that we are worth, but we will be participating in the conversation.

We are basically fighting the good fight, we are throwing punches and getting the shit kicked out of us. BUT we are Marty McFly and we will knock Biff the fuck out.

Am I desperate? FUCK YEAH!

Just tonight I sent this email:

From: Kevin Dent [mailto:kevin@XXX.XXXX]
Sent: Thursday, May 10, 2012 8:50 PM
To: ‘Andy McNamara’
Subject: Front cover

Hi Andy,

Who do I have to screw to get the front cover for ReKoil?

Cheers,

Kevin

As of today, we are not on Kickstarter, we are totally self-funded and we are totally throwing ourselves under the bus in an attempted to make a better game.

Let me be clear, Andy is an amazing person and I love him dearly. He is smart, contiencious and endearing. BUT Game Informer is owned by Game Stop and those dudes are hardcore publisher fuckers, there is zero chance of us getting the cover.

There is no way that, me, you or your freak parents will get us on the front cover of GI and nor should it. Brilliant games get on there 12 a year at least.

That said, it was worth a shot, we are living in the era of the indie.

There has been so many amazing titles in the last twelve months made by people with way more talent than me.

I rejoice at the next gen of game creators.

I am humbed by them.

But the truth is, that Jason Brice, asked me to jump off a cliff and I made the leap.

Who will catch me?

Kevin

The lizard brain is not merely a concept. It’s real, and it’s living on the top of your spine, fighting for your survival. But, of course, survival and success are not the same thing. The lizard brain is the reason you’re afraid, the reason you don’t do all the art you can, the reason you don’t ship when you can. The lizard brain is the source of the resistance.”
― Seth Godin, Linchpin: Are You Indispensable?

I’ve adapted a guest post I wrote over at becoming minimalist to speak more to game development. Thanks to Joshua Becker for the opportunity to write for his blog and his help with the original post.

I’ve been giving a lot of thought to single tasking, focus and distractions in my professional and personal life. Here is what I believe to be the four biggest factors contributing to distracting us from doing great creative work. By the way “creative work” to me means art, code, design, music you name it. Many facets and roles in gamedev are creative, perhaps not at first glance, but any creative problem solving applies, in my humble opinion.

#1 We’re distracted by notifications
Everyone is “always on”. We have our email open all day and internal instant messaging clients humming – not to mention the external social networks and IM chats. Instant notifications are being pushed from everywhere to wherever we are at the moment, sitting at our desktops or walking about with our phones. Most of us are on digital leashes of some kind even when we don’t need to be. We’re in a state of constant multiple deliverables or actionable tasks as well as completely open to distractions. How do we get anything done?

Some studies even propose that people can be addicted to the micro-endorphin rush we get when we get an email or a tweet. That’s why we can’t stop checking our phones when we’re away from work or even when we’re driving. Most folks agree that these short bursts of interruptions are changing our brains.

Can you read through this entire 1500 hundred word post without a notification going off?

#2 We’re distracted by geography
Office geography has the entire team sitting in cube farms. These close-knit cubes are designed to “enhance communication.” For example, artists can quickly look over the half-wall and ask a programmer a question. This is great for immediate problem solving, however, the creative tasks we perform require dedicated and focused work in order to be fully realized. The constant interruptions hinder our productivity. What’s the tradeoff? Does the immediate communication outweigh the lack of focus?

Interruptions are hurting productivity, not helping it.

#3 We can’t focus
In this land of distractions, how do we concentrate long enough on a creative task to produce quality? Mental context switching is hard to quantify as there are so many variables. However, I have heard that switching between 3 tasks costs a person 40% of their available work time. You can just throw (at least) one of those tasks out the window, I hope it wasn’t important.

Does your company have a lot of meetings? Meetings in the middle of the day can be a drastic productivity and creativity killer.

Multi-tasking just doesn’t work. Is there a “sweet spot” in your workday that you can work uninterrupted for more than an hour on a single task?

#4 We’re comfortable and afraid
Our lizard brains want us to be stuck in a rut. Safety is the Lizard Brain’s primary objective. Safe is free of conflicts and challenges. Safe is the same thing over and over again. Safe is not standing out in the crowd. Safe, as the lizard brain defines it, is not good for your creativity and thereby the projects you work on.

Image courtesy of Alex Moore

Here are some of the things that I’m using or trying out to combat the lizard brain, distractions and multi-tasking.

Control your time.
Batch processing your email can be effective. Once at the beginning of the day and again at the end. This is more efficient than shifting to outlook with each new notification and processing each single email. If you think that is too long to be out of contact, do the email dance for 10 minutes every two hours. Or whatever works for you, just try to keep the notifications off. If there truly is an emergency you will know, someone will come find you.

Turn off your work IM. Same rules as above. Maybe you’re available and on IM only in the afternoon. Or set your status to do not disturb and teach people politely what that means. Try things and see how they work for you.

Do one thing at a time. Make a list of the 3 or 5 most important things for that day. Do them as early in the day as you can. You’ll accomplish the “big things” early and everything else after that is icing on the cake.

All of this can be tricky at first if your culture wants you to respond immediately, but I think it pays off in the end. You have more uninterrupted time to do something good. Talk to your manager if you need to before implementing a radical change.

Have the good kind of meetings.
The morning standup with your team is probably the best type of meeting. It’s early, not in the middle of the day. Everyone says what they are working on, if anything is blocking them and what they’ll do next. Preferably everyone stands. It is called a stand up after all. This is a bold choice, but decline any meeting that you think is a waste of time. Reply that you can’t make it and ask for a recap. Meeting Agendas and Recaps are such a great tool for meetings, we seem to forget about them.

The Agenda: When booking the meeting provide a list of what topics to discuss, who is responsible for talking for each topic and a guesstimate on how long they’ll have to talk. Send the agenda out a couple of days beforehand and make it known that the attendees are expected to be prepared. Simon Cooke suggests formulating the agenda as questions to get people thinking and vested.

During the meeting, stick to the schedule. Get in, do good stuff, get out.

The Recap: briefly state what was discussed.

Here’s a sample meeting agenda:

Todays Date, Galactic Domination Meeting Agenda

• Overview, Palpatine, 15 minutes
• Death Star #2 construction status, Tarkin, 5 minutes
• Various disturbances in the force, Vader, 10 minutes

And the recap:

Overview, Palpatine, 15 minutes

Squash rebellion by destroying secret base.

Eliminate Galactic Senate via political maneuvering.

Death Star #2 construction status, Tarkin, 5 minutes

We fixed the exhaust port / proton torpedo bug

Various disturbances in the force, Vader, 10 minutes

Obi-Wan is dead, but no one is sure why he just disappeared.

Follow up on plan to turn Son of Skywalker to the dark side.

Wait.
Can you wait a while to talk to the artist? Look at him, he has his headphones on (game developer code for “Do Not Disturb”) and is obviously “in the zone” sculpting in the 3D modeling application with a ton of reference images on the other monitor. Don’t interrupt him asking for his hours spent this week on his tasks. Let him do his job. That User Interface Engineer who’s sliding her chair back and forth from her development kit and her two-screen computer rig? She’s fixing a bug in the UI on her computer, testing it on the kit, and she’s obviously busy. Do not walk over and ask her how it’s going right now. Let her do her job. Your opportunity to ask your question will come… but you may need to wait your turn to ask it.

Get people on board.
Use team signals to tell others “do not disturb.” I’ve seen flags put up by artists to denote when they are in the zone and to please come back later. I’ve also seen tech teams have one team member wear a hat signaling that he/she is the person on call for help that day. Find a way of signaling people and teach them to recognize and respect it. Work it out with your team so you can be efficient together.

Protect your industry’s most valuable asset.
The most valuable asset any company has are its people. People make the art, tech and design of a game. They support the game or the other people making the games. There are certain parts of the day when creativity runs wild and other parts of the day that are more suitable for less creative tasks. Do what you can to help reduce roadblocks and ask your Manager for support.

I certainly don’t have all the answers. But I’m trying to communicate with one person at a time, handle one problem at a time, and work on one idea at a time. All as a part of my effort to bring the principles of minimalism, simplicity and single tasking into my personal and professional life. I’d love to hear your thoughts on these topics.

This post covers how to set up Visual Studio to allow you to easily look at the optimised assembly code generated for simple code snippets like the ones we deal with in this series. If you wonder why I feel this is worth a post of its own here’s the reason – optimising compilers are good, and given code with constants as input and no external output (like the snippets I give as examples in this series) the compiler will generally optimise the code away to nothing – which I find makes it pretty hard to look at. This should prove immensely useful, both to refer back to, and for your own experimentation.

Here are the backlinks for preceding articles in the series in case you want to refer back to any of them (warning: the first few are quite long):

1. http://altdevblogaday.com/2011/11/09/a-low-level-curriculum-for-c-and-c/
2. http://altdevblogaday.com/2011/11/24/c-c-low-level-curriculum-part-2-data-types/
3. http://altdevblogaday.com/2011/12/14/c-c-low-level-curriculum-part-3-the-stack/
4. http://altdevblogaday.com/2011/12/24/c-c-low-level-curriculum-part-4-more-stack/
5. http://altdevblogaday.com/2012/02/07/c-c-low-level-curriculum-part-5-even-more-stack/
6. http://altdevblogaday.com/2012/03/07/c-c-low-level-curriculum-part-6-conditionals/

Assumptions

Strictly speaking, dear reader, I am making tonnes of assumptions about you as I write this – that you read English, that you like to program etc. but we’ll be here all day if I try to list those so let’s stick to the ones that might be immediately inconvenient if they were incorrect.

I will be assuming that you have access to some sub-species of Visual Studio 2010 on a Windows PC, and that you are familiar with using it to do all the everyday basics like change build configurations, open files, edit, compile, run, and debug C/C++.

Creating a project

Open Visual Studio and from the menu choose “File -> New -> Project…”.

Once the new project wizard window opens (see below):

• go to the tree view on the left of the window and select “Other Languages -> Visual C++”
• in the main pane select “Win32 Console Application   Visual C++”
• give it a name in the Name edit box
• browse for a location of your choosing on your PC
• click OK to create the project

Once you have clicked OK just click “Finish” on the next stage of the wizard – in case you’re wondering, the options available when you click next don’t matter for our purposes (and un-checking the “Precompiled header” check box makes no difference, it still generates a console app that uses a precompiled header…).

Changing the Project Properties

The next step is to use the menu to select “Project -> <YourProjectName> Properties”, which will bring up the properties dialog for the project.

When the properties dialog appears (see image below):

• select “All Configurations” from the Configuration drop list
• select “Configuration Properties ->General” in the tree view at the left of the window
• in the main pane change “Whole Program Optimisation” to “No Whole Program Optimisation”.

Next, in the tree view (see image below):

• in the tree view, navigate to “C/C++ -> Code Generation”
• in the main pane, change “Basic Runtime Checks” to “Default” (i.e. off)

Finally (see image below):

• in the tree view, go to “C/C++ -> Output Files”
• in the main pane change “Assembler Output” to “Assembly With Source Code /(FAs)”
• once you’ve done that click “OK”

Now, when you compile the Visual Studio compiler will generate an .asm file as well as an .exe file. This file will contain the intermediate assembly code generated by the compiler, with the source code inserted into it inline as comments.

You could alternatively choose the “Assembly, Machine Code and Source (/FAcs)” option if you like – this will generate a .cod file that contains the machine code as well as the asm and source.

I prefer the regular .asm because it’s less visually noisy and the assembler mnemonics are all aligned on the same column, so that’s what I’ll assume you’re using if you’re following the article, but the .cod file is fine.

So, what did we do there?

Well, first we turned off link time code generation. Amongst other things, this will prevent the linker stripping the .asm generated for functions that are compiled but not called anywhere.

Secondly, we turned off the basic runtime checks (which are already off in Release). These checks make the function prologues and epilogues generated do significant amounts of (basically unneccessary) extra work causing a worst case 5x slowdown (see this post by Bruce Dawson on his personal blog for an in depth explanation).

Finally, we asked the compiler not to throw away the assembly code it generates for our program; this data is produced by the compilation process whenever you compile but is usually thrown away, we’re just asking Visual Studio to write it into an .asm file so we can take a look at it.

Since we made these changes for “All Configurations” this means we will have access to .asm files containing the assembly code generated by both the Debug and Release build configurations.

Let’s try it out

So in the spirit of discovery, let’s try it out (for the sake of familiarity) with a language feature we looked at last time – the conditional operator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

#include "stdafx.h"
 
int ConditionalTest( bool bFlag, int iOnTrue, int iOnFalse )
{
    return ( bFlag ? iOnTrue : iOnFalse );
}
 
int main(int argc, char* argv[])
{
    int a = 1, b = 2;
    bool bFlag = false;
    int c = ConditionalTest( bFlag, a, b );
    return 0;
}

The question you have in your head at this moment should be “why have we put the code into a function?”. Rest assured that this will become apparent soon enough.

Now we have to build the code and look in the .asm files generated to see what the compiler has been up to…

First build the Debug build configuration – this should already be selected in the solution configuration drop-down (at the top of your Visual Studio window unless you’ve moved it).

Next build the Release configuration.

Now we need to open the .asm files. Unless you have messed with project settings that I didn’t tell you to these will be in the following paths:

<path where you put the project>/Debug/<projectName>.asm

<path where you put the project>/Release/<projectName>.asm

.asm files

I’m not going to go into any significant detail about how .asm files are laid out here, if you want to find out more here’s a link to the Microsoft documentation for their assembler.

The main thing you should note is that we can find the C/C++ functions in the .asm file by looking for their names; and that – once we find them – the mixture of source code and assembly code looks basically the same as it does in the disassembly view of Visual Studio in the debugger.

main()

Let’s look at main() first. This is where I explain why the code snippet we wanted to look at was put in a function. I can tell you’re excited.

Here’s main() from the Debug .asm (I’ve reformatted it slightly to make it take up less vertical space):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

_TEXT    SEGMENT
_c$ = -16                        ; size = 4
_bFlag$ = -9                        ; size = 1
_b$ = -8                        ; size = 4
_a$ = -4                        ; size = 4
_argc$ = 8                        ; size = 4
_argv$ = 12                        ; size = 4
_main    PROC                        ; COMDAT
; 9    : {
    push    ebp
    mov    ebp, esp
    sub    esp, 80                    ; 00000050H
    push    ebx
    push    esi
    push    edi
; 10   :     int a = 1, b = 2;
    mov    DWORD PTR _a$[ebp], 1
    mov    DWORD PTR _b$[ebp], 2
; 11   :     bool bFlag = false;
    mov    BYTE PTR _bFlag$[ebp], 0
; 12   :     int c = ConditionalTest( bFlag, a, b );
    mov    eax, DWORD PTR _b$[ebp]
    push    eax
    mov    ecx, DWORD PTR _a$[ebp]
    push    ecx
    movzx    edx, BYTE PTR _bFlag$[ebp]
    push    edx
    call    ?ConditionalTest@@YAH_NHH@Z        ; ConditionalTest
    add    esp, 12                    ; 0000000cH
    mov    DWORD PTR _c$[ebp], eax
; 13   :     return 0;
    xor    eax, eax
; 14   : }
    pop    edi
    pop    esi
    pop    ebx
    mov    esp, ebp
    pop    ebp
    ret    0
_main    ENDP
_TEXT    ENDS

As long as you’ve read the previous posts, this should mostly look pretty familiar.

It breaks down as follows:

• lines 1-8: these lines define the offsets of the various Stack variables from [ebp] within main()’s Stack Frame
• lines 10-15: function prologue of main()
• lines 17-20: initialise the Stack variables
• lines 22-30: push the parameters to ConditionalTest() into the Stack, call it, and assign its return value
• line 32: sets up main()’s return value
• lines 34-38: function epilogue of main()
• line 39: return from main()

Nothing unexpected there really, the only new thing to take in is the declarations of the Stack variable offsets from [ebp].

I feel these tend to make the assembly code easier to follow than the code in the disassembly window in the Visual Studio debugger.

And, for comparison, here’s main() for the Release .asm:

1
2
3
4
5
6
7
8
9
10
11
12
13

_TEXT    SEGMENT
_argc$ = 8                        ; size = 4
_argv$ = 12                        ; size = 4
_main    PROC                        ; COMDAT
; 10   :     int a = 1, b = 2;
; 11   :     bool bFlag = false;
; 12   :     int c = ConditionalTest( bFlag, a, b );
; 13   :     return 0;
    xor    eax, eax
; 14   : }
    ret    0
_main    ENDP
_TEXT    ENDS

The astute amongst you will have noticed that the Release assembly code is significantly smaller than the Debug.

In fact, it’s clearly doing nothing at all other than returning 0. Good optimising! High five!

As I alluded to earlier, the optimising compiler is great at spotting code that evaluates to a compile time constant and will happily replace any code it can with the equivalent constant.

So that’s why we put the code snippet in a function

It should hopefully be relatively clear by this point why we might have put the code snippet into a function, and then asked the linker not to remove code for functions that aren’t called.

Even if it can optimise away calls to a function, the compiler can’t optimise away the function before link time because some code outside of the object file it exists in might call it. Incidentally, the same effect usually keeps variables defined at global scope from being optimised away before linkage.

I’m going to call this Schrödinger linkage (catchy, right?). If we want our simple code snippet to stay around after optimising we only need to make sure that it takes advantage of Schrödinger linkage to cheat the optimiser.

If the compiler can’t tell whether the function will be called, then it certainly can’t tell what the values of its parameters will be during one of these potential calls, or what its return value might be used for and so it can’t optimise away any code that relies on those inputs or contributes to the output either.

The upshot of this is that if we put our code snippet in a function, make sure that it uses the function parameters as inputs, and that its output is returned from the function then it should survive optimisation.

It’s really a testament to all the compiler programmers over the years that it takes so much effort to get at the optimised assembly code generated by a simple code snippet – compiler programmers we salute you!

ConditionalTest()

So, here’s the Debug .asm for ConditionalTest() (ignoring the prologue / epilogue):

1
2
3
4
5
6
7
8
9
10
11
12
13

; 5    :     return( bFlag ? iOnTrue : iOnFalse );
    movzx    eax, BYTE PTR _bFlag$[ebp]
    test    eax, eax
    je    SHORT $LN3@Conditiona
    mov    ecx, DWORD PTR _iOnTrue$[ebp]
    mov    DWORD PTR tv66[ebp], ecx
    jmp    SHORT $LN4@Conditiona
$LN3@Conditiona:
    mov    edx, DWORD PTR _iOnFalse$[ebp]
    mov    DWORD PTR tv66[ebp], edx
$LN4@Conditiona:
    mov    eax, DWORD PTR tv66[ebp]
; 6    : }

As you should be able to see, this is doing the basically same thing as the code we looked at in the Debug disassembly in the previous article:

• branching based on the result of testing the value of bFlag (the mnemonic test does a bitwise logical AND)
• both branches set a Stack variable at an offset of tv66 from [ebp]
• and both branches then execute the last line which copies the content of that address into eax
  

Again, the assembly code is arguably easier to follow than the corresponding disassembly because the jmp mnemonic jumps to labels visibly defined in the code, whereas in the disassembly view in Visual Studio you generally have to cross reference the operand to jmp with the memory addresses in the disassembly view to see where it’s jumping to…

Let’s compare this with the Release assembler (again not showing the function prologue or epilogue):

1
2
3
4
5
6
7

; 5    :     return( bFlag ? iOnTrue : iOnFalse );
    cmp    BYTE PTR _bFlag$[ebp], 0
    mov    eax, DWORD PTR _iOnTrue$[ebp]
    jne    SHORT $LN4@Conditiona
    mov    eax, DWORD PTR _iOnFalse$[ebp]
$LN4@Conditiona:
; 6    : }

You will note that the work of this function is now done in 4 instructions as opposed to 9 in the Debug:

• it compares the value of bFlag against 0
• unconditionally moves the value of iOnTrue into eax
• if the value of bFlag was not equal to 0 (i.e. it was true) it jumps past the next instruction…
• …otherwise this moves the value of iOnFalse into eax

As I’ve stated before I’m not an assembly code programmer and I’m not an optimisation expert. Consequently, I’m not going to offer my opinion on the significance of the ordering of the instructions in this Release assembly code.

I am, however, prepared to go out on a limb and say it’s a pretty safe bet that the Release version with 4 instructions is going to execute significantly faster than the Debug version with 9.

So, why such a big difference between Debug and Release for something that when debugging at source level is a single-step?

Essentially this is because the unoptimised assembly code generated by the compiler must be amenable to single-step debugging at the source level:

• it almost always does the exact logical equivalent of what the high level code asked it to do and, specifically, in the same order
• it also has to frequently write values from CPU registers back into memory so that the debugger can show them updating

Summary

What’s the main point I’d like you to take away from this article? Optimising compilers are feisty!

You have to know how to stop them optimising away your isolated C/C++ code snippets if you want to easily be able to see the optimised assembly code they generate.

This article shows a simple boilerplate way to short-circuit the Visual Studio optimising compiler – mileage will vary on other platforms.

There are other strategies to stop the optimiser optimising away your code, but they basically all come down to utilising the Schrödinger linkage effect; in general:

• use global variables, function parameters, or function call results as inputs to the code
• use global variables, function return values, or function call parameters as outputs from the code
• if you’re not using Visual Studio’s compiler you may also need to turn off inlining

A final extreme method I have been told about is to insert nop instructions via inline assembly around / within the code you want to isolate. Note that you should use this approach with caution, as it interferes directly with the optimiser and can easily affect the output to the point where it is no longer representative.

Epilogue

So, I hope you found this interesting – I certainly expect you will find it useful :)

The next article (as promised last time!) is about looping, which is another reason why it seemed like a good time to cover getting at optimised assembly code for simple C/C++ snippets.

I will be referring back to this in future articles in situations where looking at the optimised assembly code is particularly relevant.

If you’re wondering what you should look at first to see how Debug and Release code differ, and want to get practise at beating the optimiser, I’d suggest starting with something straight forward like adding a few numbers together.

Lastly, but by no means leastly, thanks to Rich, Ted, and Bruce for their input and proof reading; and Bruce for supplying me with the tip that made this post possible.

The TL;DR of John Cleese’s talk

Closed Mode

Purposeful Highly Productive, but not creative. Good for getting things done. Default Mode at Work.

Open Mode

Playful, Curious, Fun, Humorous, Relaxed, Contemplative without goals.

I had a lot of takeaways from that but my biggest takeaway was what could I do to be more open and enable openness around me. In the past I’ve drawn mostly from my dramatic background and improv lessons, always saying yes. See Lisa Brown’s brilliant article on that topic.

Humour and Time are key to being in the Open Mode. Also Goals, Hierarchy and Authority are not conducive to the Open Mode. An interesting thought for those of you who read the Valve Employee Handbook floating around the net at the moment. The key thought this lead me to is simple…

You Can’t Play by Proxy

Too often I espouse the brilliance of the programming designer. The real magic is simple, you cannot play by proxy. A designer who can’t play (and change) with gameplay is like a chef who gets someone else to taste the food and describe it. An artist who cannot change the appearance of something in-game without someone else is like painting on canvas by instructing a monkey holding a paintbrush. A programmer who cannot go outside the spec is like a dancer in a straight jacket.

You need to hold the ball in your hand and feel it bounce. Always ensure your source control, build server and other tools do not stop someone’s experimentation.

The Open Mode is Unachievable in Crunch Time

A common example of the open mode in game development: someone has just implemented a system which due to an amusing bug is glitching out or not behaving in the designed fashion. In a crunch time, i.e. closed environment, a bug is filed, profanities are hurled and the bug is squashed. In an open mode and a lighter environment, possible earlier in the project, the group plays with the bug and says something like, “Isn’t that interesting…”

Now to be clear, most times nothing immediately comes from that comment though at times a brilliant idea or mechanic emerges. Often these moments lend themselves to interesting gameplay moments. For a prime example of game development project by open mode style thinking, look at Minecraft. Many gameplay features started as bugs or unintended behaviour which in a traditional environment would have been eradicated but the players enjoyed it and Notch left it in.

In order to achieve things we need to be in a closed mode. Though it’s easy to get stuck in that mode and with tunnel vision run down our path missing all the brilliant opportunities that are being thrown up during game development.

Playful Tools

When designing or creating a tool or pipeline think about how your tools enable playfulness, experiment and toy around with your tools. A willingness to create without fear. Observe how playful we are with pens, for many of us our primary tool.

Two Developers, one creating a scene in notepad with fragile .xml, the other drawing a blueprint in photoshop. The drawing based solution is more fluid and open to play. The developer can draw a “humorous shape” and see it in game without fuss. Leading to greater comfort and competence, discovering creative uses of the tools.

Nothing can frustrate and break playfulness more than a deep pipeline or frustrating tools. A buggy or frustrating tool ensures you’re in the Closed Mode. As in the Closed Mode we are less likely to make mistakes and can follow rigid guidelines.

While being Creative Nothing is Wrong

We live in a digital world of hacks and cheats, we care for results not method. The thing which appears on screen is a beautiful lie. So if your engine is refresh sensitive and those silly European televisions are giving you trouble, well, just turn up the earth’s gravity*. Remember the product is the experience, not the game.

Too often a professional artist, designer, programmer or otherwise will dismiss or stamp on a comment because it’s patently silly. They insist on saying it’s too complex or saying it’s daft. Now please, respect the professional skill for which the individual was hired. Though at no point should someone be crushed or demotivated by the HAMMER of authority.

Jam, Jive and Jiggle

Encourage people to take time out to play in fixed time periods to jam, jive and jiggle. I write this after having just finished a weekend of Ludum Dare fun. The result of the Jam is less important than the time to play. The time to ponder and play with an idea. If you have a sprint structure try to take some time out after a sprint or a whole sprint, WITHOUT GOALS! Just play. See what comes from it.

People need to feel free to play with ideas, in order to have great ideas.

* A real humorous example of an engine hack seen in the wild.

In short, lots of profilers can tell you what your program is doing, but few profilers are excellent at telling you why your program is doing nothing.

Our story so far

Xperf has a high learning curve. Therefore I highly recommend that you start by reading some previous articles from this series. The entire series can be found here, but the most important ones are:

• Xperf Basics: Recording a Trace – this covers installing xperf (takes just a few minutes), recording traces, and adding ETW events to your game
• Xperf Analysis Basics – the essential knowledge of how to navigate the xperf UI, including how to set up symbol paths

The rest of this post assumes that you have installed xperf (preferably the Windows 8 version) and have familiarity with at least Xperf Analysis Basics.

Wait Analysis Victories

I’ve had good luck using Wait Analysis to find many performance problems. Some of these delays were short enough to be difficult to notice, yet long enough to matter. Others were debilitating. All were difficult or impossible to analyze through CPU sampling or other ‘normal’ CPU profilers. Some examples include:

• Finding the cause of brief startup hangs in Internet Explorer and various games
• Profiling Luke Stackwalker to find out why it caused frame rate glitches in the game it was profiling
• Finding the cause of a 10x perf-reduction when upgrading to a newer version of Windows
• Finding the cause of frame rate hitches during fraps recording
• Finding the chain of lock contention that caused frame rate hitches on a heavily loaded system
• Finding the cause of (and a workaround for) repeated 2-6 second hangs in Visual Studio’s output window

The last investigation is the one I want to cover today. It is sufficiently simple and self-contained that I can cover it end-to-end in a single (long) post.

Finding the hang

When profiling a transient problem such as a frame-rate glitch or a temporary hang the first challenge is to locate the hang in the trace. A trace might cover 60 seconds, and a hang might last for 2 seconds or less, so knowing where to look is crucial. There are a number of ways to do this:

• Find the key stroke that triggered the hang, through logging of input events
• Use instrumentation in the functions of interest
• Look for patterns in the CPU usage or other data
• Use OS hang-detection events

I’ve used all four of these techniques. Our internal trace recording tool has an optional input event logger which puts all keyboard and mouse input into the trace (watch for it). If a hang is triggered by a particular key press or mouse click then finding its start point in the trace is trivial.

Custom instrumentation (emitting ETW events at key points in your game, see the Recording a Trace post) is also a common technique. Emitting an event every frame makes a frame rate hitch plainly visible. However this doesn’t work when investigating performance problems in other people’s code, such as in Visual Studio.

In some cases a hang will be plainly visible in the CPU consumption. One recent hang showed a significant hole in the otherwise consistent CPU usage, plain as day.

A specific event that indicates the time and duration of a hang would be ideal, and Windows 7 actually has such an event. The Microsoft-WindowsWin32k ETW user provider will emit an event whenever a thread resumes pumping messages after a significant delay. Windows Vista and earlier users are out of luck, but on Windows 7 this is often exactly what is needed, and this provider is enabled by my recommended trace recording batch files.

It’s hands on time

I’ve uploaded a .zip file of a sample trace to VisualStudioF8Hang.zip. If you download this trace, load it into xperfview, and make sure that the Microsoft symbols can be found then you can follow along. This is by far the best way to learn wait analysis.

This trace covers over over ten minutes for some types of data, but the detailed sampling and context switch data only covers 28 seconds, from about 782 to 810 seconds.

Start by selecting the region where we have full data, from 782 to 810 s and cloning this selection.

Our path now depends on whether you are using the new (Windows 8) version of xperfview.exe.

Hands on with old versions of xperfview

While this exact technique is only applicable to (and only works with) old versions of xperfview, the general concept is still applicable and the exploration of generic events is crucial whether looking for your custom events or exploring the built-in events.

Scroll down to the Generic Events table. Right-click the selected region and bring up a summary table. Enable the Process Name column and put it first. Enable the ThreadID column and put it after Field 3. Move the Time (s) column and put it after the ThreadID column. I also hid a couple of columns in order to get my screenshot to fit, but that’s less critical. Now we have all of the information we need in a convenient and easy to read place.

If we drill into the data for devenv.exe and select the MessageCheckDelay Task Name we should see something like this:

The Zen of summary tables is all about looking for data columns that seem useful, enabling them, and fearlessly rearranging columns to group/sort/pivot/spindle the data to answer your question. In this case our question was when does devenv.exe (group by Process Name) hang (group by the Microsoft-Windows-Win32k provider, Task Name equals MessageCheckDelay or InputProcessDelay), and for those events, look at the TimeSinceInputRemoveMs, Thread ID, and Time (s) data.

So now, with relatively little effort, I know that devenv.exe hung (didn’t check for messages) for 5,304 ms, its message pump is running on thread ID 9,536, and the hang ended at 805.666 seconds into the trace.

Cool.

Hands on with the new version of xperfview (6.2.8229)

Microsoft is continuing to develop xperf and if you install the latest version (released Feb 29, 2012, and linked to from here) then there are a couple of options. Wpa.exe has a new UI which shows pretty graphs for UI delays:

I don’t know how to dig in deeper so I can’t tell if it is any use, so that’s all I have to say about it.

The new xperfview.exe has removed the MessageCheckDelay and InputProcessDelay events from Generic Events but has added a new UI Delay Information Graph. If you scroll down to this graph and zoom in around 800 s (in the area where we have full detail) then you should see five reports of hung apps. VTrace.exe (my trace recording application) hung for a while, there are three spurious reports of Internet Explorer hanging, and there is a MsgCheckDelay report for devenv.exe. It’s really too easy.

You can right-click to change the threshold for what delays are reported, or to bring up a summary table of delays. You’ll need to bring up the Delay Summary Table to find out the UI thread ID for devenv.exe.

Select the region around the devenv hang and we’re ready for the next step.

Finding the cause

The MessageCheckDelay is emitted at the end of the hang (805.666 seconds) and it tells us the length of the hang (5.304 s) so we now know the range of the hang quite accurately.

The hang runs from 800.362 to 805.666 seconds so we should zoom in on that area of the graphs in the xperf main window and look at CPU Usage by Process. My system has eight hardware threads (four hyperthreaded cores) so 12.5% utilization represents one busy thread. Even without that context we can see from the graph below that my CPUs are idle for most of the time. There’s a bit of devenv activity (the two blue spikes), but mostly this is an idle hang.

When analyzing an idle hang you should select the entire region of the hang, and it is particularly important to select the end of the hang. It is better to select a few extra tens or hundreds of milliseconds at the end rather than risk missing the crucial events that end the hang. This selection can be done with the mouse or by right-clicking and using the Select Interval command. For easy reproducibility I right-clicked and used the Select Interval command to select the region from 800.0 s to 806.0 s. I then used Clone Selection to copy it to all of the graphs.

Who woke whom?

If a thread is not running, and it then starts running, then there was a context switch that started it (the new thread) running. That context switch is recorded in our ETW trace and contains all sorts of useful information. Include in this information is (for the traces recorded with my recommended batch files) the new process name and thread ID, the call stack which the thread woke up on (which is the same one it went to sleep on), the length of time it was not running and, for threads that were waiting on some synchronization primitive, the thread that woke it up.

Ponder that, because it’s crucial. An ETW trace tells you, for each context switch, how long the thread was not running, and who woke it up. That’s why it is important to have the end of the hang selected, because that is (presumably) the time of the context switch that gets the thread running again.

In the main xperf window go to the CPU Scheduling graph (make sure the correct time range is selected), right click on the selection, and select “Summary Table with Ready Thread” to view all context switches for the selected region together with the readying thread information. Make sure the columns are in this order:

1. NewProcess – this is the process whose thread is being scheduled to run
2. NewThreadId – this is the thread ID of the thread being scheduled to run
3. NewThreadStack – this is the stack that the thread will resume running at
4. ReadyingProcess – this is the process, if any, that readied the new thread
5. ReadyingThreadId – guess. Go ahead, you can figure it out.
6. ReadyingThreadStack – this is the stack of the readying thread when it readied the new thread
7. Orange bar – columns to the left of this are used for grouping, columns to the right are for sorting and data display
8. Count – how many context switches are summarized by each row
9. Sum:TimeSinceLast (us) – the time the new thread was not running (time since it last ran) summed across all context switches summarized by each row

There are more columns, and for deeper analysis they can be useful, but we don’t need them today.

With our columns thus arranged we can quickly find our problem. Find devenv.exe (be sure to find the correct PID if multiple versions are running) and expand it, find the thread of interest (9,536, from the MessageCheckDelay event), then expand the stack. If you click the “Sum:TimeSinceLast (us)” column so the little arrow is pointing down then as you drill down into the stacks (hint: select the top node and then repeatedly press right-arrow) it will go down the hot call stack. In the sample trace, over the selected region, thread 9,536 starts with a total of about 5.523 s of non-running time over 316 context switches. As we drill down we get to a single context switch that ended an idle gap of 5.202 s. That’s our hang, clear as day.

The NewThreadStack for this 5.202 s call stack starts at _RtlUserThreadStart and winds through a lot of Visual Studio code. Microsoft is kind enough to publish symbols for much of VS, as well as for Windows and about fifty rows down we get to the interesting details:

It’s a single context switch (‘count’ goes down to one when we got a bit lower in the call stack) that put the Visual Studio UI thread to sleep for 5.202 s. It doesn’t get much clearer than that.

If we go down to the bottom of the stack and expand the next three columns (compressed in the screen shot above for size reasons) then we can see who woke us, which can also be described as “who we were waiting for”:

In this case it was the System process (thread 5880) in an IopfCompleteRequest call that goes through MUP.SYS. If we know what MUP.SYS is then that gives us another clue as to the root cause, but even without that we know that Visual Studio called CreateFileW and it took a long time to return.

What about the other threads?

In our selected region their are context switch events for 11 threads in devenv.exe. For all of those threads the Sum:TimeSinceLast value is greater than for 9,536, the thread we are investigating. So why aren’t we looking at them?

It’s important to understand that Sum:TimeSinceLast just measures how long a thread was idle, and there is nothing wrong with a thread being idle. A thread being idle is only a problem if it is supposed to be doing something and isn’t. In fact, if devenv.exe has 11 threads then they had better be idle most of the time or else my six-core machine is going to be constantly busy.

Many of the threads have a Sum:TimeSinceLast time of about 15 s, which is significantly longer than the 6 s time period selected. That’s because this summary table shows all of the context switches that occurred during this time period, and the first context switch for these threads was after they had been idle for a very long time, most of that time outside of the selected region.

The reason we are looking at thread 9,536 is because (according to the MessageCheckDelay event) it is the UI thread and it went for 5.304 s without pumping messages. It kept me waiting, and that makes me angry. You wouldn’t like me when I’m angry.

File I/O summary table

Since we know that the hang is related to file I/O we should look at what file I/O is happening during this time period. Luckily this information is recorded by the xperf batch files that I recommend.

On the main xperf window go to the File I/O graph and bring up a summary table, for the same time region we’ve been using so far. Arrange the columns as shown below and drill in as usual. I’m sure this screen shot won’t show up very well, but I can’t shorten it any more. It contains too much glorious information. Click on the image for deeper details:

We can see here that a Create file event, from devenv.exe, thread 9,536, took 5.203 s, trying to open \Device\Mup\…, and that ultimately the network path was not found.

Wow.

It turns out that \Device\Mup, or MUP.sys, means the network. The hang is because Visual Studio tried to open a non-existent network file, and sometimes that causes a 5.2 s network timeout. Hence the hang.

The remainder of the hang is from a few other context switches and CPU time that account for the rest of the 5.304 s, but the one long bit of idle time is all that matters in this case. It’s particularly clean.

What’s the cause?

The file name associated with this hang is quite peculiar. The full name is:

\Device\Mup\perforce\main\src\lib\public\win64\vdebug_tool.lib#227 – opened for edit

That doesn’t look like a file name. That looks more like the output from Perforce. And that’s exactly what it is. At Valve we store build results in Perforce so we have pre-build steps to check these files out. The checkout commands print their results to the Visual Studio output window like this:

//perforce/main/src/lib/public/win64/vdebug_tool.lib#227 – opened for edit

Visual Studio ‘helpfully’ reverses the slashes and decides that this represents a file name on \\perforce\main. Since this whole thing started with me pressing F8 (actually double-clicking the output window in this reenactment) this means that Visual Studio was trying desperately to treat this potential file name as a source-file name associated with an error or warning message.

Oops.

That explains the CResultList::AttemptToNavigate entry on the hang call stack – everything makes more sense once you understand the problem.

Conclusion

Once the cause of the hang was understood I modified our pre-build step to pipe the output through sed.exe and had it rewrite the output so that Visual Studio would no longer find it interesting. This avoids the hang, but also made it so that F8 would take the selection to interesting errors and warnings, instead of to these mundane progress messages. A little sed magic replaces “//” with the empty string, and “…” with “—“ :

sed -e s!//!! -e s!\.\.\.!—!

This changes the hang-prone results before:

to the hang-proof benign text after:

I also reported the bug to the Visual Studio team. Having a trace is very powerful for this because it meant that I could tell them definitively what the problem was, and I could share the trace in order to let them confirm my findings. Just like minidump files are a powerful way to report crash bugs, xperf traces are a powerful way to report performance bugs. The Visual Studio team has told me that this bug will be fixed in Visual Studio 11 – UNC paths will be ignored by the output window’s parser.

Mup.sys is the driver used for network file I/O. Therefore its presence on the Readying Thread stack was a clue that a network delay was the problem. Doing file I/O on the UI thread is always a bit dodgy if you want to avoid hangs, and doing network file I/O is particularly problematic, so watching for mup.sys is a good idea.

Wait chains

Some wait analysis investigations are more complicated than this one. In several investigations I have found that the main thread of our game was idle for a few hundred milliseconds waiting on a semaphore, critical section, or other synchronization object. In that case the readying thread is critical because that is the thread that released the synchronization object. Once I find out who was holding up the main thread I can move the analysis to that thread and apply either busy-thread analysis (CPU sampling) or idle thread analysis (finding what that thread was waiting on). Usually just one or two levels of hunting is needed to find the culprit, but I did recently trace back across six context switches in four different processes in order to track down an OS scheduling problem.

When following wait chains it is important to understand the order of events. If thread 1234 is readied by thread 5678 at time 10.5 s, then any context switches or CPU activity that happen to thread 5678 after that point are not relevant to the wait chain. Since they happened after thread 1234 was woken they can’t be part of its wait chain.

For CPU activity this is dealt with by selecting the region of interest. For context switches this is dealt with by drilling down all the way and then looking at the SwitchInTime (s) column (which you may want to move to a more convenient location). This column records the time of the context switch.

It’s worth pointing out that if you busy wait (spinning on some global variable flag) or use your own custom synchronization primitives (CSuperFastCriticalSection) then these techniques will not work. The OS synchronization primitives are instrumented with ETW events that allow, in almost all cases, perfect following of wait chains. Even if your custom synchronization code is faster (and it probably isn’t) it isn’t enough faster to make up for the loss of wait analysis. The ability to profile your code trumps any small performance improvement.

Can’t any profiler do this?

Sampling profilers and instrumented profilers might be able to tell you that your program is idle, and they might even be able to tell you where your program is idle, but they generally can’t tell you why your program is idle. Only by following the chain of readying threads and looking at other information can you be sure to find the cause of your idle stalls.

It’s also convenient that you can leave xperf running in continuous-capture mode, where it is constantly recording all system activity to a circular buffer. When you notice a problem you can just record the buffer to disk, and do some post-mortem profiling.

It’s not baking

Baking is all about precisely following a recipe – improvisation tends to lead to failure. Wait analysis, on the other hand, is all about creativity, thinking outside the box, and understanding the entire system. You have to understand context switches, you have to think about what idle time is good and what is bad, you have to think about when to look at CPU usage and when to look at idle time, and you often have to invent some new type of analysis or summary table ordering in order to identify the root cause. It’s not easy, but if you master this skill then you can solve problems that most developers cannot.

While that view is mostly true for old school languages such as C++ and Java, the game is different for dynamic languages: Lua, JavaScript, Python, etc. That can be easy to forget if you spend most of your time in the static world, so in this article I’m going to show some of the tricks you can apply when everything is fluid and malleable.

At Bitsquid our dynamic language of choice is Lua. Lua has the advantage of being fast, fully dynamic, small, simple and having a traditional (i.e. non-LISP-y) syntax. We use Lua for most gameplay code and it interfaces with the engine through an API with exposed C functions, such as World.render() or Unit.set_position().

I will use Lua in all the examples below, but the techniques can be used in most dynamic languages.

1. Read-eval-print-loop

Dynamic languages can compile and execute code at runtime. In Lua, it is as simple as:

loadstring("print(10*10)")()

This can be used to implement a command console where you can type Lua code and directly execute it in the running game. This can be an invaluable debugging and tuning tool. For example if you need to debug some problem with the bazooka:

World.spawn_unit("bazooka", Unit.position(player))

Or tune the player’s run speed:

Unit.set_data(player, "run_speed", 4.3)

2. Reload code

The console can be used for more than giving commands, you can also use it to redefine functions. If the gameplay code defines a scoring rule for kills:

function Player.register_kill(self, enemy)
	self.score = self.score + 10
end

you can use the console to redefine the function and change the rules:

function Player.register_kill(self, enemy)
	if enemy.type == "boss" then
		self.score = self.score + 100
	else
		self.score = self.score + 10
	end
end

Executing this code will replace the existing Player.register_kill function with the new one. All code that previously called the old function will now call the new one and the new scoring rules will apply immediately.

If you take some care with how you use the global namespace you can write your Lua code so that all of it is reloadable using this technique. Then the gameplay programmer can just edit the Lua files on disk and press a key to reload them in-game. The game will continue to run with the new gameplay code, without any need for a reboot. Pretty cool.

You can even get this to work for script errors. If there is an error in the Lua code, don’t crash the game, just freeze it and allow the gameplay programmer to fix the error, reload the code and continue running.

3. Override system functions

The functions in the engine API don’t have any special privileges, they can be redefined just as other Lua functions. This can be used to add custom functionality or for debugging purposes.

Say, for example, that you have some units that are mysteriously popping up all over the level. You know they are being spawned somewhere in the gameplay code, but you can’t find where. One solution would be to override the World.spawn_unit function and print a stack trace whenever the offending unit is spawned:

old_spawn_unit = World.spawn_unit
function World.spawn_unit(type, position)
	if type == "tribble" then
		print "Tribble spawned by:"
		print_stack_trace()
	end
	old_spawn_unit(type, position)
end

Now, whenever a tribble is spawned by the script, a call stack will be printed and we can easily find who is doing the spawning.

Note that before we replace World.spawn_unit, we save the original function in the variable old_spawn_unit. This enables us to call old_spawn_unit() to do the actual spawning.

This technique could also be used to find all (potentially expensive) raycasts being done by the script.

4. Handle deprecated functions

Sometimes we need to deprecate functions in the engine API. It can be annoying to the people using the engine of course, but backwards compatibilty is the mother of stagnation. If you never throw away old code, you will eventually have a huge ugly code mess on your hands.

Luckily, since the script can create functions in the engine namespace, the script can provide the backwards compatibility when needed.

For example, we used to have a function PhysicsWorld.clear_kinematic(world, actor). That naming was inconsistent with some of our other functions so we changed it to Actor.set_kinematic(actor, false).

One way of dealing with this change would be to go through all the code in the project, find all uses of PhysicsWorld.clear_kinematic and change them to use Actor.set_kinematic instead. But another way would be to just implement PhysicsWorld.clear_kinematic in the script:

function PhysicsWorld.clear_kinematic(world, actor)
	Actor.set_kinematic(actor, false)
end

Now the rest of the code can go on using PhysicsWorld.clear_kinematic without even caring that the function has been removed from the engine API. You could even use a combination of the two strategies — implementing the deprecated function in Lua for a quick fix, and then looking into removing the uses of it.

5. Dynamically inserting profiling

Top-down profiling with explicit profiler scopes is a good way of finding out where a game is spending most of its time. However, to be useful, explicit profiler scopes need to be inserted in all the “right” places (all potentially expensive functions).

In C we need to guess where these right places are before compiling the program. In Lua, we can just insert the profiler scopes dynamically. We can even create a function that adds profiling to any function we want:

function profile(class_name, method_name)
	local f = _G[class_name][method_name]
	_G[class_name][method_name] = function (...)
		Profiler.start(class_name .. "." .. method_name)
		f(...)
		Profiler.stop()
	end
end

When we call this function as profile(‘Player’, ‘update’) it will first save the existing Player.update function and then replace it with a function that calls Profiler.start(“Player.update”) before calling the original function and Profiler.stop() before returning.

Using this techinque, we can dynamically add profiling to any function we want during our optimization session.

6. Tab completion

If you implement an interactive Lua console, it is nice to support tab completion, so the user doesn’t have to remember all function names. But how do you build the list of callable functions to use with tab completion?

Using Lua of course! Just find all tables (i.e., classes) in the global namespace and all functions stored in those tables:

t = {}
 
for class_name,class in pairs(_G) do
	if type(class) == 'table' then
		for function_name,function in pairs(class) do
			if type(function) == 'function' then
				t[#t+1] = class_name .. '.' .. function_name
			end
		end
	end
end

After running this, t will contain the full list of function names.

7. Looping through all objects

By recursing through _G you can enumerate all reachable objects in the Lua runtime.

function enumerate(f)
	local seen = {}
	local recurse = function(t)
		if type(t) ~= 'table' then return end
		if seen[t] == true then return end
		f(t)
		seen[t] = true
		recurse(getmetatable(t))
		for k,v in pairs(t) do
			recurse(k)
			recurse(v)
		end
	end
	recurse(_G)
end

Calling enumerate(f) will call f(o) on all objects o in the runtime. (Assuming they are reachable from _G. Potentially, there could also be objects only reachable through Lua references held in C.)

Such an enumeration could be used for many things. For example, you could use it to print the health of every object in the game.

function print_health(o)
	if o.health then print(o.health) end
end
enumerate(print_health)

The technique could also be used for memory optimizations. You could loop through all Lua objects and find the memory used by each object type. Then you could focus your optimization efforts on the resource hogs.

This has also been posted to The Bitsquid Blog.

Usually, in graphics application, one can limit it to the three-dimensional space. In that case, there are four possible approaches, all of which guarantee a uniform distribution(BTW, as for what the ‘uniform distribution’ exactly means, [6] has some explanations). If a n-dimension support is required, one is out, so three remain. Let’s take stock of each.

Rejection sampling ([2][4][5])

One simple way is something called ‘rejection sampling’. For each x, y, z coordinates, choose a random value of a uniform distribution between [-1, 1]. If the length of the resulting vector is greater than one, reject it and try again. Obviously, this method can be generalized to n-dimension. But the bigger the dimension gets, the higher the rejection rate gets, so the less efficient the technique becomes.

Normal deviate ([2][5])

This technique chooses x, y and z from a normal distribution of mean 0 and variance 1. Then normalize the resulting vector and that’s it. [2] shows why this method can generate a uniform distribution over a sphere. In short,

It works because the vector chosen (before normalization) has a density that depends only on the distance from the origin.

as [5] explains. This also generalizes to n-dimension without a hassle.

Trigonometry method ([1][3][4][5])

This one works only for a three-dimensional sphere(called 2-sphere in literatures, which means it has two degrees of freedom), but is an easiest one to intuitively grab how it works. [1] nicely explains why it works from Archimedes’ theorem:

The area of a sphere equals the area of every right circular cylinder circumscribed about the sphere excluding the bases.

The exact steps are as below:

• Choose z uniformly distributed in [-1,1].
• Choose t uniformly distributed on [0, 2*pi).
• Let r = sqrt(1-z^2).
• Let x = r * cos(t).
• Let y = r * sin(t).

This is the one I used for my shader effect. Since I had to use a very small number of samples for the sake of performance, I did a stratified sampling with this method. A straightforward extension to a stratified sampling is another advantage of this technique.

Coordinate approach ([2][3][5])

The last one is applicable to general n-dimensions and [2] explains its quite math-heavy derivation in detail. This technique first gets the distribution of a single coordinate of a uniformly distributed point on the N-sphere. Then, it recursively gets the distribution of the next coordinate over (N-1)-sphere, and so on. Fortunately, for the usual 3D space(i.e. 2-sphere), the distribution of a coordinate is uniform and one can do a rejection sampling on 2D for the remaining 1-sphere(i.e. a circle). The exact way is explained in [5] as a variation of the trigonometry method.

Codes and Pictures

Even if you haven’t got it all fully up to this point, don’t worry. The source code will fill up the gaps in your understanding. You can find my naive C++ implementations of techniques above here: http://ideone.com/oYEVR

And some pretty pictures of random points generated by each method:

500 points by rejection sampling

500 points by normal deviate

500 points by trigonometry method

500 points by coordinate approach

BTW, all the images above were plotted by Winplot.

TL;DR

Just use the trigonometry method above and add a stratified sampling, if necessary. That’ll be enough for the most of cases. ;)

References

1. http://repository.upenn.edu/cgi/viewcontent.cgi?article=1188&context=cis_reports
2. http://www-alg.ist.hokudai.ac.jp/~jan/randsphere.pdf
3. http://mathworld.wolfram.com/SpherePointPicking.html
4. http://cgafaq.info/wiki/Uniform_random_points_on_sphere
5. http://www.math.niu.edu/~rusin/known-math/96/sph.rand
6. http://www.math.niu.edu/~rusin/known-math/95/sphere.faq

(This article has also been posted to my personal blog.)

Part 1: Effective use of SQL Server

SQL Server is almost completely devoid of “magic performance tricks.”  This article isn’t going to be about tuning configuration settings; instead, it’s going to cover how to give yourself enough flexibility and freedom to spot performance problems early, and make changes to improve performance when you identify a problem.

Design Flexibility

Before we start, let’s cover a few helpful design ideas that can make performance tuning easier in the future.  Each one of these could be discussed as its own topic, but let’s do a quick top level view.

• Use SQL only for data that must be ACID compliant, relational, changeable, persistent, and cannot be recreated easily.  For example, SQL is a perfect place to store player account data, item possession data, and auction house data.  It’s not an ideal place for text chat logs (not relational – just store them in a text repository), multiplayer game invitations (not persistent – users will recreate them after a crash), images (these don’t really change – just save them in files that can be varnish-cached), and so on.
• Design your SQL environment to be split apart.  If you have eight or nine types of data in your system, make them each their own databases.  That way, when your game suddenly explodes in size, each one can be scaled upward at its own pace.  Within a single database, I find that it’s useful to give each subsystem its own three-character prefix so that I can rapidly group related objects together. The only drawback is that you won’t be able to use foreign keys across distributed databases; but in most cases that’s necessary for high performance anyway.
  

  Isolating Tables

• Use SQL licensing effectively.  Since SQL Express is basically the same engine as SQL Standard with a size limitation, I like to use it in proof of concept environments.  When the database grows larger, I can migrate it to SQL Standard with just a new connection string.
• If you have a physical SQL server, put your TempDB on its own, physically isolated disk drive.  Ideally, use a sub-$100 SSD for this.  There are lots of nifty tricks you can do with a high performance TempDB drive.
• Get an MSDN subscription and use your development SQL license to have extra non-production servers sitting around.  This means you can always experiment with a new database configuration.
• Don’t write SQL directly into your application.  Using stored procedures mostly removes the SQL language parser overhead.  Additionally, when you discover a performance problem, a stored procedure can be surgically altered while still in production.
• Make use of your database maintenance plans.  Note that these only work on SQL Standard; but it can be awfully useful to have your database run a checkpoint and incremental backup every hour.  Indexes should be rebuilt nightly.
• Learn “Third Normal Form”.  The full details of 3NF are quite complex, but in practice, what it means is that each datum should exist in the database once and only once.  A classic example is storing the department name for an employee database; here’s the right and wrong way to do it:
  
  

  Move your data into specialized tables with useful primary keys.

I should also warn you that databases are easy to take for granted. SQL Server is a very friendly, deceptively simple program.  Often the dev database you create will seem so fast that you’ll be tempted to ignore performance tuning, until the production server grinds to a halt on deploy day. So let’s review a few tricks to enable you to see these problems before they get pushed live.

Spotting Performance Problems

Everyone’s best friend should be “Show Execution Plan.”  For each query you write, click the little button that shows either the estimated or actual execution plan. The results will look like this:

An execution plan that could be optimized.

In this screen, you see a “Table Scan” that consumes a large amount of time that could be optimized out by creating an index.  It’s the equivalent to having an unstructured array and scanning through every element blindly.  Adding an index allows your database to use the equivalent of adding a red/black tree or hash table to look up data faster.  The great value of SQL Server is that you can dynamically tune the indexes your application uses without modifying your code!

You should look through the execution plan for the following:

• Missing indexes. SQL Server will helpfully tell you, highlighted in green, the exact index it wishes it had.  Right click on it and select “Missing Index Details;” you can then give it a name and create it for an instant performance boost …… but remember that this isn’t always the best idea.  Each index you create slows down inserts, updates, and deletes.  Indexes also consume memory, which can result in page faults when you’re under memory pressure.  If you can, make all your indexes use integer IDs, since they are very memory-dense and fast to search.  Organize indexes with multiple values so that the most restrictive values are first – that will allow the query optimizer to exclude large volumes of rows rapidly.
• Table Scans. As described above, if SQL server is spending a significant amount of time scanning your tables, try to figure out ways to modify your indexes to convert them to “Index Seek” instead.
• Too many joins.  If your execution plan is really tall and wide, this usually means that SQL server is joining lots of tables together in a single step.  Lots of joins are an inescapable side effect of a normalized database; I’ll write a future article on the subject in detail. But for the moment, look for any tables that can be optimized away. Try declaring a variable to contain some of the values you want and selecting them early.  For example, look at these two equivalent SQL snippets – one of them only joins two tables, and the other joins three:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

-- SLOW WAY: Get weapons sales by querying three tables
SELECT i.item_name, i.item_id, h.price, h.unit_count
  FROM act_auctionhistory h
       INNER JOIN itm_items i ON i.item_id = a.item_id
       INNER JOIN itm_itemclasses ic ON ic.itemclass_id = i.itemclass_id
 WHERE ic.itemclass_name = 'weapon'
       GROUP BY i.item_id, i.item_name
 
-- FAST WAY: Get the same data but use only two tables at a time
DECLARE @cid INT
SELECT @cid = itemclass_id FROM itm_itemclasses WHERE itemclass_name = 'weapon'
 
SELECT i.item_name, i.item_id, h.price, h.unit_count
  FROM act_auctionhistory h
       INNER JOIN itm_items i ON i.item_id = a.item_id
 WHERE i.itemclass_id = @cid
       GROUP BY i.item_id, i.item_name

Here’s another tip. Log on to your live server (make sure you use read-only privileges of course!). Right click on your SQL Server in the left hand pane and select “Activity Monitor”, then “Recent Expensive Queries”.  I find that I get a lot of mileage out of just checking this page every few hours:

Check this page regularly!

So when you get see this screen, what do you look for and how do you make use of the information? Start by right clicking on each query and show the execution plan for each one.

Log Your Database Activity

Another worthwhile tip is to modify your code to write a log of database queries to your debug output.  When you’re using advanced ORM tools like Entity Framework or NHibernate, it’s very easy to write code that has a side effect of generating unexpected database hits.  For example, consider this simple logic:

1
2
3
4
5

for (int i = 0; i < itemarray.Length; i++) {
    if (itemarray[i].IsQuestObject()) {
        return true;
    }
}

In most cases this logic is fine and fast.  But imagine that the programmer down the hall fixes a bug by making IsQuestObject() ping the database – all of a sudden you’ve got a nearly-invisible massive performance penalty. The database is often so fast that you don’t notice these kinds of performance sinks. So I modify my ORM to emit debug output every time it executes a SQL statement.

As a result, when running in debug, I see the following:

1
2
3
4
5
6
7
8

SQL: list_OrderedUserAccounts (Time: 00:00:00.0030000)
SQL: get_PermissionsByUser (Time: 00:00:00.0020000)
SQL: RetrievePermissionGroupObject (Time: 00:00:00.0020000)
SQL: get_JoinsByLabel (Time: 00:00:00.0010000)
SQL: get_JoinsByLabel (Time: 00:00:00.0010000)
SQL: get_JoinsByLabel (Time: 00:00:00.0010000)
SQL: ListProductObject (Time: 00:00:01.3680000)
SQL: list_OrderedPlatforms (Time: 00:00:00.0020000)

When I look at this output, I immediately check to see if I’ve accidentally written a loop that calls get_JoinsByLabel too many times.  Then next I investigate the query “ListProductObject” to see why it took 1.5 seconds.

Combine multiple queries

Nothing improves performance quite like reducing the number of queries you execute.  Rather than submitting ten queries rapidly, why not make use of a single stored procedure that returns ten result sets?  Let’s say you’re working on the front page of your in-game auction house, but when the user mouses over an item name or a player name, you want to show some secondary details.

You could do this by writing one query for the auction house page and a second query for each time the user hovers their mouse.  But that would produce dozens of queries every time the mouse moved.

Instead, let’s create a single compound query that returns both the auction page and the popup text for everything.  Even though at first glance you may think you’re doing unnecessary work, SQL Server is already looking up all the objects using hash tables, and you’re reducing query overhead significantly by reducing volume.  Here’s roughly how to do it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

CREATE PROCEDURE get_AuctionHouseFirstPage AS
 
-- Preparation: Select basic into a temporary table
SELECT auction_id, character_id, auction_date, …
  FROM ACT_Auctions, …
  INTO #temp_auction_page
 
-- First result set: Return the auction page
SELECT ... FROM #temp_auction_page
 
-- Second result set: Return pop-up information about characters
SELECT user_name, guild_name, …
  FROM CHR_Characters c
       INNER JOIN #temp_auction_page t ON c.char_id = t.char_id
 
-- Third result set: Return pop-up information about items
SELECT item_name, item_enchantment, ...
  FROM ITM_Items i
       INNER JOIN #temp_auction_page t ON i.item_id = t.item_id

For Next Time

Many of you may be asking – what about SQL vs NoSQL databases? I encourage you to tread lightly and consider them carefully on their own merits. Unlike SQL, which provides specific guarantees to data behavior at the cost of performance, NoSQL databases have their own unique advantages and drawbacks. There are many situations in gaming that are extremely well suited for NoSQL implementations; but this is a complex issue that really deserves its own article.

For the next SQL Server performance article, I’ll look at the speed of database inserts and updates, and the side effects of some basic choices in software style.

Continue with the previous post, after filling the triangle with scan line or half-space algorithm, we also need to interpolate the vertex attributes across the triangle so that we can have texture coordinates or depth on every pixel. However we cannot directly interpolate those attributes in screen space because projection transform after perspective division is not an affine transformation (i.e. after transformation, the mid-point of the line segment is no longer the mid-point), this will result in some distortion and this artifact is even more noticeable when the triangle is large:

interpolate in screen space

perspective correct interpolation

Condition for linear interpolation

When interpolating the attributes in a linear way, we are saying that given a set of vertices, vi (where i is any integer>=0) with a set of attributes ai (such as texture coordinates), we have a function mapping a vertex to the corresponding attributes, i.e.

Say, to interpolate a vertex inside a triangle in a linear way, the function f need to have the following properties:

which means that we can calculate the interpolated attributes using the same weight ti used for interpolating vertex position. For functions having the above properties, those functions will be an affine function with the following form:

Depth interpolation

When a vertex is projected from view space to normalized device coordinates(NDC), we will have the following relation (ratio of the triangles) between the view space and NDC space:

substitute equation 1 and 2 into the plane equation of the triangle lies on:

So, 1/zview is an affine function of xndc and yndc which can be interpolated linearly across the screen space (the transform from NDC space to screen space is a linear transform).

Attributes interpolation

In last section, we know how to interpolate the depth of a pixel linearly in screen space, the next problem is to interpolate the vertex attributes(e.g. texture coordinates). In view space, we know that those attributes can be interpolated linearly, so those attributes can be calculated by an affine function with the vertex position as parameters e.g.

Similar to interpolate depth, substitute equation 1 and 2 into the above equation:

Therefore, u/zview is an another affine function of xndc and yndc which can be interpolated linearly across the screen space. Hence we can interpolating u linearly by first interpolate 1/zview and u/zview across screen space, and then divide them per pixel.

The last problem…

Now, we know that we can interpolate the view space depth and vertex attributes linearly across screen space. But during the rasterization state, we only have vertices in homogenous coordinates (vertices are transformed by the projection matrix already), how can we get the zview to do the perspective correct interpolation?

Consider the projection matrix (I use D3D one, but the same for openGL):

After transforming the vertex position, the w-coordinate will be the view space depth!

i.e. w-homogenous = zview 

And look at the matrix again and consider the transformed z-coordinates, it will in a form of:

After transforming to the NDC,

So the depth value can be directly interpolated using z-NDC  for depth test.

Demo

A javascript demo to rasterize the triangles can be viewed here(although not optimized…). And the source code can be downloaded here.

Screen shot of the demo

Conclusion
In this post, the steps to linear interpolate the vertex in screen space is described. And for rasterizing the depth buffer only (e.g. for occlusion), the depth value can be linearly interpolated directly with the z coordinate in NDC space which is even simpler.

References
[1] http://www.lysator.liu.se/~mikaelk/doc/perspectivetexture/
[2] http://www.gamedev.net/topic/581732-perspective-correct-depth-interpolation/
[3] http://chrishecker.com/Miscellaneous_Technical_Articles
[4] http://en.wikipedia.org/wiki/Affine_transformation

 

Currently I’m writing the story for the game Connectorium. It’ll be the second story I’m writing in full, after co-writing the Revival short film (I’m not counting the one or two short stories here and there, and a failed attempt at writing a fantasy novel after watching the first Lord of the Rings film. Who didn’t do that, right?). Here are just a collection of random thoughts, observations, and experiences about the process. Obviously these are not the opinions of an expert; I’m merely hoping it opens up the way for a conversation and invites thoughts from you.

From Abstraction to Realization

This is something that is universal to the creative process. You begin with an empty canvas. Maybe a concept that is completely abstract and vague. Then with every sentence, with every stroke of a brush, with every added note, or with every line of code, you bring that abstraction one step closer to existence (and also the number of possibilities of what that end product will be reduces with every step). But there is a key thing I have realized: this is a two way process. The original idea, or concept affects what you create. But what you create also affects the idea over time. To a point where the final product may in no way resemble the original idea. I think this a very important part of the creative process: the organic nature of it.

As far as a story goes, that initial concept and idea can be many different things. Maybe it’s a particular character, or a specific plot point. Maybe it’s a particular setting. Maybe it’s a mechanic in the game you are designing. Either way, it’s important to keep in mind that your completed story may be nothing like what you had initially conceived. And that’s OK. In fact it’s more than OK. It’s usually a good thing.

Working Backwards

When I first started working on Connectorium, I had a general idea for the story. The game is about systems and connections, so the story was going to be about a little girl who wakes up one morning to a world where all connections have gone missing. Her adventure would be about her meeting various characters, helping them restore the missing connections, and solving the mystery. For some time though, I stalled fleshing out the story more. Eventually I asked myself, why am I wasting time? Why don’t I just write the story? And it occurred to me: it’s because I didn’t know how it’s going to end.

So one morning I decided to take my iPad, go to a quiet park, and not come back home until I have figured out how the story will end. It took a couple of hours, but eventually I came up with an idea, quite suddenly really. I had a big smile on my face right at that moment, because I knew I could start writing the story now.

Maybe this is more a function of the kinds of story that I enjoy and like to write, but I find that I really need to know the ending early on. Everything in the plot, the characters, the gameplay in the case of a game, is pushing the audience towards that ending. It’s what keeps the story coherent to me.

Characters or Plot

One of my favorite writers, Isaac Asimov, is often criticized for having somewhat uninteresting and 2D characters. Nevertheless he is an amazing story teller.

But one can’t argue that the best of stories combine a great plot, with believable and great characters. What I have noticed is that personally I’m much more interested and focused on the plot. So I always need to be conscious of the “flatness” of my characters. For that reason, after I have written the initial draft of the story, I’ll do an iteration where I’ll focus specifically on each character, writing more back story, fixing the dialog, descriptions, and so on, of course adjusting the plot where necessary. I can imagine the reverse can work just as well: building a detailed and interesting character, and developing the story around that character (or characters).

Dialog, Dialog, Dialog

For me, probably the hardest part of writing a story is the dialog. Not only is it really hard to write a believable, natural, and flowing conversation between two or more characters, it’s even harder to have all your characters not sound exactly the same! Exactly like…you!

More than anything, it just requires time, and rewrites to improve this. It is also important to have back stories for characters, even if none of it is ever revealed to the audience. Where do they come from? What do they do? What do they eat? What was their childhood like? What are their relationships like? What is their motivation? All of these impact how a character speaks, how they would react to a situation, and how they’d express themselves.

Another thing that has helped me is trying to picture a real life person acting out that character. Maybe someone you know, or an actor. Putting a face and voice to a line of dialog goes a long way to help you see if it’s the right fit. Sometimes reading it out loud in the voice that you think the character would be speaking in also helps here.

On the Subject of Games

I’ve been talking a lot about stories, and haven’t really talked much about games. Here is point I want to make which I can expect at least some to disagree with.

I feel that the gameplay must reinforce the story as much as possible. At the very least it shouldn’t contradict it, because that takes you out of the immersion that you might otherwise have. How often do you run around in a game, killing various things, and collecting numerous items, stats, etc, just to be reminded by a cut scene that you’re actually trying to resolve a much greater conflict.

And here is another (potentially less popular) thought. Given that there are practically infinite possible stories, why is it that a good percentage of games, especially those with plots and characters, include combat in some form as their core mechanic? Is it that we are simply avoiding stories where combat isn’t an integral component? Or are we throwing in combat into the mix, regardless of whether or not it reinforces the story?

Just a thought. Would love to hear yours.

Procedural Generation is definitely in vogue, and I personally have believed that it is the way forward in video gaming for many years now. Using procedural generation in games is nothing new of course, as fans of games such as Elite or The Sentinel will know that we’ve been seeing it in games for a good 25 years.

Older titles made good use of it due to the memory constraints of the hardware of the time. It was simply more efficient to have generated levels rather than hand crafted ones, but that is no excuse for games not to make better use of it now that we have better specced hardware.

Fans of the RPG genre will no doubt remember The Elder Scrolls II: Daggerfall, which had one of the largest in-game worlds ever seen, and still to this day tramples almost every RPG made in terms of world size. I recall reading somewhere that the in-game world of Daggerfall was equal to twice the landmass of the British Isles.

That is a heck of a lot of world to explore, and – from a game design perspective – a nightmare to recreate by hand. Through clever use of procedural generation, however, it is easily possible, which is what Bethesda Softworks did with Daggerfall. The settlements and towns were hand-crafted, with the wilderness in between being generated by the game.

But why stop there? Why have world borders at all? Procedural generation code hasn’t changed much in the last 25 years. People are still stuck using fractals and diamonds and blobs to do everything, which becomes repetitive and quite simply looks like procedurally generated content. To any programmer looking at it, it virtually smells of procedural generation. On top of all this, if you get it wrong, it will end up VERY wrong. The indie crowd seems to do it best, with titles like Dwarf Fortress generating MASSIVE worlds with lush histories and more world than you could ever hope to explore. But still, they aren’t pushing the envelope. My aim was to fix that by making it work. An infinite game world should be possible, and indeed it is.

The Idea

Just over two years ago I began assembling a team to make the first truly infinite, fully 3D fantasy RPG, entitled Malevolence: The Sword of Ahkranox. It was to be played in a style similar to the classic grid-based, first person RPGs of the late 80s and early 90s such as Might & Magic, Eye of the Beholder and Dungeon Master, but set in a literally infinite world. We had originally thought to make it a planet-sized world, but in the end decided on the story being that the game’s world was being created within the imagination of a sentient sword, which would act as a way to “explain” the infinity of it.

After much experimentation and very complex math, we got it working, but all in raw data. Nothing really playable. But we had in front of us an infinite world filled with infinite dungeons and infinite cities filled with infinite NPCs. We then worked to get a game working in such a world (some of the efforts of which, you may have read about in my last post)

Now, just to confirm, this world wasn’t being randomly generated. It was both infinite AND persistent. Without going into too much detail, this is achieved by making the world dynamically affected by the passing of time. Every part of the world is identified as either affected by time or timeless. The lay of the land with its hills and caverns… That’s all timeless, and never changes. Because those parts never change and cannot be affected by the player, they only need to be loaded into memory when the player can see them (or if they are needed to generate quest information, etc). However, if an object is affected by time (for example, the contents of a chest), then they have a time coefficient applied to the procedural algorithm that generates them. This means that a chest in a dungeon, for example, will have different items in it depending on WHEN the player opens the chest. If the player was the clear out that chest, that act is stored in a database of player changes, but then re-set when a certain amount of time has passed. This ensures that the database of player changes to the world never exceeds a certain size (which is estimated to be around 250mb at the very most, but more realistically around 50mb)

This generation accounts for almost everything in the game. Spell creation, item creation, weapon creation, potion creation, NPC dialogue system, even the spell effects that happen on the screen. Due to this, the world that the player explores will be ever-changing and infinite. They won’t keep finding the same old weapons or items, there will be no end to the number of spells they can find or use, they won’t even keep having the same conversations with NPCs. This is necessary to keep a player interested for long enough in an infinite world.

Public Acceptance

Back when the game Elite was first being worked on, it was planned to have around 282 trillion galaxies with around 256 star systems in each one, but their publisher, Firebird, were worried that such a large in-game universe would be intimidating to players and put them off. I have to say I had wondered at that, and was interested to see how the public would react to an even bigger in-game world.

I was surprised at the results.

We’ve been quite public with our development process for the game so far and generated a small cult following on communities such as IndieDB, but very few people seem to quite grasp the scale of an infinite world, despite our thorough descriptions of it. We had put up renders of the world generation data, showing just a tiny fraction of the world:

And then, we showed them this:

That inland sea is around the size of the entire in-game world of Skyrim. Funnily enough, the largest response we got from this information was disbelief. Many called us liars and that it simply wasn’t possible. Others began to believe that the world size of Malevolence was the entire above image, rather than infinite. Only about 20% of people really understood.

So, from a marketing perspective, it’s been a bit of a nightmare to have an infinite world. We’ve even had many suggest that Malevolence is just a rip-off of Legend of Grimrock, despite the fact that Malevolence was started about a year before. But that’s always going to happen, no matter what the game. What happens upon release will happen, and that’s just how the cookie crumbles with game development. Funnily enough, that hasn’t been the hardest bit. The hardest bit has been the math involved in making a world like this one.

The Math

Being infinite, procedural AND persistent, most of the mathematics behind Malevolence is theoretical math – that is, mathematics with few or no fixed/known values acting in a volatile space. But we’ve broken the world creation down into multiple layers.

The first layer is the one you saw above. A large world segment is generated which covers an area of about 400x400km. This is the only layer of the game that uses a standardised procedural generation system (perlin noise)

That is then broken down into chunks that are around 3x3km, calculating the biome information within that area, like so:

In the end, all of these steps need to be completed when each new world segment is generated in order to turn the raw data into this:

That is just for the overworld. Every world segment that is VISIBLE to the player (as in the view above) is given a unique code, generated by the procedural algorithm. If there is a dungeon entrance in that segment, the dungeon is generated using this unique code, ensuring that every time the player returns to that spot, the same dungeon will be there:

This same method is used for town generation, graveyards, ruins or anything else that the player may encounter. And this goes on forever. If a player was to turn off collision and hold down the ‘move forward’ button, it would take them just under three weeks to walk from one end of a world segment to another, and then they would simply move to a new world segment seamlessly, and then another, forever.

The biggest question we have been given is how we have dealt with the data type limitations on player co-ordinates, but unfortunately we can’t give away all our secrets :) But I can tell you that Malevolence doesn’t suffer from the Minecraft world-edge issue, it just keeps going on and on.

Conclusion

Using procedural generation in your game can be a rewarding experience, but definitely don’t rush into it. It takes good planning, clever usage and most of all it needs to feel seamless, otherwise the public simply won’t accept it.

If you’d like to read more about Malevolence: The Sword of Ahkranox, you can check out these links:

My pragmatic summary:  A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in.  In a multithreaded environment, the lack of understanding and the resulting problems are greatly amplified, almost to the point of panic if you are paying attention.  Programming in a functional style makes the state presented to your code explicit, which makes it much easier to reason about, and, in a completely pure system, makes thread race conditions impossible.

I do believe that there is real value in pursuing functional programming, but it would be irresponsible to exhort everyone to abandon their C++ compilers and start coding in Lisp, Haskell, or, to be blunt, any other fringe language.  To the eternal chagrin of language designers, there are plenty of externalities that can overwhelm the benefits of a language, and game development has more than most fields.  We have cross platform issues, proprietary tool chains, certification gates, licensed technologies, and stringent performance requirements on top of the issues with legacy codebases and workforce availability that everyone faces.

If you are in circumstances where you can undertake significant development work in a non-mainstream language, I’ll cheer you on, but be prepared to take some hits in the name of progress.  For everyone else: No matter what language you work in, programming in a functional style provides benefits.  You should do it whenever it is convenient, and you should think hard about the decision when it isn’t convenient.  You can learn about lambdas, monads, currying, composing lazily evaluated functions on infinite sets, and all the other aspects of explicitly functionally oriented languages later if you choose.

C++ doesn’t encourage functional programming, but it doesn’t prevent you from doing it, and you retain the power to drop down and apply SIMD intrinsics to hand laid out data backed by memory mapped files, or whatever other nitty-gritty goodness you find the need for.

Pure Functions

A pure function only looks at the parameters passed in to it, and all it does is return one or more computed values based on the parameters.  It has no logical side effects.  This is an abstraction of course; every function has side effects at the CPU level, and most at the heap level, but the abstraction is still valuable.

It doesn’t look at or update global state.  it doesn’t maintain internal state.  It doesn’t perform any IO.  it doesn’t mutate any of the input parameters.  Ideally, it isn’t passed any extraneous data – getting an allMyGlobals pointer passed in defeats much of the purpose.

Pure functions have a lot of nice properties.

Thread safety.  A pure function with value parameters is completely thread safe.  With reference or pointer parameters, even if they are const, you do need to be aware of the danger that another thread doing non-pure operations might mutate or free the data, but it is still one of the most powerful tools for writing safe multithreaded code.

You can trivially switch them out for parallel implementations, or run multiple implementations to compare the results.  This makes it much safer to experiment and evolve.

Reusability.  It is much easier to transplant a pure function to a new environment.  You still need to deal with type definitions and any called pure functions, but there is no snowball effect.  How many times have you known there was some code that does what you need in another system, but extricating it from all of its environmental assumptions was more work than just writing it over?

Testability.  A pure function has referential transparency, which means that it will always give the same result for a set of parameters no matter when it is called, which makes it much easier to exercise than something interwoven with other systems.   I have never been very responsible about writing test code;  a lot of code interacts with enough systems that it can require elaborate harnesses to exercise, and I could often convince myself (probably incorrectly) that it wasn’t worth the effort.  Pure functions are trivial to test; the tests look like something right out of a textbook, where you build some inputs and look at the output.  Whenever I come across a finicky looking bit of code now, I split it out into a separate pure function and write tests for it.  Frighteningly, I often find something wrong in these cases, which means I’m probably not casting a wide enough net.

Understandability and maintainability.  The bounding of both input and output makes pure functions easier to re-learn when needed, and there are less places for undocumented requirements regarding external state to hide.

Formal systems and automated reasoning about software will be increasingly important in the future.  Static code analysis is important today, and transforming your code into a more functional style aids analysis tools, or at least lets the faster local tools cover the same ground as the slower and more expensive global tools.  We are a “Get ‘er done” sort of industry, and I do not see formal proofs of whole program “correctness” becoming a relevant goal, but being able to prove that certain classes of flaws are not present in certain parts of a codebase will still be very valuable.  We could use some more science and math in our process.

Someone taking an introductory programming class might be scratching their head and thinking “aren’t all programs supposed to be written like this?”  The reality is that far more programs are Big Balls of Mud than not.  Traditional imperative programming languages give you escape hatches, and they get used all the time.  If you are just writing throwaway code, do whatever is most convenient, which often involves global state.  If you are writing code that may still be in use a year later, balance the convenience factor against the difficulties you will inevitably suffer later.  Most developers are not very good at predicting the future time integrated suffering their changes will result in.

Purity In Practice

Not everything can be pure; unless the program is only operating on its own source code, at some point you need to interact with the outside world.  It can be fun in a puzzly sort of way to try to push purity to great lengths, but the pragmatic break point acknowledges that side effects are necessary at some point, and manages them effectively.

It doesn’t even have to be all-or-nothing in a particular function.  There is a continuum of value in how pure a function is, and the value step from almost-pure to completely-pure is smaller than that from spaghetti-state to mostly-pure.  Moving a function towards purity improves the code, even if it doesn’t reach full purity.  A function that bumps a global counter or checks a global debug flag is not pure, but if that is its only detraction, it is still going to reap most of the benefits.

Avoiding the worst in a broader context is generally more important than achieving perfection in limited cases.  If you consider the most toxic functions or systems you have had to deal with, the ones that you know have to be handled with tongs and a face shield, it is an almost sure bet that they have a complex web of state and assumptions that their behavior relies on, and it isn’t confined to their parameters.  Imposing some discipline in these areas, or at least fighting to prevent more code from turning into similar messes, is going to have more impact than tightening up some low level math functions.

The process of refactoring towards purity generally involves disentangling computation from the environment it operates in, which almost invariably means more parameter passing.  This seems a bit curious – greater verbosity in programming languages is broadly reviled, and functional programming is often associated with code size reduction.  The factors that allow programs in functional languages to sometimes be more concise than imperative implementations are pretty much orthogonal to the use of pure functions — garbage collection, powerful built in types, pattern matching, list comprehensions, function composition, various bits of syntactic sugar, etc.  For the most part, these size reducers don’t have much to do with being functional, and can also be found in some imperative languages.

You should be getting irritated if you have to pass a dozen parameters into a function; you may be able to refactor the code in a manner that reduces the parameter complexity.

The lack of any language support in C++ for maintaining purity is not ideal.  If someone modifies a widely used foundation function to be non-pure in some evil way, everything that uses the function also loses its purity.  This sounds disastrous from a formal systems point of view, but again, it isn’t an all-or-nothing proposition where you fall from grace with the first sin.  Large scale software development is unfortunately statistical.

It seems like there is a sound case for a pure keyword in future C/C++ standards.  There are close parallels with const – an optional qualifier that allows compile time checking of programmer intention and will never hurt, and could often help, code generation.  The D programming language does offer a pure keyword:  http://www.d-programming-language.org/function.html  Note their distinction between weak and strong purity – you need to also have const input references and pointers to be strongly pure.

In some ways, a language keyword is over-restrictive — a function can still be pure even if it calls impure functions, as long as the side effects don’t escape the outer function.  Entire programs can be considered pure functional units if they only deal with command line parameters instead of random file system state.

Object Oriented Programming

Michael Feathers @mfeathers   OO makes code understandable by encapsulating moving parts. FP makes code understandable by minimizing moving parts.

The “moving parts” are mutating states.  Telling an object to change itself is lesson one in a basic object oriented programming book, and it is deeply ingrained in most programmers, but it is anti-functional behavior.  Clearly there is some value in the basic OOP idea of grouping functions with the data structures they operate on, but if you want to reap the benefits of functional programming in parts of your code, you have to back away from some object oriented behaviors in those areas.

Class methods that can’t be const are not pure by definition, because they mutate some or all of the potentially large set of state in the object.  They are not thread safe, and the ability to incrementally poke and prod objects into unexpected states is indeed a significant source of bugs.

Const object methods can still be technically pure if you don’t count the implicit const this pointer against them, but many object are large enough to constitute a sort of global state all their own, blunting some of the clarity benefits of pure functions.  Constructors can be pure functions, and generally should strive to be – they take arguments and return an object.

At the tactical programming level, you can often work with objects in a more functional manner, but it may require changing the interfaces a bit.  At id we went over a decade with an idVec3 class that had a self-mutating void Normalize() method, but no corresponding idVec3 Normalized() const method.  Many string methods were similarly defined as working on themselves, rather than returning a new copy with the operation performed on it – ToLowerCase(), StripFileExtension(), etc.

Performance Implications

In almost all cases, directly mutating blocks of memory is the speed-of-light optimal case, and avoiding this is spending some performance.  Most of the time this is of only theoretical interest; we trade performance for productivity all the time.

Programming with pure functions will involve more copying of data, and in some cases this clearly makes it the incorrect implementation strategy due to performance considerations.  As an extreme example, you can write a pure DrawTriangle() function that takes a framebuffer as a parameter and returns a completely new framebuffer with the triangle drawn into it as a result.  Don’t do that.

Returning everything by value is the natural functional programming style, but relying on compilers to always perform return value optimization can be hazardous to performance, so passing reference parameter for output of complex data structures is often justifiable, but it has the unfortunate effect of preventing you from declaring the returned value as const to enforce single assignment.

There will be a strong urge in many cases to just update a value in a complex structure passed in rather than making a copy of it and returning the modified version, but doing so throws away the thread safety guarantee and should not be done lightly.  List generation is often a case where it is justified.  The pure functional way to append something to a list is to return a completely new copy of the list with the new element at the end, leaving the original list unchanged.  Actual functional languages are implemented in ways that make this not as disastrous as it sounds, but if you do this with typical C++ containers you will die.

A significant mitigating factor is that performance today means parallel programming, which usually requires more copying and combining than in a single threaded environment even in the optimal performance case, so the penalty is smaller, while the complexity reduction and correctness benefits are correspondingly larger.  When you start thinking about running, say, all the characters in a game world in parallel, it starts sinking in that the object oriented approach of updating objects has some deep difficulties in parallel environments.  Maybe if all of the object just referenced a read only version of the world state, and we copied over the updated version at the end of the frame…  Hey, wait a minute…

Action Items

Survey some non-trivial functions in your codebase and track down every bit of external state they can reach, and all possible modifications they can make.  This makes great documentation to stick in a comment block, even if you don’t do anything with it.  If the function can trigger, say, a screen update through your render system, you can just throw your hands up in the air and declare the set of all effects beyond human understanding.

The next task you undertake, try from the beginning to think about it in terms of the real computation that is going on.  Gather up your input, pass it to a pure function, then take the results and do something with it.

As you are debugging code, make yourself more aware of the part mutating state and hidden parameters play in obscuring what is going on.

Modify some of your utility object code to return new copies instead of self-mutating, and try throwing const in front of practically every non-iterator variable you use.

Additional references:

http://www.haskell.org/haskellwiki/Introduction

http://lisperati.com/

http://www.johndcook.com/blog/tag/functional-programming/

http://www.cs.kent.ac.uk/people/staff/dat/miranda/whyfp90.pdf

http://channel9.msdn.com/Shows/Going+Deep/Lecture-Series-Erik-Meijer-Functional-Programming-Fundamentals-Chapter-1

http://www.cs.utah.edu/~hal/docs/daume02yaht.pdf

http://www.cs.cmu.edu/~crary/819-f09/Backus78.pdf

http://fpcomplete.com/the-downfall-of-imperative-programming/

The Premise

My project team have the distinct honor of listing actor Kevin Bacon in the ‘Special Thanks’ portion of our credits, but I doubt he has any idea that he’s in there. The fact is, he’s had a very large – albeit unknown – influence on the game’s development.

The game is Malevolence: The Sword of Ahkranox, which I have been project lead on for over two years now, and its engine is quite different to most games due to the fact that the entire game world is procedurally generated, infinite and also persistent.

The Challenge

Being infinite AND persistent, the team’s main challenge was to keep the game interesting so long as the player kept playing (not an easy task). We, of course, took the path of procedural item/weapon creation, even going so far as to make the game procedurally generate the graphics for the weapons, to ensure plenty of new gear to find. That, however, can only last for so long, and procedurally generated countryside, dungeons and towns can only entertain a player for so long before they all start to look the same. So we put our heads together and came up with a solution. We all agreed that would couldn’t keep the players interested infinitely, but we can take steps to ensure they get maximum enjoyment and re-playability out of the game while they do play it.

The Solution

What we came up with in the end was the quest and dialog system, and the way they interacted. Both were to be procedurally generated and intricate enough to ensure long-time interest from the player. But how does Kevin Bacon get involved with this? The answer lies in his namesake header file:

Some readers will be familiar with the Six Degrees of Kevin Bacon game back in 1994. The basic concept was that you could take the name of anyone involved in the Hollywood film industry, whether they be an A-List actor or an isolated gaffer somewhere on an obscure film, and within no more than 6 steps, be able to link them to Kevin Bacon. For example, one of the voice cast in my game Malevolence, Karen Kahler, was in the short film “The Magician” with actress Jackie Zane, who was in the film “Burning Palms” with actor Nick Stahl who acted in the movie “My One and Only” with Kevin bacon. Thus, her “Bacon Index” is 3 (and through Karen, mine is 4!). Make sense?

Well, it’s that system (or a modification of it) which the NPCs in Malevolence work with.

So How Does it Do it?

Let’s say that the player enters a town and talks to an NPC. The game determines that this NPC will have a quest for them, and so the game spreads its feelers out and works out what is relatively close to the town, and how far away each location is. Since the game world is generated procedurally, it does this process dynamically:

First the game procedurally generates the quest. The engine first selects what type of quest to generate and settles on an item centric quest. It then generates an item, and an incident and comes up with a backpack which was lost. Once this is done, the NPC tells the player that they need help recovering their backpack from a dungeon that they were exploring. Only the catch is, they fled the dungeon so quickly that they don’t remember where it was.

While this is happening, the engine consults the memory map shown above and looks at the area around the town for a few kilometers, then chooses a dungeon that is close enough to not be too far away, and far enough away that the player will have to search for it. However, once it has found a dungeon, it doesn’t let the player know where it is like most games. It is now the player’s mission to search for it.

Finding the Unfindable

Now that you know that there is a dungeon out there somewhere with a backpack in it, you can ask around to get more info. The engine, however, is processing the dungeon’s “Kevin Bacon Index” in the background. The main difference, however, is that not everyone knows everyone else perfectly well, so if one person gives you information, they may not be 100% sure about the information. So when you get a map of people like this:

You can see the percentages between them all. That shows the familiarity of the characters between them. So, if you speak to NPC ‘A’ and ask them about nearby dungeons, they will tell you that they have no idea about that sort of thing, but their friends Steve and Kyle might. If you speak to Steve, he may refer you to Kyle or pass you on to Bob the Blacksmith, who he’s fairly sure knows a mapmaker and a woodsman, who would probably have a better idea about dungeons in the area. When you speak to Bob the Blacksmith, he’ll tell you about Keith the Woodsman, who is familiar with the local wilderness, but will more likely put you on to Kevin the Mapmaker, who knows Keith the Woodsman quite well and may be able to help you himself (with his maps). These NPCs may all be in the same town, or they may be spread between multiple towns. It’s all generated by the procedural engine, but it’s how the player FOLLOWS the path that defines how well the quest will work out.

If a player is clever and good at deduction, they may have an easy time of it – for example, if they followed the path ABDEF, while it may not be the most direct route to Keith, they will get some accurate maps out of it, and maybe even a new weapon to help clear out the dungeon to find the backpack. But if they don’t follow good advice, they may have a far less fortuitous way. And keep in mind that due to the lack of familiarity between certain NPCs, sometimes the player will get false information in their searching, which can slow them down quite a bit.

That’s Very Long Winded

Very true, but not all quests in the game will work like this. The quests are divided into two quest types. There are the multi-tiered quests, as mentioned above, and “b*tch quests” which are your standard “there’s a dungeon, clean it out” or “there are rats in my basement, kill them”. On top of that, even when a player is assigned a multi-tiered quest, sometimes they will have 3 steps to complete, sometimes they will have 20, it all depends on how the cookie crumbles in the procedural generation.

Conclusion

Making a game which is infinite AND persistent has provided countless challenges to us as a dev team, but the solutions to intricate problems are often the most unique. I hope you enjoyed reading about this little section of our game, and if you ever end up playing it and you see Kevin Bacon in the credits, you’ll now know why.

So this is a small, but important problem. We want to somehow get the velocities of the animated objects and then apply them to the bodies in the ragdoll. The only snag is that animated objects typically don’t know anything about velocities. Also, we need some way of matching up the physics bodies with the animated objects.

First, some background information. In the Bitsquid engine, physics, scene graph and animation are completely separate systems. We strongly believe in minimizing the couplings between different systems since that makes the engine easier to understand, reason about, modify, optimize and rewrite.

• The physics system simulates a number of bodies, possibly connected by joints.

• The scene graph handles local-to-world transforms for a collection of nodes in a hierarchy.

• The animation system evaluates and blends animation curves for bones.

Bones and bodies hold references (just integer indices, really) to nodes in the scene graph and this how the systems communicate. After the animation has been evaluated, the resulting local transforms are written to the bones’ nodes in the scene graph.

For keyframed physics (animated hit bodies), the animation drives the physics, which means the physics’ bodies will read their world transforms from the corresponding nodes in the scene graph. For ragdolled physics, the world transforms of the bodies are written to the scene graph after the simulation has completed.

For partial ragdolls (such as a non-functioning, but still attached limb) or powered ragdolls (ragdolls driven by motors to achieve animation poses) it gets a little more involved (perhaps a topic for a future post), but the basic setup is the same.

Given this setup there are two ways of calculating the animation velocities:

• We can calculate the velocities directly by differentiating the animation curves.

• We can record a node’s transform at two different time steps and compute the velocity from the difference.

The first approach is doable, but not very practical. Not only do we have to differentiate all the animation curves, we must also take into account how those velocities are affected by the blend tree and local-to-world transforms. And even if we do all that, we still don’t account for movements from other sources than animation, such as scripted movements, IK or interactions with the character controller.

The second option is the more reasonable one. Now all we need is a way of obtaining the transforms from two different time steps. There are a number of possible options:

• We could add an array of Matrix4x4:s to our scene graph’s last_world where we store the last world transform of every object. So whenever we want to go to ragdoll we always have a last_world transform to calculate velocities from.

• We could simulate the character backwards in time when we want to go to ragdoll and obtain a last_world transform that way.

• We could delay the transition to ragdoll one frame, so that we have enough time to gather two world transforms for computing the velocity.

The first approach is conceptually simple, but costly. We are increasing the size of all our scene graphs by about 50 % (previously they contained local and world transforms, now they will also need last_world). In addition we must memcpy(last_world, world) before we compute new world transforms. That’s a significant cost to pay all the time for something that happens very seldom (transition to ragdoll).

The second appraoch sounds a bit crazy, but some games actually already have this functionality. Servers in competetive multi-player fps games often need to rewind players in time in order to accurately determine if they were able to hit each other. Still, I find the approach to be a bit too complicated and invovled just to get a velocity.

The third aproach seems simple and cheap, but it violates one of our Bitsquid principles: Thou Shalt Not Have Any Frame Delays. Delaying something a frame can be a quick fix to many hairy problems, but it puts your game in a very weird transitional state where it at the same time both is and isn’t (yet) something. The character isn’t really a ragdoll yet, but it will be the next frame, whether I want to or not.

This new slightly self-contradictory state invites a host of bugs and before you know it, little logic pieces will start to seep into the code base “do this unless you are in the special transition-to-ragdoll state”. Congratulations, you have just made your codebase a lot more complicated and bug prone.

If this is not enough, consider the poor sucker who just wants to write a routine that does A, B, C and D, when A, B and C requires frame delays. Suddenly what was supposed to be simple function got turned into a state machine that needs to run for four frames to produce it result.

The simple rule that actions should take place immediately protects against such insanity.

So three options, none of them especially palpable.

I actually went with the first one, to always compute and store last_world in the scene graph, but with a flag so that this is only used for units that actually need it (characters that can go to ragdoll). When there is no clear winner, I always pick the simplest solution, because it is a lot easier to optimize later if the need should arise. (We could for example track last_world only for the nodes which have a corresponding ragdoll actor. Also we could store last_world as (p,q) instead of as a matrix.)

For completion, given the two transforms, the code for compting the velocities will look something like this:

Vector3 p0 = translation(tm_0);
Vector3 p1 = translation(tm_1);
Vector3 velocity = (p1 - p0) / dt
 
Quaternion q0 = rotation(tm_0);
Quaternion q1 = rotation(tm_1);
Quaternion q = q1 * inverse(q0);
AxisAngle aa = q.decompose();
Vector3 angular_velocity = aa.axis * aa.angle / dt;

This has also been posted to The Bitsquid Blog.

The IEEE standard for floating-point math defines five exceptions that shall be signaled when certain conditions are detected. Normally the flags for these exceptions are raised (set), a default result is delivered, and execution continues. This default behavior is often desirable, especially in a shipping game, but during development it can be useful to halt when an exception is signaled.

Halting on exceptions can be like adding an assert to every floating-point operation in your program, and can therefore be a great way to improve code reliability, and find mysterious behavior at its root cause.

This article is part of a series on floating-point. The complete list of articles in the series is:

• 1: Tricks With the Floating-Point Format – an overview of the float format
• 2: Stupid Float Tricks – incrementing the integer representation of floats
• 3: Don’t Store That in a Float – a cautionary tale about time
• 3b: They sure look equal… – special bonus post (not on altdevblogaday)
• 4: Comparing Floating Point Numbers, 2012 Edition – tricky but important
• 5: Float Precision—From Zero to 100+ Digits – what does precision mean, really?
• 5b: C++ 11 std::async for Fast Float Format Finding – special bonus post (not on altdevblogaday)
• 6: Intermediate Precision – their effect on performance and results
• 7.0000001: Floating-Point Complexities – a lightning tour of all that is weird about floating point
• 8: Exception Floating point – (return *this;)

Let’s get it started again

The five exceptions mandated by the IEEE floating-point standard are:

1. Invalid operation: this is signaled if there is no usefully definable result, such as zero divided by zero, infinity minus infinity, or sqrt(-1). The default result is a NaN (Not a Number)
2. Division by zero: this is signaled when dividing a non-zero number by zero. The result is a correctly signed infinity.
3. Overflow: this is signaled when the rounded result won’t fit. The default result is a correctly signed infinity.
4. Underflow: this is signaled when the result is non-zero and between -FLT_MIN and FLT_MIN. The default result is the rounded result.
5. Inexact: this is signaled any time the result of an operation is not exact. The default result is the rounded result.

The underflow exception is usually not of interest to game developers – it happens rarely, and usually doesn’t detect anything of interest. The inexact result is also usually not of interest to game developers – it happens frequently (although not always, and it can be useful to understand what operations are exact) and usually doesn’t detect anything of interest.

That leaves invalid operation, division by zero, and overflow. In the context of game development these are usually truly exceptional. They are rarely done intentionally, so they usually indicate a bug. In many cases these bugs are benign, but occasionally these bugs indicate real problems. From now one I’ll refer to these first three exceptions as being the ‘bad’ exceptions and assume that game developers would like to avoid them, if only so that the exceptions can be enabled without causing crashes during normal game play.

When can divide by zero be useful?

While the ‘bad’ exceptions typically represent invalid operations in the context of games, this is not necessarily true in all contexts. The default result (infinity) of division by zero can allow a calculation to continue and produce a valid result, and the default result (NaN) of invalid operation can sometimes allow a fast algorithm to be used and, if a NaN result is produced, a slower and more robust algorithm to be used instead.

The classic example of the value of the division by zero behavior is calculation of parallel resistance. The formula for this for two resistors with resistance R1 and R2 is:

Because division by zero gives a result of infinity, and because infinity plus another number gives infinity, and because a finite number divided by infinity gives zero, this calculation calculates the correct parallel resistance of zero when either R1 or R2 is zero. Without this behavior the code would need to check for both R1 and R2 being zero and handle that case specially.

In addition, this calculation will give a result of zero if R1 or R2 are very small – smaller than the reciprocal of FLT_MAX or DBL_MAX. This zero result is not technically correct. If a programmer needs to distinguish between these scenarios then monitoring of the overflow and division by zero flags will be needed.

Resistance is futile

Assuming that we are not trying to make use of the divide-by-zero behavior we need a convenient way of turning on the ‘bad’ floating-point exceptions. And, since we have to coexist with other code (calling out to physics libraries, D3D, and other code that may not be ‘exception clean’) we also need a way of temporarily turning off all floating-point exceptions.

The appropriate way to do this is with a pair of classes whose constructors and destructors do the necessary magic. Here are some classes that do that, for VC++:

// Declare an object of this type in a scope in order to suppress
// all floating-point exceptions temporarily. The old exception
// state will be reset at the end.
class FPExceptionDisabler
{
public:
    FPExceptionDisabler()
    {
        // Retrieve the current state of the exception flags. This
        // must be done before changing them. _MCW_EM is a bit
        // mask representing all available exception masks.
        _controlfp_s(&mOldValues, _MCW_EM, _MCW_EM);
        // Set all of the exception flags, which suppresses FP
        // exceptions on the x87 and SSE units.
        _controlfp_s(0, _MCW_EM, _MCW_EM);
    }
    ~FPExceptionDisabler()
    {
        // Clear any pending FP exceptions. This must be done
        // prior to enabling FP exceptions since otherwise there
        // may be a 'deferred crash' as soon the exceptions are
        // enabled.
        _clearfp();
 
        // Reset (possibly enabling) the exception status.
        _controlfp_s(0, mOldValues, _MCW_EM);
    }
 
private:
    unsigned int mOldValues;
 
    // Make the copy constructor and assignment operator private
    // and unimplemented to prohibit copying.
    FPExceptionDisabler(const FPExceptionDisabler&);
    FPExceptionDisabler& operator=(const FPExceptionDisabler&);
};
 
// Declare an object of this type in a scope in order to enable a
// specified set of floating-point exceptions temporarily. The old
// exception state will be reset at the end.
// This class can be nested.
class FPExceptionEnabler
{
public:
    // Overflow, divide-by-zero, and invalid-operation are the FP
    // exceptions most frequently associated with bugs.
    FPExceptionEnabler(unsigned int enableBits = _EM_OVERFLOW |
                _EM_ZERODIVIDE | _EM_INVALID)
    {
        // Retrieve the current state of the exception flags. This
        // must be done before changing them. _MCW_EM is a bit
        // mask representing all available exception masks.
        _controlfp_s(&mOldValues, _MCW_EM, _MCW_EM);
 
        // Make sure no non-exception flags have been specified,
        // to avoid accidental changing of rounding modes, etc.
        enableBits &= _MCW_EM;
 
        // Clear any pending FP exceptions. This must be done
        // prior to enabling FP exceptions since otherwise there
        // may be a 'deferred crash' as soon the exceptions are
        // enabled.
        _clearfp();
 
        // Zero out the specified bits, leaving other bits alone.
        _controlfp_s(0, ~enableBits, enableBits);
    }
    ~FPExceptionEnabler()
    {
        // Reset the exception state.
        _controlfp_s(0, mOldValues, _MCW_EM);
    }
 
private:
    unsigned int mOldValues;
 
    // Make the copy constructor and assignment operator private
    // and unimplemented to prohibit copying.
    FPExceptionEnabler(const FPExceptionEnabler&);
    FPExceptionEnabler& operator=(const FPExceptionEnabler&);
};

The comments explain a lot of the details, but I’ll mention a few here as well.

_controlfp_s is the secure version of the portable version of the old _control87 function. _controlfp_s controls exception settings for both the x87 and SSE FPUs. It can also be used to control rounding directions on both FPUs, and on the x87 FPU it can be used to control the precision settings. These classes use the mask parameter to ensure that only the exception settings are altered.

The floating-point exception flags are sticky, so when an exception flag is raised it will stay set until explicitly cleared. This means that if you choose not to enable floating-point exceptions you can still detect whether any have happened. And – not so obviously – if the exception associated with a flag is enabled after the flag is raised then an exception will be triggered on the next FPU instruction, even if that is several weeks after the operation that raised the flag. Therefore it is critical that the exception flags be cleared each time before exceptions are enabled.

Typical usage

The floating-point exception flags are part of the processor state which means that they are per-thread settings. Therefore, if you want exceptions enabled everywhere you need to do it in each thread, typically in main/WinMain and in your thread start function, by dropping an FPExceptionEnabler object in the top of these functions.

When calling out to D3D or any code that may use floating-point in a way that triggers these exceptions you need to drop in an FPExceptionDisabler object.

Alternately, if most your code is not FP exception clean then it may make more sense to leave FP exceptions disabled most of the time and then enable them in particular areas, such as particle systems.

Because there is some cost associated with changing the exception state (the FPU pipelines will be flushed at the very least) and because making your code more crashy is probably not what you want for your shipping game you should put #ifdefs in the constructors and destructors so that these objects become NOPs in your retail builds.

There have been various instances in the past (printer drivers from a manufacturer who shall not be named) that would enable floating-point exceptions and leave them enabled, meaning that some perfectly legitimate software would start crashing after calling into third-party code (such as after printing). Having somebody’s hapless code crash after calling a function in your code is a horrible experience, so be particularly careful if your code may end up injected into other processes. In that situation you definitely need to not leave floating-point exceptions enabled when you return, and you may need to be tolerant of being called with floating-point exceptions enabled.

Performance implications of exceptions

Raising the exception flags (triggering a floating-point exception) should have no performance implications. These flags are raised frequently enough that any CPU designer will make sure that doing so is free. For example, the inexact flag is raised on virtually every floating-point instruction.

However having exceptions enabled can be expensive. Delivering precise exceptions on super-scalar CPUs can be challenging and some CPUs choose to implement this by disabling FPU parallelism when floating-point exceptions are enabled. This hurts performance. The PowerPC CPU used in the Xbox 360 CPU (and presumably the one used in the PS3) slows down significantly when any floating-point exceptions are enabled. This means that when using this technique on these processors you should just enable FPU exceptions on an as-needed basis.

Sample code

The sample code below calls TryDivByZero() three times – once in the default environment, once with the three ‘bad’ floating-point exceptions enabled, and once with them suppressed again. TryDivByZero does a floating-point divide-by-zero inside a Win32 __try/__except block in order to catch exceptions, print a message, and allow the tests to continue. This type of structured exception handling block should not (repeat not) be used in production code, except possibly to record crashes and then exit. I hesitate to demonstrate this technique because I fear it will be misused. Continuing after unexpected structured exceptions is pure evil.

With that said, here is the code:

int __cdecl DescribeException(PEXCEPTION_POINTERS pData, const char *pFunction)
{
    // Clear the exception or else every FP instruction will
    // trigger it again.
    _clearfp();
 
    DWORD exceptionCode = pData->ExceptionRecord->ExceptionCode;
    const char* pDescription = NULL;
    switch (exceptionCode)
    {
    case STATUS_FLOAT_INVALID_OPERATION:
        pDescription = "float invalid operation";
        break;
    case STATUS_FLOAT_DIVIDE_BY_ZERO:
        pDescription = "float divide by zero";
        break;
    case STATUS_FLOAT_OVERFLOW:
        pDescription = "float overflow";
        break;
    case STATUS_FLOAT_UNDERFLOW:
        pDescription = "float underflow";
        break;
    case STATUS_FLOAT_INEXACT_RESULT:
        pDescription = "float inexact result";
        break;
    case STATUS_FLOAT_MULTIPLE_TRAPS:
        // This seems to occur with SSE code.
        pDescription = "float multiple traps";
        break;
    default:
        pDescription = "unknown exception";
        break;
    }
 
    void* pErrorOffset = 0;
#if defined(_M_IX86)
    void* pIP = (void*)pData->ContextRecord->Eip;
    pErrorOffset = (void*)pData->ContextRecord->FloatSave.ErrorOffset;
#elif defined(_M_X64)
    void* pIP = (void*)pData->ContextRecord->Rip;
#else
    #error Unknown processor
#endif
 
    printf("Crash with exception %x (%s) in %s at %p!\n",
            exceptionCode, pDescription, pFunction, pIP);
 
    if (pErrorOffset)
    {
        // Float exceptions may be reported in a delayed manner -- report the
        // actual instruction as well.
        printf("Faulting instruction may actually be at %p.\n", pErrorOffset);
    }
 
    // Return this value to execute the __except block and continue as if
    // all was fine, which is a terrible idea in shipping code.
    return EXCEPTION_EXECUTE_HANDLER;
    // Return this value to let the normal exception handling process
    // continue after printing diagnostics/saving crash dumps/etc.
    //return EXCEPTION_CONTINUE_SEARCH;
}
 
static float g_zero = 0;
 
void TryDivByZero()
{
    __try
    {
        float inf = 1.0f / g_zero;
        printf("No crash encountered, we successfully calculated %f.\n", inf);
    }
    __except(DescribeException(GetExceptionInformation(), __FUNCTION__))
    {
        // Do nothing here - DescribeException() has already done
        // everything that is needed.
    }
}
 
int main(int argc, char* argv[])
{
#if _M_IX86_FP == 0
    const char* pArch = "with the default FPU architecture";
#elif _M_IX86_FP == 1
    const char* pArch = "/arch:sse";
#elif _M_IX86_FP == 2
    const char* pArch = "/arch:sse2";
#else
#error Unknown FP architecture
#endif
    printf("Code is compiled for %d bits, %s.\n", sizeof(void*) * 8, pArch);
 
    // Do an initial divide-by-zero.
    // In the registers window if display of Floating Point
    // is enabled then the STAT register will have 4 ORed
    // into it, and the floating-point section's EIP register
    // will be set to the address of the instruction after
    // the fdiv.
    printf("\nDo a divide-by-zero in the default mode.\n");
    TryDivByZero();
    {
        // Now enable the default set of exceptions. If the
        // enabler object doesn't call _clearfp() then we
        // will crash at this point.
        FPExceptionEnabler enabled;
        printf("\nDo a divide-by-zero with FP exceptions enabled.\n");
        TryDivByZero();
        {
            // Now let's disable exceptions and do another
            // divide-by-zero.
            FPExceptionDisabler disabled;
            printf("\nDo a divide-by-zero with FP exceptions disabled.\n");
            TryDivByZero();
        }
    }
 
    return 0;
}

Typical output is:

When generating SSE code I sometimes see STATUS_FLOAT_MULTIPLE_TRAPS instead of STATUS_FLOAT_DIVIDE_BY_ZERO. This is slightly less helpful, but the root cause should be straightforward to determine.

That said, determining the root cause can be slightly tricky. On the x87 FPU, floating-point exception reporting is delayed. Your program won’t actually crash until the next floating-point instruction after the the problematic one. In the example below the fdiv does the divide by zero, but the crash doesn’t happen until the fstp after.

011A10DD fdiv        dword ptr [__fmode+4 (11A3374h)] 
011A10E3 fstp        dword ptr [ebp-1Ch] 

Normally it is easy enough to look back one instruction to find the culprit, but sometimes the gap can be long enough to cause confusion. Luckily the CPU records the address of the actual faulting instruction and this can be retrieved from the exception record. This value is printed out when applicable in my exception handler, or you can see it in the Visual Studio registers window.

The sample code can be downloaded as a VisualC++ 2010 project (32-bit and 64-bit) from here:

ftp://ftp.cygnus-software.com/pub/FloatExceptions.zip

Handle and continue

If you want to get really crazy/sophisticated then it is possible to catch a floating-point exception with __try/__except, handle it in some domain specific way (handling overflow by scaling down the result and recording that you did that) and then resume. This is sufficiently esoteric that I have no more to say about it – consult the documentation for _fpieee_flt if this sounds interesting.

SIMD

SSE and its SIMD instructions throw a few wrinkles into the mix. One thing to be aware of is that instructions like reciprocal estimate (rcpps) never trigger divide-by-zero exceptions – they just silently generate infinity. Therefore they are a way that infinity can be generated even when the ‘bad’ exceptions are enabled.

Additionally, many common patterns for SIMD instructions only use some components of the four-wide registers. This could be because the code is operating on a three-float vector, or it could be because the code is operating on an array of floats that is not a multiple of four long. Either way, the ‘unused’ component or components in the registers may end up triggering floating-point exceptions. These exceptions are false-positives (they don’t indicate a bug), but they must be dealt with in order to allow floating-point exceptions to be enabled. The best way to deal with this is to ensure that the unused components are filled with valid data, at least in the development builds where floating-point exceptions are enabled. Filling them with one or zero is generally good enough.

Filling the unused components with valid values may also improve performance. Some CPUs drop to microcode when they encounter some ‘special’ numbers (NaNs, infinities, and/or denormals) and using well behaved values avoids that risk.

Practical experience

On some projects I have been able to enable these three floating-point exceptions, fix all of the accidental-but-unimportant exceptions, and then find a few crucial bugs hidden in the weeds. On these projects, enabling floating-point exceptions during development was crucial. On other projects – big messy projects with a lot of history and large teams – I was unable to get the team to buy off on the concept, so it ultimately didn’t work.

Your mileage may vary, but as with asserts of any type, enabling them early, and ensuring that violations get fixed promptly, is the trick to getting value from floating-point exceptions. Adding them to a large existing codebase is trickier, but can be dealt with by only enabling them in particular parts of the code where their value exceeds their cost.

Practical experience, hot off the presses

I’ve been trying to improve the usability of debug builds on my current project and one persistent problem was a NaN that would show up in the particle system early on, triggering many different asserts. I couldn’t tell where this NaN was being generated so I knew I had to enable floating-point exceptions, using the classes described above. This project was not designed to have floating-point exceptions enabled so there were several challenges. The process was:

• Enable floating-point exceptions in three key functions that called out to all of the particle system code
• Disable floating-point exceptions in one child function that did floating-point overflow by design
• Pad one of our arrays of particle system data with valid data to the next multiple of four so that the unused SIMD lanes wouldn’t trigger spurious exceptions
• Find and fix five bugs that were causing floating-point exceptions

It worked. All of the bugs were worth fixing, and one of them was the source of the NaNs. After most of a day of investigation the crucial fix was to change one letter – from ‘e’ to ‘t’ – and this was enough to prevent us from dividing zero by zero. Now our debug builds are significantly more usable, and a genuine bug that was causing (apparently unnoticed) glitches is gone.

Homework

The summary is that while floating-point exceptions, even the ‘bad’ ones, aren’t necessarily bad, you can often find bugs in your code by treating them as errors. By using the classes shown above, with appropriate #ifdefs so that they go away in retail builds, you can enable floating-point exceptions in most parts of your code, and thereby improve reliability and avoid unexpected behavior.

But please, don’t use the __try/__except block, in debug or in retail code. It is an ugly and dangerous hack that should only be used in specialized demonstration code.

Game Design

Game design is always about looking back before looking forward. Sometimes this is done consciously, other times it is done unconsciously, but it always happens. Every great new idea is built on improving an one or more old ideas, and the best game designers are well aware of this. One of my biggest pet peeves is when someone says to me “don’t reinvent the wheel”, which they often quickly regret saying as I begin to lecture them on how if no-one ever reinvented the wheel we would never have tyres, suspension, alignments, treading, etc. All things which have made the wheel more efficient, smoother and just generally better.

A great design for a reinvented wheel

But for a wheel to be reinvented, one must start with a wheel to begin with, and that is what we are always doing in the games industry. A good example is the recently successful iOS game, Jetpack Joyride, from HalfBrick studios. An excellent game, and quite obviously modelled off of those Amiga/Commodore-era “don’t touch the sides of the tunnel with your helicopter” games. They took that solid idea, and evolved it with pickups, achievements, special gear you can earn, etc, and made a great game out of it. They reinvented the wheel and it worked.

An example screen from Jetpack Joyride

Does it Always Work?

No, it doesn’t always work. Some past ideas weren’t that great, let’s be honest. But some have not only been ridiculously successful, but have also done the unthinkable and withstood the test of time. Two good examples in relatively different genres are Quake 3: Arena (1999) and Diablo II (2000). There are many more, of course, but these are my case studies for this particular article.

Well over a decade on and these two games are still being played with distinct regularity, are favourites at private LAN parties and have graphics which still stack up fairly well against their more modern competitors. Why is this and what can modern game developers do to make a game that withstands the test of time so well? The answer lies in how the button is pressed.

…The Button?

I mean this metaphorically, of course. But let me explain with a story. In my day job, I am quite involved in the gambling industry, and as such am exposed to many new products before they hit the market. One such product came about which was a “re-launch” (as it were) of the old-style “one-armed bandit” poker machines of yesteryear, which fell out of use in Australia decades ago.

What the pokies used to look like

In an attempt to latch onto the “retro rush” that seems to be going on… Well… Everywhere, people thought that re-releasing a classic-style one-armed bandit machine would inspire people’s nostalgia and that they would be a viable new product. I was quite excited when I heard the news as I remember my grandfather having one under his house back in the 80s and I loved the feel of the flywheel revving up as I pulled down the lever. I loved seeing it kick the spinners into gear and watching all of the colourful fruits lock into place on the wheeled display.

Needless to say, when the first units came into the test room, I was excited to feel that little piece of nostalgia again. The machines certainly looked the same, if not a bit more modern. The digital displays had even been replaced by the classic wheels with the printed artwork on them. It was in test mode and ready to play, so I grabbed the lever and pulled. The wheels started spinning and the lights started flashing, but it was all wrong. It felt so wrong that I thought perhaps the machine was broken… It wasn’t, of course, it worked perfectly well. But the machine was no longer clockwork, obviously, and the lever arm was simply a spring-loaded arm which pressed a small button when it was fully depressed. There was no sense of “winding” the machine up. No sense of having some sort of influence over the device and no sense of being connected to its operation in any way. It was a jarring sensation of being completely disconnected. What was once – for lack of a better term – an intimate interaction experience had now become somewhat cold, clinical and disconnected.

I voiced my concerns at this, and was – of course – laughed off. I even went so far as to suggest they could build a small flywheel mechanism into the arm that has no functional purpose other than generating that “feel” when you pull the lever. They ignored my counsel and released the product anyway. Needless to say, it was very unpopular and never really took off. The official reasoning was that “people don’t like the ‘retro thing’ in gambling”.

How is it Relevant to Video Games?

How does this relate to video games such as Quake 3 Arena and Diablo 2? Well, think of that “button” as being the interaction between the player and the game. It has to feel good, just like the poker machine arm. There are things you can do to the button to make it feel better, and be more intimate, or there are things you can do to the button to make the user feel disconnected from the machine, and they both come down to a deep level of design.

When beginning a game project, try to describe the game in a short, concise sentence. Once you’ve got that sentence, every single thing you add to the game which complies with that sentence will make the game “cleaner” to play, but the more you stray from that sentence, the more disjointed it will seem.

Quake 3 Arena:

Take Quake 3 Arena as the first example. It can easily be described as “Fast-paced, sci-fi, multiplayer deathmatching” and nothing else. The reason that game was so popular for so long, was twofold. Firstly, it was built on a very beautifully made engine made by John Carmack, which was powerful enough to drive MANY other games for the next decade (RTCW, Jedi Academy, Call of Duty, etc) and secondly, it followed a clean design based on a simple concept which could be summed up in that one sentence. Every aspect of the game conformed to it. There was nothing confusing, or particularly intricate about it. When you played it, you were immersed in the game. No part of it “fought” against you or detached you from the experience. You had four movement buttons, a jump button and a shoot button. A control system which could be replicated on a classic GameBoy. The player movement was swift, smooth, consistent. The game never slowed down, or became badly paced at any point, yet still gave the user enough freedom to develop their own style of play and put it into the game.

Conversely, a later id Software game, Doom III, was not quite as smooth. The gameplay was slower and more disjointed. It was still a great game, yes, but things such as having to stop to punch numbers into a keypad or having to put down your weapon to pull out a torch disconnected the player from an otherwise immersive, cathartic experience.

Diablo II:

Diablo II is another fine example of this. Fans of the series will know that it has an incredible complex control system consisting of one button… Oh, yes, you can use the number buttons on the keyboard as well if you like, but there is essentially the left-click button and a couple of others which you use occasionally. The entire game is about as non-confrontational as you can get, and anyone can pick it up. It is also paced beautifully, with the first area being almost impossible to die in, yet still giving you enough of a sense of risk to keep you engaged and on your toes.

A typical scene from Diablo II

It had very simple multiplayer, a beautifully crafted procedural item and level system and enough replayability to ensure people would be able to play it for years, and indeed they did. Blizzard, in true Blizzard form, found all of the features of Diablo I that left the player feeling disconnected and removed or remodelled them, making a game which is clean, simple, elegant and extremely replayable. They followed a simple design sentence of “Explore, kill and loot while progressing character” and every aspect of the game reflects that. Nothing gets in your way, nothing slows the progress. There are no repetitive cut scenes or cloned combat encounters or scripted events, all of which can make a procedural game lag something awful. This “neatness” has ensured the game’s continuing success both in the single player and multiplayer worlds and place on game store shelves over a decade after it first was released.

So What are you Saying?

When designing a game, before doing anything else, describe what you want it to be in one concise sentence. After that, with every decision you make or feature you go to add, check to see if it fits neatly with that short description. If it strays too far out of that, or the description ends up being too long, you will end up with a convoluted game and the player will feel disconnected from it. Remember that games are a form of escapism, and the player should forget that there is even a keyboard or other controller between them and the game. The graphics help, yes. The story helps, yes. But it’s the FEEL of the game that will ensure a captive audience. It’s just like the one-armed bandit. The aim of the poker machine is to pull the lever and get the wheels spinning, but if you just focus on making the lever spin the wheels, people won’t play it. They need to enjoy the act of pulling the lever and pushing the button just as much as the result of that action on the screen.

Anyway, thanks for listening to my little rant :) I hope it helps to inspire all of you other game developers.

First though, some reader interaction… dig out the schedule for your current project and look through it. Is there a task anywhere on it titled “localization”? If it is on the schedule I would put good money on it being close to the end of the project usually within five tasks of the Alpha build.

If you have a task titled “localization setup” in the first half of your projects development cycle then you or someone on your team has probably already been through the pain of localization and is already preparing for it.

Location, Location, Location

What most people focus on primarily with localization is usually the language and although it is important it is only half the equation. The “Location” is the other half and this is where things get subtle and you need to become a little culturally aware.

Let’s consider this post, you might initially be unaware that I have localized it for English American. I am from England and I am British which means that I learnt to spell color as colour, localization as localisation and I pronounce ‘Z’ as “Zed” rather than “Zee”. These differences come about not because of the language but because of the differences in culture.

Keeping localization in mind during development means that when issues arise they can be fixed quickly rather than waiting until the end of the project and trying to fix it with a hack.

I do not profess to be an expert on these various countries and cultures, what follows are issues that I have experienced during the localization process. By sharing them I am hoping that you will firstly have a clearer idea of what to watch out for but also I would encourage you to share your own experiences in the comments section below.

Hand Gestures

Animators use hand gestures to breathe life into the characters, unfortunately there are quite a few hand gestures that can cause problems when dealing with localization. The “Okay” gesture, the “Thumbs up” gesture and the extended hand with palm outward are just three gestures that I have seen removed from various games due to how different cultures react to them.

You could go down the route of generating a different animation tree localized for each country but that does increase the amount of data to be maintained and also the likely hood of a bug being introduced in a specific country.

A quick internet search can produce some good descriptions of various gestures that can be problematic, this one even showed gestures I was unaware of before writing this post: Telegraph.

In game visuals

Imagine that in your game there is a wall upon which scribbled in a marker pen is a message that is important for putting the player on the right track. Now image that Portuguese is your language, unless that texture has been translated into a different language you have just made it extremely difficult for those players to progress.

Apart from doing a texture swap based upon the selected language you might also consider adding a language trigger where if the player gets close to the wall it will pop up a translation.

The Language of Color

Color plays a huge part in our emotional and physiological state even though for the most part we are completely unaware of it most of the time, what is really interesting is that the specific colors actually change from one culture to another. For example in Western countries black is used to represent mourning however in South America the color to use would more appropriately be purple.

I am mentioning this here more for completeness since I can only remember only ever once changing a game color for localization. The specific issue was relating to the color that a map of Germany had been painted.

This is quite a useful and interesting chart:  informationisbeautiful.

Violence

Germany has very strict laws about violence especially where humans are concerned, as a consequence there have been a few games where the red blood has been changed to (robot) green. Or in the case of one of my projects the characters were to be tortured for information, in the German version we simply covered the animations up hiding the offending visuals from the players in Germany.

Adult Content

Worth remembering that not every country believe that games are for adults, one such example is Australia which at present does not allow a rating of 18+ for games.

Have a look of this list on wikipedia of banned games in different countries, useful to know why a game was banned since it isn’t always about violence and adult content.

Audio

Another facet of the localization process is obviously to make sure the correct spoken speech / dialog is triggered in game in the correct language. It is easy to look through an excel document and see when translation text is missing however it is a lot harder to check the audio which is why I try to encourage artists / engineers / designers on different platforms to spend time working in the different language and locations as early in a projects development as possible.
Hopefully you have finished reading this and there wasn’t anything new that you didn’t know already, if anything here was surprising and new then you probably should kick localization up your schedules to do list.

Why is the #ME3 ending like Javascript in the browser?

With @ArchAzrael and @mike_acton
Check out what Double Cluepon is doing. I’m a big fan of their open development model.

Game tools

With @gorlak, @fuzzybinary and @mike_acton
Make sure to join up with the IGDA Tools SIG (#toolsmiths) if you’re interested in a community of tools developers. And check out The Toolsmiths blog

(Mini warning: this post turned from a simple idea to a fairly epic 2500 words.  Get a cuppa and a chocolate biscuit before attempting to read.)

To answer that, we need to look at what the current constrains actually are. There are lots of small issues but they nearly all fall into one of two categories: cost or artistic desire.

Cost

It’s a sad fact but the cost of making AAA games has sky rocketed. On average, on the original PlayStation, a team of about 10 people could make a 60$ game in about a year. For the PS2 that went up to more like 30 – 50 people for two years. For this generation of console there is no average – it’s as many people as you can get for as long as you can get. 100 people taking 3 years is fairly normal, but it’s also common for teams made up of many more. If the price of games reflected the increased man-year development time, they’d now cost about 1800$ each.

Why?

As a very broad stroke: all those pretty images take a lot of developing. The quality of the art in games is far higher than ever before, and the code bases that drive everything are far larger and complex than we ever imagined they’d be.

The biggest single factor slowing us down, and thus costing us money, is our tools and pipelines. Turnaround for an idea has slowed to an almost archaic crawl because of the time it takes to create content in a AAA game right now. The knock on of this is that it has a detrimental effect on the quality of the games we are making, and people don’t want to pay 60$ a pop anymore, let alone 1800$. There are two ways of fixing this: as an industry we need to invest in finding a different approach to creating content or we need to develop better tools.  Or, preferably, both.

Different approaches to creating content

Once upon a time, coders creating everything in the game.  Art was a means to an end.  Nowadays we have teams of artists sat at desks across the globe, all beavering away on the small details.  That nice crumbly wall in Uncharted 3 probably had three different artists working on it: one for the basic collision mesh, one for the render mesh and one creating the texture and normal map in the first place.  Throw in a lighting artist or two, and a particle effects guy to make the nice clouds of dust when it gets shot.  Oh, and a guy to make the impact sound effects.  Let alone the coders required to make all these systems work together.

Back in 1993, a role playing game called Dungeon Hack came out. It wasn’t unique by any stretch, but I mention it partly because of how movement systems used to work. The dungeon, as with most games of the time, existed on a grid. Each move you took moved you to the adjacent square, assuming there wasn’t a wall there. The world was rendered in first person, but you had no freedom to look around at anything other than 90 degrees, and move one square at a time. (Completely unrelated to me writing this article Legend of Grimrock, which uses this movement system in a modern game, has just been released.)

From a top down view, a dungeon looked a bit like this:

The main reason I mention Dungeon Hack is because of how the dungeons were actually created.  If you look again at the image above you’ll see that it’s possible to create the layout by using just four files, rotated into position:

By using a pre-defined set of tiles and an algorithm running a randomly generated number to lay them out, a unique dungeon was created every time you started a new game.  Rules then layered on top to place items and enemies.  In theory, there were limitless combinations and once the system for creating the content was up and running new levels could be generated almost instantly.  If user control was desired, an editor allowed players to make their own layout.

Tile based generation still exists in games like Civilisation, but it has proved very difficult to create modern 3D games with such a system, partly because of the demands on artistic quality and partly because of the desire for unique locations to visit.

Prefabs

Prefabs are a more modern way of generating content, and in essence work in a similar way to tiles, except they are stand alone units rather than part of an interconnected network.  Humans are very good at spotting repetition though, which reduces the effectiveness of using this system. Houses fairly quickly get spotted as clones, and players tend to lose interest in discovering new places if they think they’ve already seen everything there is to offer.

Procedural Generation

This is based heavily on maths and simple data structures.  The best example of what can be achieved with this method is still the demo scene:

As with the randomly generated dungeons there is a set of rules controlling the overall scope of the images here.  Similar methods can easily create large scale environments.  There are limitations though, the main one being creating areas that flow well into one and other.

On their own then, none of these approaches really gives us the flexibility or quality that we desire.  But by combining the three methods into a toolset we could help open up new ways of creating large scale content, rather than just relying on adding more artists into the equation.

Character Creation

Every story needs characters, and in this regard there are already tools that allow us to create a lot of unique models quickly.

I’ve used this video before because it’s a great example of what can be created given the right motivation and time.  We can also do very similar for animations: with a base set we can quickly add variety by using additive animations, such as a limp, or letting physics code take control.

The tools are the game

Another way is to give players the ability to create content themselves.  Creating levels for Doom was the reason I got into designing videogames in the first place, more recently Minecraft created a game that was, initially, a tool where players could create their own content.  There’s a huge demand for this type of game, where the editor is effectively the game: FarmVille and DrawSomething are other examples. They are very different types of game from the Mass Effects of the world, but it feels like there should be a way of learning from their methodologies to enable us to improve our content driven games too.

Clint Hocking has already done a very in-depth series of posts on convergence like this, and  in far more detail than I am here.  In his eight part series he suggests that other gamers could provide the context for the story, a sentiment that I agree with but relies on playing online.  EVE Online does this to a degree and some of the stories of conspiracy that come out of it are great reads.  There’s a flaw with this method though: even with relatively small player numbers (compared to World of Warcraft), each player feels like a very small fish in a very large pond.  Your influence on world events is minimal.  The reason people like playing the hero is because of the feeling of power and fantasy it gives them, and single player games give you this ability.

So, we can no doubt create worlds and characters much faster than we currently do, but that doesn’t solve the main issue I’m trying to tackle with this post: is there a way to create a story this way?

Story as a mechanic

Glance back up the page at the diagram of the dungeon for a moment.  There’s another way of showing its layout, and that is to instead show each square that requires a change in direction as a node, and the route through as connections:

If you read my previous article on content this should look familiar: this is, more or less, how we currently move through stories in content driven games.

In games today we take freedom of movement for granted: if we want to go and spend time looking at a blade of grass, we can. This is because movement systems have developed from the very simple set of rules Dungeon Hack et al had, to a complex mechanic. An almighty array of things are happening in the background to grant you that freedom of movement, from how to respond to the amount you’ve moved the analogue sticks through to how the scene behind that broken window renders.

Can we take the same steps with story? Can we create sets of rules that allow the player to drive the available story events at their will? Possibly even the freedom to create their own stories? It’s not going to be easy, and it’s not going to happen overnight, but I think we can.

How?

Firstly, take the steps required to make content creation faster and cheaper.  Then we need to understand what a story actually is. It’s well documented that all stories fall into one of seven types, which seems like a good place to start:

Overcoming the Monster: The hero learns of a great evil and goes on a journey to destroy it. Star Wars qualifies. Braveheart. Jaws. Any movie with Nazis in it. Some of the Rocky movies. (Is it obvious I am a guy?)

Rags to Riches: A sad-sack beginning that leads to a happily ever after. A lot of Dickens’ stuff fits here. Disney princess movies. Harry Potter. Most every rom-com.

The Quest: Everybody loves a quest where the hero goes on a journey to find something, which can be a Lost Ark (literal of figurative), a body (Stand By Me), or even something unknown and unseen, which is known in Hollywood as a MacGuffin. Sometimes the hero brings his entourage, too. A lot of epics are Quest stories. Like The Goonies. Some of my favorite biblical stories are quests, like Abram and The Wise Men.

Voyage and Return: Like The Wizard of Oz, where Dorothy goes to a weird place with weird rules but ultimately returns home better off. I suppose I like Oz alright, but I’d rather give props to Back to the Future, because I’m of that ilk.

Comedies get their own category, too. For some reason, two people can’t be together, which creates all sorts of antics. They eventually figure it out, though. Again, most every rom-com ever, like When Harry Met Sally, or The Money Pit. (Note: you can make anything into a comedy. For example, Monty Python is a funny Quest movie, but the category here refers to a specific kind of plot, not just anything with humour.)

Tragedies are like riches to rags, where the villian gets it in the end. MacBeth and King Lear are classic examples. Or most slasher pictures if you go for that sort of thing.

Rebirth is like a tragedy but where the hero realizes his error before it’s too late, like in It’s a Wonderful Life. Which makes me wonder, are there any slasher movies where the bad guy cleans up and catches a ray of sun at the end?

(source http://lenwilson.us/seven-stories/)

The story outlines that most single player games use falls into ether Overcoming the Monster or The Quest which is something else that we ideally will address as the medium matures.  Aside from that though, these seven rules of thumb give us a great basis for the forming of a mechanic.  How so?  Well, a mechanic is really just a set of rules, and each of these seven categories has a very definable goal for each story category.  This means that we can create a fairly simple set of IF statements to set the story goal based on the category chosen.  Whether that category is chosen by the player or randomly doesn’t matter.

Of course, a good story requires much more than that: it also requires structure. Typically these are defined into sections called Acts. Act 1 introduces the audience to the characters and scenario, Act 2 fleshes everything out, and Act 3 has the conclusion.  More rules, which means we can add another layer of complexity to our mechanic.

The numbers of main characters, their traits and motivations: all these can be defined to some level either by the player or with weighted random selections (which is where you guarantee at least one of type X and so on).

Would a story generated in this fashion be any good? Well… probably not. The missing touch is, of course, humanity. This, then, is where the player really comes in: their decisions and actions influence the actions and motivations of the computer-controlled characters alongside them. The game would adapt on the fly, so if the player decides to be friends with the bad guy for instance, a new enemy or threat would have to emerge. But the player could have the freedom to influence the story as they move through it.

The thing we still can’t do very well is to create dialogue: both in the writing of it and in the audible generation of it. Computer generated voices still sound horribly robotic, and that instantly ruins any sense of immersion or belief. (Unless, of course, you’re doing a game about robots.)

There’s another problem as well, which is the second of the issues I mentioned at the start of this article:

Artistic Desire

This is the biggest problem really: what story do we want to tell? As shown above, the general structure of a good story is always the same, and Act 3 is the conclusion. Without closure you don’t really have a story in the sense that literature and film define them. As humans we like telling stories, we like crafting them and we like the warm feeling inside that we get when we, the the hero, reaches the end.

Unless, instead, we turn our game into something more akin to a soap opera.  They have moments of closure as story arcs come to an end, but the world in which those arcs exists continues to persist and expand.

Can we do the same with a game?

Do we want to?

Well, frankly, I wasn’t any good at it at first. But through a bit of luck and a bit of perseverance, over the past few years I’ve trained and graduated a dozen interns, many of whom joined my software development team as positions opened up. It’s been an incredible experience, and I’ve been grateful to all of them for the opportunity to see them grow in talent and ability.

I have also heard from lots of people who have had bad internship experiences; and I’d like to pass on what I can tell you about making the experience a success for everyone.

What Benefits Can You Expect?

Having an internship program can really help your organization, as well as provide meaningful learning opportunities for the interns themselves. I believe the only good internship programs are ones that are mutually beneficial, providing guidance and technical skills for the intern and effective management experience for the company.

There’s a good reason that videogame companies attract interns: we do fun and challenging work; and we use skills that are complex and multidisciplinary. An intern in the videogame field has the potential to learn a lot in a very short time, especially if you involve them in the business and show them how their work contributes to a goal.

Interns know this too; and that’s why they’re willing to put up with low pay and low status in order to get started. But your company shouldn’t just take advantage of this desire – you should be prepared to provide value back to the intern.

Doing so helps to establish a company culture of caring rather than taking. A good internship program can provide a halo to your company: successful interns who enjoyed your program, even the ones who get fulltime positions elsewhere, will raise your company’s status as an employer. And designing this mentoring process helps to build your business’ ability to grow all employees, not just interns.

Your Responsibilities

Before you start, make sure you’re prepared to go through the hard work to have an intern. It isn’t all fun and games!

• Your staff should be able to dedicate 1-3 hours per day to mentoring each intern separately. The goal is to make the intern independent, but to still give them opportunity to learn. I find that 1-3 hours per day is a good start; try dividing it up between two senior employees, one the lead and one the secondary mentor.
• You should have a clearly defined introductory task for the intern; ideally a side project that doesn’t require them to learn tons of processes before they can get started. Look around – there are probably a few one-off ideas lying dusty on the shelf. When the intern has succeeded at their first task, you should gradually increase the complexity and interconnectedness of their work.
• Your intern should have a computer and enough space to get quiet work done. You want an intern to balance time between asking questions and researching their project; not all interns can shut out the distractions. If the office is noisy in general, offer headphones. Interns don’t do well with telecommuting; they need in-person supervision and reinforcement.
• You should be prepared to pay your intern. Although the US government has some rules that permit some internships to be unpaid, the rules are tricky and you’re best off not going that route. If you read the six criteria, they’re pretty vague; and your company can be on the receiving end of a serious lawsuit. Pick a wage and offer something.
• You must be excited about your own work before you can bring on an intern. Don’t neglect this! Interns don’t work well unless they have an opportunity to see and share in your passion for your business.

When you’ve ensured that your company can support an intern, it’s time to begin searching.

Acquiring your First Intern

Internship ads are written differently than traditional job listings. Make it clear that the internship is for a fixed length of time; I prefer three months. This is necessary to be fair to the intern; you need them to know that, at the end of this time period, they will be successfully on their way to the fulltime career they want.

Focus on the rich skills you will teach, the excitement of your business, and explain how your uniquely professional work environment will best prepare the intern for a full career. Limit your “requirements” to general talents, basic knowledge, and motivation; and encourage the candidate to explain to you what sets them apart. For example:

Mobile Developer Internship – Foo Inc, the world’s leading developer of Bars, has a three month paid internship position available to help its mobile app development team writing web services and user interface code. During your internship, you’ll learn the most advanced techniques for mobile user interface design from industry heavyweights. Requirements: Familiarity with mobile phones and HTML, a sharp mind, attention to detail, and a commitment to getting the job done.

Next, let’s get this advertisement in the field. Try to identify five to ten targets for your job posting. Craigslist is one place to start; each advertisement costs $25 and, in my experience, generates a decent response. Don’t forget to post the internship on your website and advertise it on LinkedIn, Facebook, and Twitter. You can also contact the dean of a local university or community college and offer your internship to their students, but be prepared to have your company vetted before you can share listings directly with students.

With the posting in hand, it’s time to begin! Schedule all the ads to start on the same day, and keep up the momentum. Interns only have a brief moment of time when they can consider an internship, so respond quickly, ideally within 24 hours after they send their resume to you. I find it works best to speak to two or three candidates each day by phone, and interview in person about a half dozen top tier candidates.

When you move on to in-person interviews, the goal is to identify three things within about an hour:

• Does the intern have enough basic knowledge to be useful in a three month window?
• Can the intern listen effectively and ask good questions?
• Does the intern have the motivation and drive to succeed?

When you’ve found a candidate that passes the test, make your pick fast; probably within the first week after interviews start. Let the intern know clearly that you will give them a great opportunity, and in return for their hard work you’ll give them a career.

Daily Experience

Once the intern starts, you should monitor them every day and provide the following:

• A limited but useful amount of mentoring. You can’t have the intern asking questions every five minutes and sapping the team’s concentration; but neither can you have an intern firing aimlessly when a quick question would get them on target. Tell your intern to ask two really good questions about their project every day.
• Tasks that can be completed. Find simple tasks and get the intern to succeed at those before moving onto more complex and risky tasks. Many interns just need to see the fruits of their labor in use to level up. Gradually increase the task complexity, but only after a task is fully seen through to completion!
• Opportunities to shape their own unique career path. After each task completes, ask them what they liked about it and what they didn’t. Seize every chance to give the intern work they find interesting. I am reminded of Harry Truman’s phrase, “The best way to give advice to your children is to find out what they want and then advise them to do it.”

In return, you need to start communicating expectations to the intern. Since many interns come straight out of college or out of a different career path, you may want to explain to them the standards required by your business, such as punctuality, professionalism, office culture, dress codes, HR rules, or adjusting to the workload. Be patient with them, and give them time to cope, but also be clear: meeting the business rules is a condition of the internship.

As work progresses, find opportunities to compliment your intern and gradually expose them to more and more advanced projects. I like to provide about nine compliments for each criticism I give. If you fall far south of that line, perhaps either the intern or your motivational style needs to change. Constructive criticism is your friend; honest and straightforward feedback is a critical necessity for career development. If an intern doesn’t improve after a warning, it’s best to let them move on to a place they can be more successful.

Ending an Internship

At the end of three months – or whatever time frame you picked – you should release your intern. During the last few weeks of the time frame, try to give them general career counseling. Set them up a few meetings where you can answer their questions about jobs in their chosen field.

If you are lucky enough to have a full-time position open that is suitable for the intern, encourage them to apply for the position and give them a proper hearing. Doing so professionally results in a clear transition from intern to employee.

If your business doesn’t have a position available, don’t fret. Hold your intern’s hand up high and celebrate their work. Give them a strong reference, and offer to hand-deliver their information to colleagues you may know at other companies. Take the intern out to lunch to celebrate the launch of a successful career – you’ve both earned it!

UDK

http://udk.com/

Most are familiar with the Unreal Development Kit.  It’s a proven engine that’s been used in a tonne of AAA titles, but how does it fare for indie developers?  The first thing you’ll notice with UDK is the learning curve.  It’s steep.  Developers I’ve spoken to have all expressed this same sentiment, and my own experiences with UDK left me feeling that UDK seemed needlessly complicated.  I had taken a few courses using the UDK in the past and while practice makes perfect, even when I became more familiar with UDK I found I simply didn’t like using it compared to other alternatives.  The second strike is the need to learn Unreal Script.  It’s a fairly straight forward language in my opinion however just that you need be confined to Unreal Script can take away from valuable development time when you’re starting off.

UDK is capable of delivering high quality graphics out of the box but it seems geared towards First Person Shooters (much like CryEngine).  I’ve heard some complain about the difficulties involved in trying to bend UDK to other genres.  FPS games developed with UDK also have a tendency to end up feeling like Unreal Tournament clones.  UDK now supports iOS development in addition to Windows but don’t expect to port your projects over to anything else.

UDK is free for non-commercial use.  Plan on selling your game and you’ll need to fork over $100 with no royalties to worry about until $50,000.  After that, expect to pay a 25% royalty, which when you consider IOS development and the 30% Apple takes, can certainly add up.  UDK is a bit of a sacred cow for some in the development community, but for indie developers it’s big, unwieldy and just limited in supported platforms.  If you’re looking to add your shooter to a saturated shooter market, UDK may work for you but I wouldn’t recommend for smaller teams of developers.

Notable title:  The Ball

Shiva3D

Points in Homogenous coordinates
In our usual Cartesian Coordinate system, we can represent any points in 3D space in the form of (X, Y, Z). While in Homogenous coordinates, a redundant component w is added which resulting in a form of (x, y, z, w). Multiplying any constant (except zero) to that 4-components vector is still representing the same point in homogenous coordinates. To convert a homogenous point back to our usual Cartesian Coordinate, we would multiply a point in homogenous coordinates so that the w component is equals to one:

In the following figure, we consider the x-w plane, a point (x, y, z, w) is transformed back to the usual Cartesian Coordinates (X, Y, Z) by projecting onto the w=1 plane:

figure 1. projecting point to w=1 plane

The interesting point comes when the w component is equals to zero. Imagine the w component is getting smaller and smaller, approaching zero, the coordinates of point (x/w , y/w , z/w, 1) will getting larger and larger. When w is equals to zero, we can represent a point at infinity.

Line Segments in Homogenous coordinates
In Homogenous coordinates, we still can represent a line segment between two points P0= (x0, y0, z0, w0) and  P1= (x1, y1, z1, w1) in parametric form:

Then we can get a line having the shape:

figure 2. internal line segment

The projected line on w=1 is called internal line segment in the above case.
But what if the coordinates of P0 and P1 having the coordinates where w0 < 0 and w1 > 0 ?

figure 3. external line segment

In this case, it will result in the above figure, forming an external line segment. It is because the homogenous line segment have the form L= P0 + t * (P1-P0), when moving the parameter from t=0 to t= 1, since w0 < 0 and w1 > 0, there exist a point on the homogenous line where w=0. This point is at infinity when projected to the w=1 plane, resulting the projected line segment joining P0 and P1 passes through the point at infinity, forming an external line segment.

The figure below shows how points are transformed before and after perspective projection and divided by w:

figure 4. region mapping

figure 5. external triangle

In the above figure, the shaded area represents the external triangle formed by the points P0, P1 and P2. This kind of external triangles may appear after the perspective projection transform. And this happens in our real world too:

an external triangle in real world

the full triangle of the left photo

In the left photo, it shows an external triangle with one of the triangle vertex far behind the camera while the right photo shows the full view of the triangle and the cross marked the position of the camera where the left photo is taken.

Triangles clipping
To avoid the case of external triangles, lines/triangles should be clipped in homogenous coordinates before divided by the w-component. The homogenous point (x, y, z, w) will be tested with the following inequalities:

(The z clipping plane inequality is 0 <= z <= w in the case for D3D, it depends on how the normalized device coordinates are defined.) Clipping by inequality 1,2,3 will effectively clip all points that with w < 0 because if w < 0, say w = -3:

3 <= x <= -3     =>     3 <= -3

which is impossible. But the point (0, 0, 0, 0) is still satisfy the first 3 inequalities and forming external cases, so inequality 4 is added. Consider a homogenous line with one end as (0, 0, 0, 0), it will equals to:

L= (0, 0, 0, 0) + t * [ (x, y, z, w) - (0, 0, 0, 0) ] = t * (x, y, z, w)

which represent only a single point in homogenous coordinates. So triangle (after clipped by inequality 1, 2, 3) having one or two vertices with w=0 will result in either a line or a point which can be discarded. Hence, after clipping, no external triangles will be produced when dividing by w-component. To clip a triangle against a plane, the triangle may result in either  1 or 2 triangles depends on whether there are 1 or 2 vertex outside the clipping plane:

figure 6. clipping internal triangles

Then the clipped triangles can be passed to the next stage to be rasterized either by a scan line algorithm or by a half-space algorithm.

Below is the clipping result of an external triangles with 1 vertex behind the camera.

clipping external triangle in software rasterizer

Below is another rasterized result:

rasterized duck model

reference of the duck model

Conclusion
In this post, the maths behind the clipping of triangles are explained. Clipping should be done before projecting the homogenous point to the w=1 to avoid taking special cares to clip the external triangles. In the next post, I will talk about the perspective interpolation and the source code will be given in the next post (written in  javascript, drawing to html canvas).

And lastly special thanks to Fabian Giesen for giving feedback during the draft of this post.

References
[1] http://research.microsoft.com/pubs/73937/p245-blinn.pdf
[2] http://medialab.di.unipi.it/web/IUM/Waterloo/node51.html
[3] http://kriscg.blogspot.com/2010/09/software-occlusion-culling.html
[4] http://www.slideshare.net/guerrillagames/practical-occlusion-culling-in-killzone-3
[5] http://www.slideshare.net/repii/parallel-graphics-in-frostbite-current-future-siggraph-2009-1860503
[6] http://fgiesen.wordpress.com/2011/07/05/a-trip-through-the-graphics-pipeline-2011-part-5/
[7] http://devmaster.net/forums/topic/1145-advanced-rasterization/

I found him hiding behind a giant leaf, wearing a giant leaf. He was pressing binoculars against his eyes, gazing towards a nearby algae-covered pool. I crouched next to him and started asking him questions.

Ariel Gross: So, Damian, what is it that you do? Aside from questing after fabled beasts of yore?

Damian Kastbauer: I’m a technical designer for games, which is a somewhat nebulous term. It’s defined pretty well by Rob Bridgett in this article. Essentially, I try to serve as a bridge between sound content and programming/engine-side integration in order to create systems for sound playback, or just plain getting sounds in the game working and sounding right.

AG: Nice. My opinion is that sound design in games is only as good as its implementation. What do you think about that?

DK: It’s often said that the best sound can end up sounding bad if it’s not implemented properly, so, a lot of the time I’m trying to let the sound do its thang… which is not always easy. Since I don’t create content, and I’m not a programmer by definition, I usually spend a lot of time either building tools, streamlining pipelines, and strategizing audio features with the help of the programming team or working with audio middleware tools to similarly create the smoothest integration possible.

Damian finally released the binoculars, which then fell and dangled around his neck, but kept his hands in place, and continued to look off towards the pool with his hand-noculars. It was an interesting technique that I had never seen used in the field before. He made a refocusing gesture and continued to peer out through his hand holes.

AG: So… do you find yourself being a middle man between Team Audio and Team Programming? That could be challenging, what with all the unrequited love and/or red-hot rage between those two disciplines.

DK: Definitely, I’m helping to bridge the gap between the two worlds. Whenever possible, I like to enable content designers to just “design in the box” all day from the DAW without having to worry about the sometimes-labyrinthine pipeline to get sound into the game. In this way, they can focus on what they’re really into and I can go to town with making sure it all works within the context of the game. It’s a balance that requires human interaction, but I’m pretty keep on that as well.

AG: Wow, you just said labyrinthine. I’m golf clapping right now. In my head. And in my heart. I don’t want to scare off the paisley hippo. You’re welcome. So, have you found any common issues between Team Audio and Team Programming among multiple developers, and do you take measures to help both sides see eye to eye?

DK: I think one of the missing or underdeveloped pieces of some studios is the direct communication between Team Audio and the rest of the teams. It’s common for audio to touch almost every corner of the pipeline and development process.

With Team Audio often sequestered within a sound-proof cave deep within the bowels of a studio, it can be hard to get the kind of happy accident or magic moment that can come from being out in the pit or lined up in a hallway.

AG: This is painfully true.

DK: One of the things that has always seemed right about my involvement with different game studios on-site is the necessity of being “on the floor” with the other disciplines… which has always been a positive in bridging that gap.

When I work remotely, I put a tremendous amount of effort into achieving a high level of communication between the different teams on a project. Whether it’s e-mail, IM, Skype, or occasion on-site visits during development, the communication between people is among the most important aspects of working on a project.

If I am outside the fence, having an on-site advocate for audio is a definite plus, and in most cases, it is key to making my role as a facilitator for sound actually happen.

Some foliage near the murky pool began to shake. Damian dropped his hand-noculars and placed one hand over his mouth to ensure complete silence. He placed his other hand over my mouth, too. I resisted the urge to lick his hand. A rabbit bounced from the foliage. We both sighed deeply.

AG: Miv viff fuhm coffm prbflrm…

Damian removed his hand from my mouth.

AG: Is this a common problem at studios you’ve worked with? That Team Audio doesn’t pop their heads up through the manhole enough to be part of the greater discussion?

DK: Whew, well, I think that as a matter of course it is hard to have visibility across the team in the same way as being out in an open floor plan, when part of your daily work requires you to be locked away in a padded room.

That is, people can’t SEE you and demand your attention by rolling up to your workstation. I mean, there’s a door… you have to turn the knob… like, maybe even knock… and then get blasted in the face by high decibel explosions in order to interface. We know how averse most people are to moving in the first place, let alone opening a door.

I chuckled.

DK: A little dry, mid-west sarcasm for you.

I chuckled more, leading to a whole-body belly laugh, leading to a maniacal, snorting cackle, ending finally with a very feminine giggle. Then we sat motionless for what felt like an eternity.

AG: Sorry. What you say is completely true. Before tracking you down in the Congo, I hadn’t left my seat in four years. Not even to use the restroom. I held it. What you saw at the GDC was my hologram. A hologram with mass. I digress. What is the number one reason why a studio hires a technical sound designer?

DK: The number one reason I am hired is to help ship games. Having shipped my fair share, I would like to say that I know all the tricks in the book, but this is absolutely not true.

What I hope that I bring to a project is an ability to work together with people to get things done, through thick and thin, regardless of the task at hand.

Sometimes I don’t even know what I’m going to be doing until I get involved and assess the situation… this can be true of the studios as well. They’re not sure what exactly needs to be done, but they need someone to help pull it off in the eleventh hour.

AG: You’re basically a hero that is brought in to save the day.

DK: If I can come in and take the burden off of content designers, help design complex playback systems for physics or procedural animations, help wrangle memory budgets or streaming look-ahead times, then that is one less thing for someone, who already has too much on their plate, to worry about.

For remote work, it can be at any time during the project, I work from the home studio connected to a VPN using source control while making local builds from day one.

When I do work on-site for an extended period of time, it’s usually toward the end of a project for a few months.

At this moment, the waters of the pool began to ripple as a large, paisley-patterned mound broke the surface of the water. Then, two paisley ears emerged, followed by blinking eyes. Damian fidgeted with excitement, but stayed put.

AG: You mentioned memory and streaming.  This is something that some people don’t realize. Someone on Team Audio is usually tasked with wrangling memory budgets and streaming bandwidth. It’s really techy work.

This isn’t always expertise that is considered when hiring an audio designer. I’ve found that studios tend to focus more on whether or not an applicant can make rad sounds, with less emphasis on the technical side of things.

There’s usually some requirement that says, “familiarity with middleware tools and implementation,” but I get the sense that the weighting at most companies is 80% creative aptitude and 20% technical prowess. Does that seem accurate to you? Or am I totally wrong? And ugly looking?

DK: I think there are a majority of people who come to studios with a sound design background. Until recently, it has been almost non-existent to have someone come out of school with an education in the technical side of game audio.

It used to be that “audio implementer” was the job that someone new to the industry would take on their way to becoming a sound designer… or even a musician or composer for games… that was really the only way to learn the ropes.

When I got into games, it was clear to me that the technical side was the only thing that I wanted to do. Thankfully, in the past few years, it has become a niche that has benefited from a group of people whose interests are purely in the technical, which has helped to establish legitimacy for the specialization.

AG: We used an in-game editor to place ambient audio emitters on Saints Row 2. Sometimes the frame rate would tank while I was moving an ambient emitter around, and I’d fling the emitter object 100 miles away, sometimes into a different game entirely. I think a few of those emitters landed in Red Faction: Guerrilla.

DK: Hahaha… user interface physics lerping… TOOLS!

AG: Do you ever find yourself caught up in the mystery of missing sounds? Like, the audio team is sure that a sound should be playing, but it’s not. Is that something you run into a lot?

DK: Totally! Someone did a model swap but forgot to bring the audio hooks with it, or moved the entire level geometry across the level away from the audio emitters that were placed on a different layer in the editor… happens ALL the time, and the resources to figure it out are sometimes underestimated.

With all the different ways that people can approach game audio these days: mods, Unity, UDK, FMOD, Wwise, SDK documents, Max/MSP or pd, it’s becoming common to have fresh faces joining the team who already have a good handle on the tasks at hand and are ready for the challenge.

The fabled paisley beast began to slowly emerge from the algae-covered pool. Watching the water cascade off of it’s psychedelic hide was glorious. Swirls of color and waves of sweet smells washed over me. Granted, I had also been munching on some random red berries that I’d found on the ground.

AG: I know plenty of sound designers that don’t want to do the implementation. They look at it as a boring chore. A chore-bore. A borch. Anyway, I could see some Team Audios out there wanting to hire you on full time. Do you ever get this? Have you considered an in-house gig?

DK: I’ve worked remotely on a couple of projects for a longer duration (1-2 years) serving as a conduit for getting content in the game. I’ve worked a ton with Bay Area Sound as an outsource audio solution involving their audio and music content creation in conjunction with my management of the technical side in what has been a perfect solution for smaller developers and teams that need supplemental assistance.

I worked for five years in this capacity on the games for Telltale. I integrated content provided by Bay Area Sound and Harmony Machine into The Saboteur from Pandemic Studios for about a year to help augment their in-house audio team, which included building vehicle systems in Wwise to work with the simulation team who was responsible for the cars driving around in the open world.

Most recently I’ve been working again with Bay Area Sound to integrate content and tool the pipeline for a couple of MMOs using the Hero Engine in conjunction with FMOD.

AG: How about documentation? I have seen with mine own eyes the lack of documentation out there. Do you find yourself having to trudge through undocumented systems a lot? Do you end up documenting them yourself as part of your services?

DK: Documentation is always a tricky one. A lot of developers who have been iterating on in-house engines and tool sets carry with them a ton of institutional knowledge… usually within their brains. There’s usually a pretty intense time of education at the beginning of the project where I do my best to document… for my own sanity as well as anyone who might find themselves in a similar position in the future.

I recently worked with a company and provided an evaluation of their audio middleware integration in addition to recommendations for how to move forward with some key changes. In that case, it was documentation and justification for the different recommendations, as well as education in order to get everyone thinking about the possibilities.

AG: Do you find that most companies that you work with have an audio programmer on stand-by to compliment you? Compliment both in terms of assisting you in a complimentary fashion, and also to tell you that your shirt looks very nice tucked in.

DK: It’s definitely becoming more common for developers to recognize the need for audio programming support throughout the project. While I think it’s becoming, I think there is still a false expectation that you can just throw on the UI programmer for a couple of months and sew everything up.

The need for dedicated audio programmers is definitely growing as we continue to scale in meeting the demands in quality during the current generation.

So, I guess, having an audio programmer is necessary to compliment the technical side of sound design, especially a dedicated professional who is invested in sound. Their ability to bring not just programming, but a knowledge of audio to the table can only mean an increase in the quality of the audio and the quality of life for Team Audio.

That, and yeah… someone to tell me when my dashiki doesn’t match my socks.

AG: Last question, and then you can get back to your quest. What do you think will change from a tech audio standpoint with the next generation of consoles?

DK: I think the industry at large will further close the gap between pre-rendered and in-engine… across all disciplines, a continuation to move forward into real-time.

What this means for audio is building on the sample-based methodology that has firmly taken root in the current generation while using more procedural audio, synthesis, and DSP to modify or accent sample-based content dynamically at runtime.

Additionally, increased transparency and better communication in both directions between the game audio and audio engines will develop, which will in turn necessitate a further push towards enabling this functionality within usable tool sets. This accessibility will reach into every corner of the audio production pipeline, simultaneously exposing the technology while making it easier to adapt it to the interactive needs of the game.

Hopefully the new consoles will also make espresso.

AG: Espresso feature would be cool. I’m hoping that someday I can feed a slice of bologna directly into the console. It would recognize the meat as bologna and procedurally create a game called Bologna Quest, where a young adventurer named Ariel, clad in the finest bologna armour, would climb the mountain of… Damian?

Damian, recognizing that I had started my typical bologna-related sign-off, was already planting a kiss upon the lips of the sleeping paisley hippopotamus. Then, a flash of light, and they both vanished, never to be seen again.

Disclaimer: in an ideal world I’d like to try to avoid assumed knowledge, but keeping up the level of detail in each post that this entails is, frankly, too much work. Consequently I will from now on point you at post 6 as a “how to” and then get on with it…

Here are the backlinks for preceding articles in the series (warning: it might take you a while, the first few are quite long):

1. http://altdevblogaday.com/2011/11/09/a-low-level-curriculum-for-c-and-c/
2. http://altdevblogaday.com/2011/11/24/c-c-low-level-curriculum-part-2-data-types/
3. http://altdevblogaday.com/2011/12/14/c-c-low-level-curriculum-part-3-the-stack/
4. http://altdevblogaday.com/2011/12/24/c-c-low-level-curriculum-part-4-more-stack/
5. http://altdevblogaday.com/2012/02/07/c-c-low-level-curriculum-part-5-even-more-stack/
6. http://altdevblogaday.com/2012/03/07/c-c-low-level-curriculum-part-6-conditionals/ [see near the top of this post for details on compiling & running the code snippets]

The conditional operator

I assume that everyone’s familiar with the conditional operator, also known as the “question mark”, or the ternary operator (“ternary” because it’s the only C/C++ operator that takes three operands).

If you’re not, here’s a link so you can catch up (I predict that you will be so stoked to find out about it that you will be over-using it within the week).

Personally I heartily approve of the conditional operator when used judiciously, but it’s not always great for source level debugging because it’s basically a single line if-else and can be hard to follow in the debugger (in fact I’ve heard of it being banned under the coding standards at more than one company, but there you are we can’t all be sane can we?).

Anyway, let’s have a quick look at it with some code:

1
2
3
4
5
6
7
8
9
10

#include "stdafx.h"
 
int main(int argc, char* argv[])
{
    // the line after this comment is logically equivalent to the following line of code:
    // int iLocal; if( argc > 2 ){ iLocal = 3; }else{ iLocal = 7; }
    int iLocal = (argc > 2) ? 3 : 7;
 
    return 0;
}

If you remember the the assembler that a basic if-else generated in the last article, then the assembler generated here will probably bust your mind gaskets…

Note:

1. I’ve deliberately left the function prologue and epilogue out of the asm below, and just left the assembler involved with the conditonal assignment
2. if your disassembly view doesn’t show the variable names, then you need to right click the window and check “Show Symbol Names”

1
2
3
4
5
6

     5:     int iLocal = (argc > 2) ? 3 : 7;
01311249  xor         eax,eax
0131124B  cmp         dword ptr [argc],2
0131124F  setle       al
01311252  lea         eax,[eax*4+3]
01311259  mov         dword ptr [iLocal],eax

Clearly this is not very much like the code for the simple if-else that we looked at previously.

This is because there is trickery afoot and the compiler has chosen to do sneaky branchless code to implement the logic specified by the C++ code.

So, let’s examine it line by line:

• line 1 – uses the xor instruction to set eax to 0. Anything XORed with itself is 0.
• line 2 – as in the previous if examples this uses cmp to test the condition, setting flags in a special purpose CPU register based on the result of the comparison.
• line 3 – this is a new one! The instruction setless equal sets its operand to 1 if the 1st operand of the preceding cmp was less than or equal to the 2nd operand, and to 0 if it was greater. We’ve not seen the operand al before, it’s a legacy (386) register name which now maps to the lowest byte of the eax register (if you’re a sensible person and are stepping through this code in your debugger with the register window open, you will see that this instruction causes the eax register to be set to 1 – also note that this only works because eax has already been set to 0).
• line 4 – uses the load effective address instruction do do some sneaky maths that relies on the value of eax set by setle in line 3.
• line 5 – moves the value from eax into the memory address storing the value of iLocal

That’s all fine, but how does it work?

Firstly, note that at the assembler level the comparative instruction setle is (as in the previous post’s examples) testing the opposite condition to the conditonal specified in the C++ code.

This means that the eax register will be set to 0 in line 3 if argc is greater than 2, which in turn means that the eax*4+3 part of line 4 will evaluate to (0*4)+3 - i.e. 3.

Conversely, if argc is less than or equal to 2, the eax register will be set to 1 which in turn means that line 4 will evaluate to (1*4)+3 - i.e. 7.

So, as you can see, the assembler is doing the same branchless set of instructions  regardless of the condition, but using the 0 or 1 result of the conditional instruction in the maths to cancel out or include one of the terms and give what I like to call a “mathematical if”. Clever.

Incidentally this sort of branchless-but-still-conditional code has been a bit / lot of a hot topic over the last few years, especially on consoles  since their CPUs are particularly branch mis-prediction sensitive.

Judicious use of the “branchless condtional” idiom is a tool that can be used to combat branch (mis-)prediction related performance issues – for an example of this, see the use of the fsel PPU instruction in this ADBAD post by Tony Albrecht, and for brief a discussion of branch prediction issues (primarily PC related) see this article by Igor Ostrovsky (who works for Microsoft).

The conditional operator (part deux)

So, clearly our above super-simple-sample resulted in the compiler generating clever assembler because of the constant values in it; interesting certainly, but not necessarily representative of most “real world” assembler.

Let’s see what happens if we use variables with the conditional operator…

1
2
3
4
5
6
7
8
9
10

#include "stdafx.h"
 
int main(int argc, char* argv[])
{
    int iOperandTwo = 3;
    int iOperandThree = 7;
    int iLocal = (argc > 2) ? iOperandTwo : iOperandThree;
 
    return 0;
}

And, here’s the relevant disassembly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

     5:     int iOperandTwo = 3;
00CF1619  mov         dword ptr [iOperandTwo],3
     6:     int iOperandThree = 7;
00CF1620  mov         dword ptr [iOperandThree],7
     7:     int iLocal = (argc > 2) ? iOperandTwo : iOperandThree;
00CF1627  cmp         dword ptr [argc],2
00CF162B  jle         main+25h (0CF1635h)
00CF162D  mov         eax,dword ptr [iOperandTwo]
00CF1630  mov         dword ptr [ebp-50h],eax
00CF1633  jmp         main+2Bh (0CF163Bh)
00CF1635  mov         ecx,dword ptr [iOperandThree]
00CF1638  mov         dword ptr [ebp-50h],ecx
00CF163B  mov         edx,dword ptr [ebp-50h]
00CF163E  mov         dword ptr [iLocal],edx

Since the conditional operator is now assigning from variables we’d expect it to generate something that looks more like the sort of code we saw from the basic if-else we looked at last time, which it has.

We have the expected cmp followed by a conditional jump testing against the opposite of the conditional, then two blocks of assembler, the first of which (lines 7 to 9) unconditionally jumps over the second (lines 10 and 11) if it executes, so essentially it’s behaving more or less as expected; however there’s clearly some interesting stuff happening in there:

1. the two branches use different registers to store their intermediate values; the first uses eax, the second uses ecx
2. both branches store their result to the same memory address in the Stack (see this post if you don’t know or can’t remember about Stack Frames) – i.e. [ebp-50h]
3. the code that assigns the value to iLocal (lines 12 and 13) only exists once and is executed regardless of which branch was taken; it takes the value from[ebp-50h] and writes it into iLocal using uses a third register (edx)

The use of different registers for the different branches in step 1 looks like it might be significant but (according to several expert sources) this is apparently perfectly normal compiler behaviour and not anything to read into.

Steps 2 and 3 show that the that generated from the conditional operator (at least with VS2010) isn’t directly equivalent to the intuitively equivalent if-else statement:

1
2
3
4
5
6
7
8
9
10
11

// intuitively equivalent if-else of
// int iLocal = (argc > 2 ) ? iOperandTwo : iOperandThree;
int iLocal;
if( argc > 2 )
{
    iLocal = iOperandTwo;
}
else
{
    iLocal = iOperandThree;
}

Rather than choosing between one of two assignments like this if-else, the assembler generated for our use of the conditional operator does exactly what we told it to: choose one of two values (store it temorarily in the Stack) and assign iLocal from it.

A few final notes on the ? operator:

1. You can see that less lines of C++ code does not equate to less assembler
2. It can be nested, but don’t do it! It’s hideous and will also be very hard to follow when source-level debugging
3. Be very careful with operator precedence when using it. Use brackets to ensure it will resolve the way you intend.

Switch Statements

The final type of conditional statement we’ll be looking at is the switch statement. Like the conditional operator, the switch statement is an often abused and maligned construct that you wouldn’t want to live without.

To be fair to the switch statement it’s not the fault of the switch statement that it’s possible for maniacs to write brittle and insane code using them.

An aside about switch statements

Where I have consistently found really horrific examples of switch statements is when an originally stateless synchronous system has been forced to become asyncronous and state driven under time pressure. This specific situation seems always to somehow spawn the kind of monolithic, hard to follow, difficult to change, architecturally brittle switch statements that have given the switch statement a bad rep over the years.

Code that has had network functionality retrofitted to it is (in my experience) an extremely common place to find problem switch statements. It’s always better to fix a system properly if it starts to look systemically broken than it is to soldier on regardless, and if it looks like you need to introduce a set of states into a system to then (in my experience) it’s architecturally more sensible to use polymorphic behaviour (e.g. a state class with one or more virtual functions) than a switch statement.

Where were we?

Sorry, let’s get on and take a look at a switch statement…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#include "stdafx.h"
 
int main(int argc, char* argv[])
{
    int iLocal = 0;
 
    // n.b. no "break" in case 1 so we can
    // see what "fall through" looks like
    switch( argc )
    {
    case 1:
        iLocal = 6;
    case 3:
        iLocal = 7;
        break;
    case 5:
        iLocal = 8;
        break;
    default:
        iLocal = 9;
        break;
    }
 
    return 0;
}

And here’s the disassembly…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

     9:     switch( argc )
00C61620  mov         eax,dword ptr [argc]
00C61623  mov         dword ptr [ebp-48h],eax
00C61626  cmp         dword ptr [ebp-48h],1
00C6162A  je          main+2Ah (0C6163Ah)
00C6162C  cmp         dword ptr [ebp-48h],3
00C61630  je          main+31h (0C61641h)
00C61632  cmp         dword ptr [ebp-48h],5
00C61636  je          main+3Ah (0C6164Ah)
00C61638  jmp         main+43h (0C61653h)
    10:     {
    11:     case 1:
    12:         iLocal = 6;
00C6163A  mov         dword ptr [iLocal],6
    13:     case 3:
    14:         iLocal = 7;
00C61641  mov         dword ptr [iLocal],7
    15:         break;
00C61648  jmp         main+4Ah (0C6165Ah)
    16:     case 5:
    17:         iLocal = 8;
00C6164A  mov         dword ptr [iLocal],8
    18:         break;
00C61651  jmp         main+4Ah (0C6165Ah)
    19:     default:
    20:         iLocal = 9;
00C61653  mov         dword ptr [iLocal],9
    21:         break;
    22:     }

This is more or less exactly what you’d expect:

• line 1 stores argc into the Stack at [ebp-48h]
• then block from line 2 to 9 implements the logic of the switch by a series of comparisons of this value against the constants specified in the case statements and associated conditional jumps to the assembler generated by the code in the corresponding case statement
• if none of the conditional jumps are triggered, the logic causes an unconditional jump to the default: case.
• in particular, note that:

1. wherever the break keyword is used this causes an unconditional jump past the end of the assembler generated by the switch
2. the “drop through” from case 1: into case 3: in the high level code happens at the assembler level as a by product of the organisation of the adjacent blocks of instructions generated for the switch by the compiler, and the lack of unconditional jump at the end of the assembler for case 1:

If you look at assembler from the sample if-else-if-else in the last article; you should be able to see that the assembler generated for this switch is (more or less) what would happen if we had written the switch as an if-else-if-else and then re-organised the assembler so all the logic was in one place at the top, and the assembler generated for each code block was left where it was.

So other than the fact that the switch statement is a very useful C/C++ language convenience for managing what would often otherwise be messy looking and error prone chains of if-else-if-else statements, based on this example it doesn’t appear to be doing anything which might offer a significant advantage at the assembler level – so why would I have claimed that the compiler might generate “pretty cool assembler” for a switch?

Before we assume we’ve seen it all, let’s try using a contiguous range of values for the constants in the cases of the switch. You know, just for fun – and for the sake of simplicity let’s start at 0.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#include "stdafx.h"
 
int main(int argc, char* argv[])
{
    int iLocal = 0;
 
    switch( argc )
    {
    case 0:
        iLocal = 4;
        break;
    case 1:
        iLocal = 5;
        break;
    case 2:
        iLocal = 6;
        break;
    case 3:
        iLocal = 7;
        break;
    }
 
    return 0;
}

And here’s the disassembly it generates…

Ok, so this time something more interesting is definitely going on – n.b. I’ve used a screenshot rather than just pasting the text because we need to look in a memory window to make sense of it.

So what exactly is it doing?

• it moves argc into eax, then stores it into the Stack at [ebp-48h]
• it then compares the value stored in the address [ebp-48h] with 3 (i.e. our maximum case constant)
• if this value is greater than 3 then ja (jump above) on the next line will cause execution to jump to 8D1658h – the 1st instruction after the code generated by the case blocks, skipping the switch
• if the value is less than or equal to 3 then the value is moved into ecx, and we then have an unconditional jump to … somewhere :-/

Ok, so that final unconditional jump has some syntax we’ve not yet seen for its address operand, and which clearly isn’t a constant:

1

jmp    dword ptr    (1B1664h)[ecx*4]

This says “jump to the location stored in the memory address at an offset of 4 times the value of ecx from the memory address 8D1664h“, so how is this implementing the logic of the C++ switch statement?

To answer this question we need to look in a memory window at the address 8D1664h (n.b. to open a memory window from the menu in VS2010 when debugging go Debug -> Windows -> Memory -> … and choose one of the memory windows. To set the address just copy and paste it from the disassembly into the “Address:” input box. You will also need to right click and choose “4-byte integer” and set the “Columns:” list box to 1 to have it look like the screenshot above).

So, if you cast your eyes up to the memory window on the left of the screenshot above, you will see that the top 4 rows are highlighted, these values start at address 8D1664h and are 4 byte integers (hence the ecx*4 in the operand) – which specifically in this case are pointers.

The instruction jmp dword ptr (8D1664h)[ecx*4] will jump to the value stored in the address:

• 8D1664h + 0 = 8D1664h if the value in ecx is 0
• 8D1664h + 4  = 8D1668h if the value of ecx is 1
• 8D1664h + 8  = 8D166Ch if the value of ecx is 2
• 8D1664h + Ch  = 8D1670h if the value of ecx is 3

So, the four highlighted rows make up a jump table – since our case constant’s range is from 0 to 3 it is an array of 4 pointers – with each element of the array pointing to the execution address of the case block matching its array index.

You can verify this by checking the addresses of the first instruction generated for each case against the 4 values stored in the array.

Maybe it’s just me, but I think this is some pretty cool assembler. It’s certainly more elegant that the assembler generated by the first switch we looked at, but what – if anything – is the advantage of this over the assembler that was generated for the previous case statement?

In theory this jump table form reaches the code in constant time for all cases, whereas in the if-else-if-else form the time to reach the code corresponding to each case will be proportional to the number of previous cases in the switch statement.

You’re pretty unlikely to find that a switch statement is a performance bottleneck in your code (unless you’ve done something silly) but, all things being equal, the jump table appoach uses less instructions to get to the conditional which is normally A Good Thing and – in theory – should make it faster on average.

One final note on switch statements; I am reliably informed that in addition to the if-else-if-else alike linear search behaviour for resolving the correct case to execute, most modern compilers are also capable of generating a binary search for the cases of switch statements with appropriate ranges of case constant values.

Using a binary search rather than a linear search will improve average search time from linear to logarithmic (i.e. O(n) to O(log n)). However, in the average case a binary searched switch will still almost always take more instructions and branches to reach the correct case than a jump table switch.

It’s also possible that the compiler might choose to use one or more of these methods in a single switch, though this would probably require a large number of cases in the switch and ranges of case constants with very specific properties so it’s not likely you will come across these very often.

A couple of final things to note about switch statements:

1. the compiler should be able to generate a jump table regardless of the order in of the constants in your code (e.g. case 2: … case 1: … case 3: … should still work fine)
2. having a range of case constants that starts at 0 makes the conditional code around a jump table simpler, as it removes the lower bounds check
3. a jump table should get created as long as the overall range of constants is large enough and/or closely packed enough for the compiler to decide it’s worthwhile even if they’re not completely contiguous. Look at the disassembly if you want to check.

 Summary

So, this concludes our look at conditionals, hopefully you’ve found it interesting and illuminating ;)

A final point to take away from our look at conditionals is that whilst the compiler could generate the same assembler for an if-else as for the conditional operator it doesn’t. Similarly, it could generate the same assembler from an if-else-if-else as for a  switch statement but it doesn’t do that either.

In part, this shows the limits of the compiler but also shows the importance of using the appropriate conditional for purpose – the benefit is that which you use makes your intent clearer to human readers of your code.

We’ve now covered enough ground that you should be finding that you can apply the information I’ve given you to everyday programming problems such as debugging release code, or code you don’t have debugging information for.

The main things I’d like you to take away from our look at conditionals are all things that will help you when debugging without symbols:

1. anytime you see cmp followed by a jxx to a nearby address in the disassembly you’re probably looking at code generated by a conditional statement in the C/C++ code
2. if the address operand to the jump instruction is lower than the current instruction’s address (i.e. it’s jumping backwards) you’re most likely looking at a loop
3. assembler generated from conditionals generally tests the opposite of the test being done in the C / C++ code

By using these heuristics, looking at the values in the registers, the values in the Stack that have been written by the assembler, and by looking up your current address in the symbol file to tell you which function you’re in (if you’re not generating a symbol file for all your builds you should be – look in the documentation for your platform’s compiler toolchain to find out how) you should be able to make an educated guess at what variables in the C/C++ code are likely to be causing the current issue and this will usually tell you why it crashed, or give you a lead so you can Sherlock Holmes your way to the root of the problem – it’s certainly a lot quicker than the ubiquitous insertion of the many printf()…

Our next topic will be loops, which obviously also use conditional jumps (which is why we covered conditionals first…)

One final thing…

Thanks to Tony, Bruce, and Fabian for extra information, advice, and proof reading.

And, for those of you who like to go off and look for yourselves (hopefully most of you!), I’ve recently discovered this wiki book on x86 Assembler http://en.wikibooks.org/wiki/X86_Assembly. It has a large overlap with this series of articles and also covers programming in x86 assembler. Highly recommended – I’ve certainly found it pretty useful.

A final, final nugget of wisdom from Bruce Dawson:

Another problem I’ve seen with ?: is with people who have ‘cleverly’ created string classes that both have a const char* constructor and a const char* conversion operator. This has potential to be exquisitely dangerous and exquisitely inefficient, by allowing lots of hidden conversions. These come to their fruition with ?:. Imagine this:

return bFlag ? mStringObject : “Hello world”;

The question is, what is the type of this expression? Does mStringObject get converted to a const char*, or does “Hello world” get converted to a string object? I have no idea, and nobody should memorize the relevant rules. Such code is too fragile and dangerous.

Some people might assume that the type of the ?: will depend on the return type of the function but that is not true. They are independent. Thus, it is quite possible that “Hello world” will be converted to a string object and then (if the return type of the function is const char*) this (temporary!!!) string object will be converted back to a const char*. In addition to being inefficient this leads to undefined behavior, since we are returning a pointer to memory owned by an object that is destroyed as the function exits.

Using the ?: operator with mismatched types is evil evil evil. String classes with a const char* conversion operator are evil evil evil. Putting them together… priceless.

Though within these sepia dreams and old memories live viable mechanics which when re-examined and explored anew provide exciting areas of creativity.

Among the games of my youth three digital games stand tall, in order played: Hero’s Quest: So You Want to be a Hero^[1], X-Com: Terror from the Deep and Shadowrun on Sega Megadrive. Now some of these games have seen reboots or attempts at reboots which quite frankly angered fans and often missed the point. Thankfully through Kickstarter for Shadowrun and Firaxis for X-Com these two titles are getting a modern, caring treatment and re-examination.

X-Com Squad Turns

Many people passed over the unique quality of team-based turns. There have been many other games in the tactical genre but few have explored this idea of “I move all my units, then you move all your units”. This concept of moving multiple units simultaneously thus requiring a session of planning which manifests as a massive investment.

As control is taken away from the player, breaking a golden rule to strengthen this mechanic, a high point of tension is created as the plan unfolds.

Of interest to modern designers is that this investment, planning and then tension as you take away control from the player. For a modern interpretation with a different angle I suggest looking at Frozen Synapse. The turns are simultaneous and you plan 5 seconds increments but once again, control is taken away from the player as they watch the consequences of their plans.

Golden Rule Broken: Taking Control Away from the Player

Shadowrun Hacking

Almost a fully fledged game within the game but with deep seated roots in the core gameplay. So often “hacking” or another core ability is thrown in as a tangential mini-game. With limited or no interaction with the core-gameplay other than a binary outcome of success or failure. This follows the premise of not creating a game mode shift for the player and avoiding a development investment in what is essentially a “second game”.

Older games were much bolder in this. In Shadowrun hacking you hunted down better hacking decks with a whole subset of stats which could be upgraded. The camera shifted from isometric to third person over the shoulder with new UI and controls. You built contacts and went on missions to acquire that “better piece of software” or that “underground deck”. Your decker’s point of access, which related to hacking difficulty, is determined by their physical entry point into the system. Individual nodes on the hacking map relate to camera systems or subsystems of the physical security system. Triggering the alert system or disabling it, affect the real world alarm systems.

That massive investment in an alternate game mode layered on top of the primary mode added massive levels of depth to the world and further fleshed out the game. Looking for modern alternatives I was unable to find a good modern execution of this concept.

Golden Rule Broken: Avoid Gameplay Mode Shifts
Alternate Expression: Focus on a consistent experience.

Time Based Gameplay

In Quest for Glory many moments of interaction were determined by time of day or the day of the week. This occasionally meant as a player you were running around waiting for an event to happen or cleaning out the stables to earn some coins and some time. While some modern games have integrated NPCs timetables much more complex than their predecessors, they have been made insignificant by removing their game altering potential and turning them into minor points of flavour.

The depth of gameplay this added to the world was significant. You had to work around a real world. As a side note the fact the game required you to grind some monsters or chores like stable sweeping to earn your coin allowed you to effectively use your downtime. In Skyrim I can wake up a town blacksmith and purchase armour, removing all gameplay impact of the town schedule. This mechanic lives on in many modern games but in our fear of inconveniencing the player it has been neutered. I encourage designers to look at the gameplay affecting elements this play style offers.

Golden Rule Broken: Never Inconvenience the Player
Alternate Expression: Never waste the players time

Mixing it Up

Quest for Glory additionally mixed combat, role playing, adventure elements while providing multiple solution paths gto many problems. Environmental storytelling, usage based levelling and many other elements which have survived into modern design lexicons.

X-Com famously mixed the tactical and geoscape layer in a very complex interwoven gameplay. Though as Brendon Chung‘s GDC talk into his trouble with Atom Zombie Smashers highlighted this is not a simple task. I’ve also already talked about the Shadowrun hacking element as another example. For modern designers still bravely exploring mixing of genre and mechanics you do have to look outside mainstream metric focused development. Though I worry that their lack of polish and budget in many cases restricts their ability to smooth out the seams and truly integrate genres.

Golden Rule Broken: Keep It Simple Stupid (KISS)

Conclusion

As many of these older mechanics show, breaking what we consider a golden rule today can sometimes be key to development of an interesting mechanic. These are just a few picked examples from these games. Many other elements exist in these older games which have been gathering dust.

Many of these older mechanics first came about due to technical limitation and were discarded with the limitation. Our Golden Rules were not forged by the gaming gods but discovered through trial,error and exploration. Some of them may lead to an evolutionary dead end in design, an appendix when no longer needed. I encourage you to mine old games. Not for the IP, nostalgia, or history but for the game. Uncovering the hidden machinery of the past, broken paths and discarded branches of game evolution for old and interesting ideas that can be made new again.

[1] Later renamed to Quest for Glory: So you want to be a Hero to avoid confusion with another game, Hero Quest.

I know how to do TDD for what I do; that is, side-by-side replacement of an internal legacy codebase in a high-level language. What I don’t have experience in is TDD for expansion and maintenance of a huge, high performance, very active, legacy codebase, and specifically the graphics components and the C++ bits.

So if you have experience in these sorts of things, I’d love to hear them.

At this point I’m sticking my neck out as a TDD and unit testing advocate studio-wide, but am reluctant to evangelize too strongly outside of my areas of expertise. I don’t think it’d be fair to the very talented people in those areas, and I also don’t want to be wrong, even if I know I’m right :) So I’d really like to hear about your experiences with TDD and unit testing in the games and graphics space, and on legacy codebases, because people are coming to me with questions and I don’t have good answers. I’d love to give them places to turn, articles to read, people to contact.

Thanks for any help.

• Hierarchical Z-Buffer Occlusion Culling – Generating Occlusion Volumes
• Robust Inside and Outside Solid Voxelization

Lets start with the newest and most important information…

Update 4/13/2012

The project is now hosted on BitBucket, http://bitbucket.org/NickDarnell/oxel/overview feel free to contribute or just follow along :)

It’s a Tool Now!

Download

Oxel 1.0.0 – Win32.zip

Requirements

• .Net 4.0
• VS2010 C++ Runtime

Description

Oxel is a tool for generating occluders – primarily for use with the Hierarchical Z-Buffer method of occlusion culling.  Open an .obj file then go to Build > Voxelize to generate the proxy.  Try it out, let me know what you think.

There are some further improvements that have been made over the method that is laid out in the original Generating Occlusion Volumes post,

• Retriangulation
• Evaluating Occlusion-ness
• Filtering Polygons

Retriangulation

I went back to the CSG article Sander van Rossen and Matthew Baranowski wrote.  I noticed near the end that they recommended retriangulating the final CSG mesh because their library doesn’t handle it.

The easiest method I found to do the retriangulation was to just use David Eberly’s Wild Magic math library which can do it, see here.  Before you perform the retriangulation you need to collect all the external/internal edge loops on each surface and then merge collinear edges before performing the retriangulation.

It’s was definitely worth doing, performing the retriangulation saved about 30% of the triangles.

Evaluating Occlusion-ness

When you’re generating occluder geometry you need to ensure that the volumes you’re adding are useful enough to pay the additional polygon tax they will incur.

Oxel achieves this by measuring the number of pixels written to the color buffer by rendering the original visual mesh into the stencil buffer, and then using the stencil buffer to clip a full screen quad while a “samples passed” hardware query is performed to record the number of pixels written.  This is performed from about 64 different camera angles – then when all the samples are summed together this defines the ground truth occlusion of the mesh.

Knowing that information we can evaluate every new box we plan to add to the final occlusion proxy geometry and ask – Does this increase the coverage enough to make it worth it?.  The default threshold is set to 3%.  If a new volume does not at least cover 3% of new silhouette pixels, we don’t include it in the final occluder mesh.

Filtering Polygons

Not every game needs the occluders to be visible from all angles.  For example most racing games will never have the cars jumping so high they see on top of a building.  In those circumstances you may not want to have the occluders bothering to have polygons on the top.

So the tool offers the ability to remove Top and Bottom polygons from the final occluder mesh after everything has finished being processed.

If you decide to filter out all top polygons they’re all removed.  For bottom surfaces this has to be handled slightly differently, see below -

Imagine a bridge structure or a large over hang on a building.  You wouldn’t want the bottom portion of the occluder to be removed from the overhang since it would allow seeing into and through the occluder – because presumably you wouldn’t have double sided rendering enabled.  So to distinguish between bottom surfaces not meant to ever be seen by the player, and bottom surfaces that may be important to higher up potions of the structure, only bottom surfaces within 1 voxel’s height from the base of the mesh are removed.  (See picture above)

Work Continues

I’ve got some more ideas I need to test out but I wanted to go ahead and cut a version of where I’m at and let others take a look and give me some feedback.

One area I need to investigate next is a better way to generate the boxes.  Currently they are just expanded in all directions equally, but that’s not ideal.  I’m thinking about trying a parallel brute force method that would test all possible boxes at a specific origin point to find the best shaped box to generate from that point.

Oxel: A Tool for Occluder Generation @ nickdarnell.com

These oddities don’t make floating-point math bad, and in many cases these oddities can be ignored. But when you try to simulate the infinite expanse of the real-number line with 32-bit or 64-bit numbers then there will inevitably be places where the abstraction breaks down, and it’s good to know about them.

Some of these facts are useful, and some of them are surprising. You get to decide which is which.

• Adjacent floats (of the same sign) have adjacent integer representations, which makes generating the next (or all) floats trivial
• FLT_MIN is not the smallest positive float (FLT_MIN is the smallest positive normalized float)
• The smallest positive float is 8,388,608 times smaller than FLT_MIN
• FLT_MAX is not the largest positive float (it’s the largest finite float, but the special value infinity is larger)
• 0.1 cannot be exactly represented in a float
• All floats can be exactly represented in decimal
• Over a hundred decimal digits of mantissa are required to exactly show the value of some floats
• 9 decimal digits of mantissa (plus sign and exponent) are sufficient to uniquely identify any float
• The Visual C++ debugger displays floats with 8 mantissa digits
• The integer representation of a float is a piecewise linear approximation of the base-2 logarithm of that float
• You can calculate the base-2 log of an integer by assigning it to a float
• Most float math gives inexact results due to rounding
• The basic IEEE math operations guarantee perfect rounding
• Subtraction of floats with similar values (f2 * 0.5 <= f1 <= f2 * 2.0) gives an exact result, with no rounding
• Subtraction of floats with similar values can result in a loss of virtually all significant figures (even if the result is exact)
• Minor rearrangements in a calculation can take it from catastrophic cancellation to 100% accurate
• Storing elapsed game time in a float is a bad idea
• Comparing floats requires care, especially around zero
• sin(float(pi)) calculates a very accurate approximation to pi-float(pi)
• From 2^24 to 2^31, an int32_t has more precision than a float – in that range an int32_t can hold every value that a float can hold, and millions more
• pow(2.0f, -149) should calculate the smallest denormal float, but with VC++ it generates zero. pow(0.5f, 149) works.
• IEEE float arithmetic guarantees that “if (x != y) return z / (x-y);” will never cause a divide by zero, but this guarantee only applies if denormals are supported
• Denormals have horrible performance on some older hardware, which leads to some developers disabling them
• If x is a floating-point number then “x == x” may return false – if x is a NaN
• Calculations done with higher-precision intermediate values sometimes give more accurate results, sometimes less accurate results, and sometimes just inconsistent results
• Double rounding can lead to inaccurate results, even when doing something as simple as assigning a constant to a float
• You can printf and scanf every positive float in less than fifteen minutes

Do you know of some other surprising or useful aspects of floats? Respond in the comments.

I’ve got the slides up online now, so even though you can’t hear my dulcet tones, you should be able to imagine Morgan Freeman narrating for you as you read the accompanying speaker notes. Except this should be a weird English/American Morgan Freeman, who isn’t half as sexy.

So what’s in the deck?

Learn a variety of battle-hardened tricks and tactics for building teams, keeping them running smoothly, and how to handle your team’s worst enemy: the panic of shipping a game.

Learn:

• How to keep your team together under crunch
• How to help teams not hate the producer, especially if their last one suuuuuuucked
• Avoid crunch entirely – and justify it with research
• Stop large teams across large companies from grinding to a halt where all your games – no matter how geographically disparate your studios are – end up shipping in lockstep
• Why you need meetings, scooters, or both
• Panic leads to blame, blame leads to anger, anger leads to the dark side…
• That our brains have really buggy code/design limitations
• What Sierra’s Carrot is, who invented it, and how amazingly well it works to ship on time, every time. (Warning: singular of data = anecdote).

This slide deck (with, theoretically, audio!* and better jokes than the ones in the speaker notes that I can’t remember because my forebrain was wiped by the adrenaline) should be available in the GDC Vault soon.

The Files

Download a zip file containing the slide deck (in PowerPoint format). Download the free PowerPoint Viewer here.

Next year? I’m thinking of going for the GDC “EGOT”, and doing a talk on game design and how your brain works. Probably called “Sticks and stickiness – Primal Game Design”.

*Note: Audio may have been processed and filtered by GDC staff to not actually sound like an English Morgan Freeman at all. It’s a conspiracy, I tell ya.

And time for a small apology… I’ve been meaning to write an entire series of follow up posts to my previous one on #AltDevBlogADay ($10,000 is the Magic Number), but I’ve not had the time – yet. But I do intend to fix that, so stay tuned.

No sweat. Just partition a normalized range of 0.0 to 1.0 according to each probability and bin the result of a random function to a corresponding subrange. But the search will require O(n) time complexity(here, n is the number of dice faces). Can we do better? Sure. One can do a binary search or similar, then one can get a result of rolling a loaded dice within O(logn) time complexity. How about a constant time complexity? Can we achieve that?

Suprisingly (to me, at least) it’s possible. There is an algorithm called ‘alias method‘ which guarantees O(1) generation cost once an initialization step(i.e. preprocessing) of O(n) has been done. This algorithm is from a paper called “A Linear Algorithm For Generating Random Numbers With a Given Distribution” by Michael Vose.

Original probability distribution(top) and aliased one(bottom)

Let’s suppose a dice with four faces, each of which has a probability of 1/2, 1/3, 1/12 and 1/12. The probability distribution can be visualized as the top diagram above. The gist of the alias method is to transform the diagram to something like the bottom one. The essential point is that each column in the bottom diagram has at most two items. By simulating a fair dice to select a column and then flipping a biased coin, which matches to the probability distribution of in the column, to select between the two, one can get the result. A new item added to the original one in that column is called an alias of it, so the name of the algorithm. Quite neat, isn’t it?

I learned this algorithm from this excellent(though, rather long) article, Darts, Dice, and Coins: Sampling from a Discrete Distribution. So I highly recommend reading it for more kind & thorough explanation of the matter. Especially, you can check there how the alias table can be generated in a linear time.

The author of the article also provides a Java implementation of it here:

http://www.keithschwarz.com/interesting/code/?dir=alias-method

Here is my C++ implementation, which is a basic porting of the Java implementation:

http://ideone.com/gzFzI

(This article has also been posted to my personal blog.)

Background

Let me start with some info on Pop Corny. Pop Corny was designed to be a simple and fun casual game, covering the need for short game session experiences that mobile phone games require.

I self funded, developed, published and initially launched the game as a paid application at the $0.99 tier. Later on, it switched to free and monetizing with IAPs, but bare with me for the details of this decision.

Below are some more technical details of the game:

• Development time: 8 months
• Number of developers: 1
• Outside contractors for sound and most graphics
• Build using a fork of Sylphis3D Game Engine
• Support of all iPhone, iPod touch and iPad devices
• Source code (gamecode + game engine code excluding third party libraries):
  
-----------------------------------------
Language files blank comment code
-----------------------------------------
Lua 52 2631 1026 10415
C/C++ Header 77 1209 1111 3718
C++ 32 731 882 2552
XML 1 1 0 788
Python 9 170 136 630
Bourne Shell 13 70 167 284
Objective C 2 14 14 25
-----------------------------------------
SUM: 186 4826 3336 18412
-----------------------------------------

• Game code accounted for 5983 lines of the above LUA code
• The total source dataset reached up to 0.4 GB
• Resulting “compiled” dataset at 10MB
• Initial price $0.99 (50 days) then dropped to free
• Total sales in two months: ~150000
• IAP conversion rate: ~2.0%
• Ratings: 4.5/5 (from 600 ratings, 70% of which are 5 star)

Promotion

As I said the game was funded and published without any help from the outside. This easily translates to: no money for marketing. Therefore I had to find ways to make the best I could on the cheap. One of the major aids on that seemed to be the famous launch day.

Everyone developing on the AppStore will tell you that the launch day is one of the most important events in effectively publishing your game. You must do the best in your powers not to fail that day. And this is true. I had published 3 more applications on the AppStore prior to this game and the launch day will give you eye balls costing thousands of dollars for free. This is through the AppStore’s new releases list.

My approach was to combine that visibility with all the “social” publicity I could harvest, doing my best to leap as high as I could on the charts. Then possibly some game journalists would notice and write about it, thus keeping the fire burning. To do that I created a trailer video for the game and one week earlier I created an event on Facebook for the “unveiling of the video trailer”. I invited everyone I could and one week later (on January 31th) I presented the video. That day I also posted the video on every related forum I could. This caused some fuzz but mostly in Greece where most of my facebook friends live in.

The release day was set on February 3rd. I already had approval from Apple and I was already sending out promo codes to game journalists before that. For this I was carefully compiling a list of game review websites and journalists from early on the development cycle. I had these ordered by alexa ranking by review scores and game types. When I got the game approved by Apple, I send out all 40 promo codes I had available, and contacted 100 more without a promo code.

Early on the development cycle I took another big decision based on marketing. That was to support OpenFeint in addition to GameCenter. OpenFeint has its own top games, featured games and new releases, which can give some extra exposure to your new and unknown game. I decided that I needed that extra eyeballs, so I supported both social gaming networks.

In short what I did prior to the release day were these:

• Created a “coming soon” video trailer
• Created Facebook page and Twitter page for the game
• Organized a Facebook event for the unveiling of the game video
• Send out all 40 promo codes to game journalists (with all media packs, bells and whistles)
• Wrote posts in “new releases” parts of forums
• Prepared a cross promotion among my other apps
• Supported OpenFeint

Then all I had to do was wait for the launch day and the sweet success… or not…

Apple spilling the milk

With everything set and the game automatically scheduled to launch at the world’s AppStores on February 3rd, an unexpected nightmare started for me on February 2nd (since we live in a globe and in New Zealand it was already “tomorrow”). Of course I was viewing the New Zealand’s AppStore waiting for Pop Corny to appear, and it did. Oh! The Joy! After all these months of hard work it was there… taking its chance! Couldn’t be happier and more stressed at the same time… however the moments of joy ended shortly after I looked closer. Fear filled my heart when I noticed next to my game’s icon a release day of “Jan 31th”. I rushed to the new releases list like a maniac looking for Pop Corny which was nowhere to be found. What happened? Had the app gone live 3 days ago and I didn’t notice? Hardly… the app was not live on any other AppStore. I manually checked every other AppStore… not released…

AppStore entry showing
the wrong release day

I was optimistic, these distributed systems are slow to update and initially things like that might happen. However, Feb 3rd started to dawn in more countries and the same story was repeating. Pop Corny was appearing as a 3 days old app. 3 days when you launch on the AppStore is an eternity. I was out of the front page, before I even got the time to get on it. I tried to contact Apple through the support system but that is easier said than done.

It was then I realised how small and expendable I was. There is absolutely no way to solve a situation like this with a system that works by choosing among 15 nested problem categories before you finally get a text box to write down your problem. No way to resolve it with an one day round trip for getting a reply that was canned and had nothing to do with what I described. No way when you can’t -at least- follow up on the answer you get. I tried again and again for the next days by submitting from the start my problem rephrased in the best way I would… fail.

It was almost 7pm US time when auto-magicaly the release day of Pop Corny rolled to the correct Feb 3rd on all AppStores. However it was too late. Some AppStores were about to go on Feb 5th by then. The icing on the cake however was the reply to my last plea at Apple that told me that they don’t see any problem with the date as it is Feb 3rd. The irony!

The problem also managed to mess up OpenFeint’s releases system. OpenFeint, once you have the app approved, scans the AppStore to see when your game releases. I guess that was the reason Pop Corny didn’t got on OpenFeint’s lists. One week later I contacted them about the problem and they said that somehow the system skipped my app, and that they had to add it manually. They couldn’t find an explanation, but I could…

This way I had zero exposure from the AppStore new releases, resulting in next to zero sales. It was a disaster. Apple should really make up to me for this someday by featuring it for a day or two… I can wait.

Getting a grip and keep walking

The above disaster affected all the AppStores in the world. It resulted in Pop Corny selling 10 units on release day in the US. I know crap-ware that were released the same day and sold more (based on their top paid ranking). It was so, because no one knew of my new released game.

In Greece, however, it was another story. Here is most of my “social network”. People knew about Pop Corny and they were expecting it. Several major local iPhone blogs wrote about it and the game took off. In less than 24 hours Pop Corny was number one top paid application. That was among all applications, not just games. 5 stars reviews started coming and it was heaven. A few days at the top and it was also on the local TV media.

This contrast between Greece and all other countries was to me a great illustration of how publicity is king. You must bring your game in front of the players face. And that is hard. It is the hardest part for a small developer. No one will look at you when you are small. Getting big seems like a self fulfilling prophecy.

From the 150 emails I send to reviewers I only got replies from 10 people. 8 of them kindly suggested I pay some fee and the rest two just told me that they were too busy to bother. By monitoring the downloads on the media pack I could see that just two of them bothered to download it. It is clear that something is wrong here. It seems that the only way to get covered these days is through dollars.

Being the number one top paid application in Greece translates to 200 sales a day (yes it is a small store). This is 100 euros a day after Apple’s cut. Does not sound that bad, but given the universal decay law of the AppStore (I will talk about it next), it was not enough. Meanwhile on all other AppStores the app was doing poorly.

I knew that it was out of my reach to reverse this situation, so I decided to step back, stop desperately trying to push it, and focus on fixing some bugs, implement some new features until the day the game would go free. Going free was in my mind since the development days. The game was actually designed to work as a free game from development day one but the final decision to go paid was taken a few days before the release.

I saw this as a second chance. My means of getting more eyeballs on the game. But I knew that going free is not easy. It is a whole different arena. Going free will give you a big sales boost mainly because of applications like AppShoper. These kind of apps scan the AppStore every day and report to its users which apps has gone free, resulting in many sales. However many is not enough when competing with free applications. You need LOTS more.

There are services out there like OpenFeint’s “Free Game Of The Day” that provide you with the extra visibility you need for the day you go free. They feature one game every day, and promote it through their website and their iPhone app. I contacted them, they liked the game and I took the first slot that was available, that happened to be Thursday, Match 22nd (yeap the day Angry Birds Space came out… no fear!). They are very well organized and the process of submitting artwork was very streamlined and efficient. I submitted the appropriate banners and texts and there was nothing more left to do but wait for the day.

The day came and Pop Corny became free. The effect of this promotion was overwhelming. Pop Corny started to climb the charts all over the world at incredible speeds. It was not much later that it was ranking in the top 10 of more than 20 AppStores, and being number one of its category in 10 AppStores.

At startup the game checks for a .plist file on my server that contains possible news to present to the user. This allowed me to measure the rate of downloads in realtime. It reached a whopping 48 downloads/min. A few hours in the promotion 148apps.com featured it and even more downloads happened. This continued for the whole day (which is actually 48 hours to cover all the globe) and I had the opportunity to correlate the spikes I saw on the news file consumption with mentions on twitter, and posts on blogs, etc. A very educative experience.

With the game on so many charts it would be stupid to just turn to paid after the promotion day, switch charts, and lose the exposure. Also the revenue from the free version was already a multiple of the revenue with the paid. So I took the decision to keep it free. I believe I did the right thing. The game did 70000 downloads in the weekend following the promotion day, received very good reviews from the users and the revenue from the IAP was climbing. I was totally satisfied.

The universal decay law of the AppStore

One of the biggest myths busted by my experience with launching Pop Corny was that visibility on the AppStore is king. It seems like everyone believes that being on the top 10 is the way to sell. That this will translate to visibility and sales in a feedback process that will sustain enough to make a profit. You must do all you can to enter the charts as a way to make your game sell more. For me now, this is the biggest misconception about the AppStore.
With the huge amount of applications in the AppStore and the new ones that enter everyday, the charts only act as a capacitor. They smooth out spikes. Most sales today start outside the AppStore and not inside. The people that will buy your application because they saw it on the charts are not enough to sustain it on its position, let alone to lift it. Therefore we witness “the universal decay law of the AppStore” as I call it. Any application that is not receiving some external to the AppStore’s charts force will follow the graph below:

Let that be paid or free. The only thing that changes is the absolute amount of downloads, which is orders of magnitude larger when free. How fast is the decay depends on the competition, how much you icon and descriptions attracts buyers and what rating you have.

Bottom line is that the charts cannot sustain you these days. It probably did some years ago, but not anymore. You should never think of the charts as your means to sales. Use it just as an indicator of how well you are doing. The game is played out of this field. Get reviews, tweets, mentions, and of course get featured by Apple.

Conclusion

The AppStore is not an easy place to publish in 2012. I was not expecting it to be easy, since Pop Corny was not my first application published there. However I would really love not to have been squashed by that bug with the release day. At least it would make me feel much better if I knew I could efficiently communicate the problem to Apple.

In the end however I think that Pop Corny did very well and will do better with time. I learned a lot through the process, it has almost covered its development costs and also gave me the joy of having hundred of thousands of people playing my game. With a zero marketing budget and being my first game, it certainly exceeded my expectations. It proved that you can still have a chance on the AppStore without some big dollar marketing campaign. However it is getting really hard to do so.

Understanding

It seems to me, from the way the petition is worded, that despite investing hundreds of hours in the trilogy many gamers don’t understand the logistics involved in creating a videogame.  It could be argued that they shouldn’t need to – we watch films without knowing all the intricacies involved in getting the image and sound to the screen, so why should gamers need to know about the blood, sweat and tears that go into making games?

While some films do spark huge reaction towards the director from fans, those fans often find other outlets for their reaction, with alternative endings turning up on youtube or comic strips or even novels.  There are rarely the tools made available for alternative endings to be made for videogames, which in turn leads to fans putting pressure on developers.  Funding issues aside, there’s a big reason why a game driven by content cannot have infinite endings: it’s simply not possible to create and test that much content.

To expand on that statement, the way the Mass Effect trilogy melds the experience to the player’s actions puts a phenomenal sense of power and creation into their hands, and this naturally gives the impression that the game is crafted specifically to them.  Unfortunately, it’s not.

Exhibit A

To give an example of how a content driven game is designed I’m going to have a quick look at Dragon’s Lair. If you haven’t played it, the premise is very simple: a beautiful animation plays out and, at a pre-defined point, something happens which the player needs to react to with a button press. This is a video of the full game being played all the way through:

The playthrough doesn’t show many of the player deaths because the player knows what they’re doing. But behind the smooth flowing animation is a system that’s waiting for a player input. If the player misses the window of opportunity the game branches, and the player dies.  If they hit the window, they progress.

Different companies refer to these as different things – nodes, branches, gates.  They’re effectively all the same and, for this post, I’ll stick with the latter.  This example is the simplest gate required to create player choice.  Each and every sequence has been hand crafted, and the game planned out in intimate detail.

To design Dragon’s Lair and sketch out the flow it probably took a few weeks, though actually creating the content took 7 months and 1 million dollars (source).  That was in 1983, so in today’s money that’s a lot more.  And all this for a game that can be completed in just over 10 minutes, as shown in the above video.

Testing

As well as creating the content and code, it all needs to be tested.  Test plans come later down the development line, when the framework of the game and the main mechanics are close to being locked.  Once that happens it’s possible to create a list of tests and expected results.  From here the test team can do regular checks to ensure the game is behaving as intended.

A test plan for the the diagram above might look something like this:

One sequence with a single gate therefore might lead to eight tests.  Possibly even more.  For now though that’s enough to highlight why testing is one of the most difficult parts of making games.  It stands to reason therefore that in a game more complicated than Dragon’s Lair there’s going to be far more possible outcomes.  At any one point the player could have done one of a hundred things, let alone the thousands of small inputs they’ve made to get to where they are.  If a bug surfaces the developer then needs to find a way to recreate it so that they can understand what’s going wrong and then, hopefully, fix it.

I should note at this point that something like a shooting mechanic is very different from gated content, because that’s a system that has a set of defined rules.  When the player does certain things (such as presses the fire button) the rules determine what happens, not the content.  So the test plan doesn’t have to account for pressing the fire button at every possible position the player can get themselves into.  The sort of gates I’m describing here really only come into play at big story points, such as the ‘choose route’ type option presented to the player in Gears of War for instance.

Scaling up

Now we know the level of detail required for a single gate, we should now look at how we cope with multiple gates.  In Mass Effect each conversation often has a few possible outcomes, especially when they’re with key characters.  The interaction required by the player is very different from the reaction-heavy animation window system employed by Dragon’s Lair, but the underlying system is very similar:

Suddenly we’re facing a bit of a crisis – one conversation with three possible outcomes, each with just two more attached to them has led to six unique paths.  And the game design wants over a hundred conversations…  There’s no real way that we can create that many unique gates.  So, instead, we create paths that feed back into themselves further down the line:

You’ll notice that by using this system it’s entirely possible that the player won’t have to go through every gate to get to the end – in fact it’s desirably to do this to try and encourage replays.

Choke Points

You can keep even more control over the number of possible outcomes by creating larger, single, gates that the player has to go through to complete the game, but where they’re free to choose whatever route they desire in between:

The orange boxes in this case are choke points.

With a game like Mass Effect there are multiple choke points, but the entire game is constructed of so many that the player doesn’t have to go through every one to get to the end of the game.  In fact, there’s very purposefully a great deal of routes through the game.  But every single route has been planned, designed and has had content created for it.

This type of design method isn’t just limited to stories, but can be used for level design or ability trees too.  A good example of a game that uses this method for all of these systems is Bioshock. Several levels give you objectives that you must complete to ultimately move forward, but you can do those objectives in whatever order you desire.  Equally the story feels to develop around you, the player.  It’s up to you whether or not you delve deeper into the myth and seek out the audio diaries, and of course there’s the choice of how to deal with the little sisters.  Layered on top of that are you abilities and upgrades, all at your choice.  It works really well, all the way up until the point where you meet Ryan.  At that point all players must go through a single choke point with only one outcome.  Thus their choice is removed.  My protest with Bioshock was simply to stop playing – I still have no idea how it actually ends.

Mass Effect 3 has obviously created a similar reaction, though a much stronger one than my silent protest.  Throughout the trilogy players have been making what they believed were unique decisions, and the games responded accordingly.  The achievement by Bioware is staggering, and from an external perspective very inspirational.  Logistically though, there could never be enough endings to satisfy every possible outcome, and hopefully this post has gone some way to showing why not.

Disclaimer: I don’t work for EA or Bioware, and wasn’t involved in the creation of Mass Effect at any point.

Today I want to discuss a specific form of hacking and how it’s done. I hesitate to do so, but it’s usage is already widespread amongst hack creators and users. The damage is already dealt. By sharing knowledge of its inner workings hopefully that damage can be mitigated.

DirectX Interception

Direct X interception is a particularly naughty breed of hacking. It works by intercepting all calls from a game to Direct X [1]. The intermediate process can then do whatever it wants with that information. Most of the time the call will be passed directly onto the real DirectX without modification.

However in special cases you can do something clever. If the render call is for an enemy player then you can determine its screen position. In many cases that player may be behind world geometry and will either fail z-buffer checks or be covered by subsequent draw calls. Too late, the enemy screen location is known. At the end of the frame you can create a new draw call to DirectX to draw a box around that screen location. Boom, that’s a wallhack.

This type of wallhack is rampant. Even for the most popular of games. Here’s what it looks like for Battlefield 3.

This exists for every popular PC shooter you can imagine. All Battlefields, all Call of Duties, all Source engine games, all Unreal Engine games, etc.

The next step is to simulate mouse input by sending input messages to the app. With a known crosshair location and known screenspace enemy location this is easy. Bam, that’s an aimbot.

Flagrant aimbots can instantly snap from enemy to enemy. To avoid trivial detection the simulated input can be smoothed over time. Even more realistic aimbots overshoot the target intentionally before narrowing in. Other variations don’t move the crosshairs but do auto-fire when they are over an enemy target.

Coolest AI Ever

Wallhacks and aimbots are pretty wicked, but it’s just the beginning. What would happen if someone turned it up to 11? Matthew Fisher, a grad student at Stanford, did exactly this, and it’s the coolest thing ever.

Using the basic methods described above he wrote an AI that can play Starcraft 2. The camera jumps around so fast it’s impossible to keep up with. Please watch a few seconds of the video just to get an idea of what’s going on.

Let’s break it down.

1. Intercept Direct X calls to determine exactly what units are visible and where they are.
2. With full knowledge of screen units pick an AI action to perform.
3. Send input commands to game to select units.
4. Send input commands to game to issue orders to selected units.
5. Move camera.
6. Goto 1.

The automated player is driven by simulated mouse input which requires a single frame to process. This prevents it from running at ludicrous warp speed, but as the video demonstrates it can run the entire loop several times per second. Pro Starcraft gamers play the game around 300 actions per minute (APM). The automated player could theoretically hit about 3,600 APM.

Coolest. AI. Ever.

Matt wrote a large article on this player and his version of a DirectX interceptor. I highly recommend reading it here.

Processing the Frame

Let’s go back for a moment and explain away some hand wavey magic. How exactly do you determine what a DirectX draw calls are for an enemy unit? By careful examination of what’s being rendered of course!

If you’ve ever used PIX, or something similar, then you’ve seen a view of the screen as it’s being rendered one call after another. By examining the vertex count, textures, shaders, etc you can figure out a lot.

Matt shows a breakdown of how he does this for Starcraft 2 so let’s take a look here. Scroll down a bit and check out step 18. Upon visual inspection it’s obvious that a Void Ray unit is rendered. You can record what data was used in that call and use it to determine the screen location of all void rays in the future. Voila!

Changing Calls

I’ve discussed intercepting calls to determine unit locations. You can also add calls to render wall hacks. Finally, you can modify calls. This could be done to change textures or models.

For example an advantage can be gained by replacing player textures with bright red making enemies stand out. In a shooter you might as well go full wallhack so it’s more useful in other genres. [2]

Cheat Prevention

So how can developers prevent this sort of cheating? Unfortunately it’s not easy and there are no silver bullets. That such cheats continue to exist for games such as Battlefield, Call of Duty, and Team Fortress 2 should indicate the complexity of the situation. There are however a variety of strategies that can help.

Game data can be modified to break cheating tools. If the tool relies on vertex count then modifying the data could break the tool. Hackers are of course exceptionally fast at updating their tools and can do so faster than devs. Run-time dynamic modification may help but high poly player models will always stand out. Breaking meshes into smaller pieces with random jitter could help hide them but there are both logistical and performance issues to consider.

It’s possible to retrieve the final frame buffer and analyze it for illegitimate additions. This is unreliable and is likely slow. I have heard tales of games doing this only for the cheating tools to detect the check and turn themselves off for a frame.

Anti-virus like process detection is another option. Punkbuster attempts this but it does not appear to work well. Blizzard’s proprietary Warden software performs similar functionality and seems to be effective.

Player data analysis is always useful. The Starcraft 2 AI is exceedingly cool but would never survive in the wild. It’s not difficult to spot and ban a 3600 apm player. Pinpointing wall hackers in a shooter is more challenging but is extremely viable. You should definitely be gathering and analyzing many data points.

Some companies have attempted to utilize the legal system to shut down cheat creators. This may work within the United States, but many cheats comes from Eastern Europe or Russia. There is no chance for lawsuits to halt their operations.

IP addresses change and mac addresses can be spoofed, but they can still be useful. If you have a known cheater and those addresses match up to a second account then it may be worth investigating. It’s not quite cheating, but League of Legends uses them to help detect smurf accounts. If there is a max level account followed by a level 1 account playing extremely well then you can quickly increase the matchmaking skill rating for the level 1 account to protect the real newbs.

Cheat Detection

The fact of the matter is that wallhacks and aimbots will always exist on PC. You can’t stop them. The cat and mouse game between developers and hackers will forever be played, but it’s not the end game.

Preventing players from cheating is important. Detecting when players cheat is possibly even more important. A common strategy is once cheaters are detected to let them keep playing for a few weeks. Then you drop the banhammer on thousands of players at once. From that point on players will hesitate before cheating because the player will never know if the dev knows.

There are companies that don’t care about preventing cheats if they are rarely used. What they care about is detecting those rarely used cheats and immediately banning the users. It’s a different and non-obvious mindset that changes how you attack the problem.

For example, Blizzard does not validate all player movement within World of Warcraft. There are too many players and it requires too much work to check every single position update for every single player. With some bit twiddling it’s possible to fly, run super fast, no clip, and even teleport.

The position update itself isn’t checked, but suspicious activity can still be detected. Entering unreachable regions, visiting multiple zones impossibly fast, positions inside geometry, player reports, etc. Once suspicious activity is detected the account can be auto-flagged and then expensive validation checks can be performed for only a handful of players.

Here’s an amusing video where a WoW player teleport hacks for a mere 2 minutes before getting banned. In the comments he claims the ban wasn’t due to the teleport but from the mining of ore with a bad z-axis value. The teleport itself wasn’t prevented but other more subtle behavior led to an impressively quick ban.

Another Approach

Do you know what one of Microsoft’s top anti-cheat measures on Xbox is? Achievements. Players are incredibly invested in their profiles and achievement scores. It takes years to build up and most players won’t dare risk losing it all by cheating.

How many players are willing to risk max level characters, collections of hats, and precious achievement scores just to cheat a little? Not many. At least not after the first ban wave.

Wizardry for Good

It’s worth noting that DirectX interception isn’t just for evil deeds. If you’ve played any games on Steam then you are likely familiar with their in-game overlay accessed via Shift-Tab. This overlay works exactly like the wallhack! Steam detects the Shift-Tab keystroke, hooks into DirectX calls, and renders the overlay on top of the actual game automagically. That’s pretty slick if you think about it.

Conclusion

This has been a brief summary on cheating by means of Direct X interception. It can be used in all sorts of nasty ways and if you have a popular multiplayer game it’s probably being used. It’s hard to stop and it can be hard to detect but it’s important to know about and understand. Keep fighting the good fight my friends.

Footnotes

1. This method works perfectly with OpenGL as well. I don’t mean to call out Direct X specifically but do so for reader clarity. Sorry Microsoft!
2. In some games, including the Counter-Strike example, this can be done more easily with a simple data file replacement.

First, I’d like to say I understand some fan disappointment.  Without going into spoiler territory, Mass Effect has always been about choices and the choices the player makes throughout the game brings with it a very personal connection to the characters and events as they play out.  It’s the interactive nature of our media that differentiates the consumer experience of videogames from that of other media such as movies or television.  Bioware has never been shy about discussing and promoting the impact of the moral choices in their games, however those familiar with the series know that when it came to player decisions versus main plot points, plot points won out every time.  This should not have surprised anyone when it came down to the series conclusion.  Criticisms regarding plot holes and the lack of a satisfying ending may be warranted, but the fan reaction seems completely out of proportion.  We’ve all been disappointed by the ending of something or another and the more complicated a plot, the harder it is to wrap everything up into a neat little package.  This may come as a shock to some fans but you can’t always get what you want.

Good stories have a beginning, middle and end.  Not “ends.”  I feel it is unrealistic to expect writers to create a strong, complex plot and then be expected to create multiple, satisfying endings.  I’m happy with one satisfying ending but many would argue that Mass Effect didn’t even give players that.  Many also think the ending was just fine.  I feel the gamers being the most vocal need to ask themselves, what would actually make you happy?  If there was an update tomorrow and when you replayed the game the ending was completely different, would that erase the memory of the original conclusion you received and leave you satisfied?   If the new endings still didn’t satisfy a certain amount of fans would you demand they do it again?  What are the rules on how many fans need to be vocally dissatisfied with an ending before you feel a company should be forced to change their artistic vision for a game?  What consideration is given, if any, to those happy with the current ending and what gives you the right to change the experience for those gamers?

You can’t please everyone all the time.  Whenever anything becomes popular, there will always be a subset of fans that find a reason to hate simply because they can.  There is no pleasing them.  It doesn’t matter if it’s videogames, music, movies or television you’ll always have a vocal minority that will hate what you’ve created just for the sake of hating, especially if something becomes popular.  We saw fans turn on Rockstar after their masterpiece Grand Theft Auto 4 was released; we’ve seen a backlash against Infinity Ward despite consistently shipping an excellent product.  We’re seeing it now with Bioware.  To cave into these vocal fanatics sets a dangerous precedent and takes creative control away from the artists to see their creative visions realized.  I firmly hope Bioware doesn’t touch the ending.  All it does is teach the entitled among us that if they scream and cry long enough and loud enough, they can get their way and as a parent I can attest, that’s a recipe for future disaster.  Give in and it will never end.   Bioware needs to stand by their work and continue their strong stand against those targeting their staff directly.  If gamers don’t like it, let them vote with their wallets.

Fans have often influenced their favorite series and if a writer wishes to remain popular it is important to listen to fans to an extent, but ultimate creative control needs to reside with creators.  Sherlock Holmes appeared to be famously killed off only to be resurrected at a later point due to fan demands.  Spock met his heroic end in Star Trek 2, only to be brought kicking and screaming back from the dead in the terrible, terrible Star Trek 3 but some Mass Effect fans are asking for a complete do-over of the ending.  Bioware has commented that they were willing to possibly have loose ends addressed in upcoming DLC (and I support this move as should fans) but imagine Star Trek 2 edited so Spock makes it out alive and pops out of his coffin at the end to yell  “Hiyoooo!” because it would be more pleasing.  Expecting Bioware to simply rewrite their ending is carpet chewing mad and Bioware shouldn’t even entertain the idea.  Already we see Hollywood taking fewer and fewer risks with story-telling trying to cater to everyone.  The end result is often bland and generic.  This happens to an extent currently in the games industry with developers and publishers erring on the side of caution in an attempt to protect the massive investment associated with developing big titles.  Gamers always rally against this effect but here the message Mass Effect gamers are sending resoundingly to the game industry is “Don’t take risks and don’t upset us or we’ll turn on you in a heartbeat.”

I have your new ending to Mass Effect 3 right here.

I’ve heard others draw parallels with Stephen King’s Misery where a crazed fan holds a writer captive and submits him to numerous tortures because she’s unhappy with his recent book’s ending.  She forces him to write a follow-up novel bringing the main character back to life to continue the story.  I don’t think this is a fair comparison.  Even in her madness, Annie Wilkes never expected the author to rewrite history and change the ending as some Mass Effect fans are demanding.   For the good of games as a truly creative medium, this movement needs to be squashed.  #ChangeTheEnding

Let us know your thoughts on the results. We’re very interested in feedback on how to make #AltDevPanel better in the future.

Thanks again to Google for allowing us early access to On Air Hangouts!

*WARNING: SPOILERS*

If there’s a group of people on this planet that have mastered the art of storytelling, it’s John Lasseter and the crew at Pixar, and I firmly believe that opening to Up represents the pinnacle of their storytelling abilities. In the first 10 minutes of the film we learn everything we need to need to know about about Carl and Ellie’s relationship without a single word of dialogue being spoken. In a matter of minutes we see Carl and Ellie go from a young couple in love, to a couple devastated by an inability to start a family, to them overcoming their grief and moving on with their lives, and the pain Carl goes through as Ellie passes away. We know from the opening montage that despite the gruff exterior Carl displays in beginning, he is a loyal and caring man to those who are closest to him. We understand why he went to such extreme lengths to journey to Paradise Falls and what reaching his destination really means to him, and we understand the full emotional gravitas of his decision to sacrifice his journey in favor of saving Russel at the end of the film.

On the other end of the emotional spectrum is Ruin, which unlike Up isn’t trying to tell an emotional story, so that makes its job a little easier. Like the intro to Up, there isn’t a single word spoken throughout the entire course of short, but we learn everything we need to about the world the story is set in. From the opening establishing shots we see Ruin is set in a future where nature has begun to reclaim what was built by humans and has been at it for a t least a few decades. We’re then given a brief taste of how far humanity had progressed before its downfall with the introduction of the main character. We see him capable of creating a link with certain types of technology and controlling said technology with a mere thought. We’re shown brief hints of the possible downfall of man through quick flashes of a warning about a quarantine. Without a single word being uttered, we know everything we would need to know about this world.

Now, these are just two examples of the “Show, Don’t Tell” principle, and for every good example there are just as many, if not more, bad ones. I will be the first to admit that it far easier to apply this principle in a visual medium, such as animation and film, than it is in literature, but the principle still applies none-the-less. By showing your audience what you’re trying to say you help create a richer and deeper experience that will stay with with them long after they’ve finished watching, reading, or playing.

This is a good thing, isn’t it? So why do I hate TDD?

Because debugging is fun. There, I said it. I love debugging. I think lots of clever people like debugging. I love someone having a problem, coming to me, looking at it together, getting up to walk around, look at the ceiling, talk to myself, stand in front of a whiteboard, draw some lines that spark some idea, try it, manually test a fix out, slouch down in my chair staring at my computer lost in thought, and repeating this until I actually find and fix the problem. Not just think I fixed it, but really sure that I fixed it because suddenly it all makes sense. At which point I spring a terrific boner for my obviously superior brain power that was able to find this problem that plagued mere mortals.

So the sad fact is, since I’ve been doing TDD, I haven’t been able to go on this ego-trip a single time in our TDD-written codebase. Sure, sometimes we get a bug in the UI, but those usually manifest easily. Oh, and I’ve definitely used some frameworks and API’s improperly and caused bugs because of that. But those are the annoying types of debugging we all have to do, not the  magical mystery tour described above.

I didn’t realize how much I missed debugging until it was gone. Fortunately, there’s still lots of legacy code and API’s to get my fix from.

A rapidly growing market
It’s no secret that there’s been a ton of interest in social casino games in the past 6 months. Two social casino game companies, Playtika and DoubleDown, were recently purchased at massive, 9-figure valuations. Everyone from EA to Zynga are getting in on what is expected to be the next hot social game genre. Kontagent also commented that their fastest-growing customer segment has been social and mobile casino games. But what does this all mean for game developers looking to get into the space?

First of all, if you’re not sitting on a pile of cash, don’t bother looking at Facebook. The viral channels that made Facebook a great platform for indie game developers are dead, says Dave. While this is true for Facebook in particular, competition in the social casino games genre has driven up CPA prices on all platforms. You should expect to be at war from day 1. Many newcomers are looking at cross-platform development solutions so they can spread their eggs across multiple baskets.

When being born into wartime, it’s important to pick your beachhead early. For poker, skill-based and sports-based games, Dave recommends targeting a young male audience. For slots and other chance-based games, the audience remains primarily older females (typically 65% of slot players are female according to Dave) but is starting to drift closer to an even gender ratio. This drift is due to the changing nature of how slots are presented to the player: men can try the game for free, play for home, and play online. These changes make slots-type games more appealing to a broader audience.

Furthermore, these changes make casino-style games more appealing to a broader audience across the board. If you look at the difference in growth curves between FarmVille style games and social casino games, casino games show more consistent, wider growth curves. This shows that casino games keep players engaged longer and are less reliant on a huge launch for success.

Lastly, Dave warned social game developers that real-money gambling games are going to be launched in many if not all major platforms in the next year. Facebook is already looking into allowing real-money gambling in the UK, and the few mobile gambling apps that already inhabit the Apple and Google app stores are experiencing meteoric growth. Real-money gambling games are going to be tough to compete against because these companies can afford a much higher CPA than virtual currency game developers. However, real-money gambling companies come from a world where each player is a paying player, and aren’t as familiar with the freemium model. If you’re intelligent with your user acquisition and optimize your free-to-paid conversion, you can compete with these giants.. for now.

SlotSpot Case Study: Build, learn, iterate
Blitzoo built SlotSpot in just 6 weeks with a single-minded focus on a Minimum Viable Product, or MVP. The game had no friends bar (a staple of Facebook games), no quests, no gifts, no achievements, and only 3 slot machines to play. The game did have levels and XP, but there was no reward for levelling up. Instead, Blitzoo focused on cranking out new features each week to fill in the gaps in their product. By cultivating customer feedback via their Facebook page, they were able to quickly prioritize the most important features on their product roadmap. Then, through live A/B testing of their game, they were able to test these assumptions about what features the audience was really interested in. All in all, Blitzoo used Eric Ries’ Lean Startup methodology to impressive effect with their first social game (more on this in a later post).

When building his MVP product, Dave highlighted the importance of analytics and metrics. You need metrics for every piece of the game business:

User Acquisition. You need to segment your inbound users and see which creative, which demographics and which countries converted most effectively.

Free-to-paid Conversion. Which trigger got players to purchase? Which offer got that finicky player to finally buy? What creative was most effective?

Game Balance. What is the average session length? How many spins does it take until they are out of money? How does this change as they level up?

Player Temperature. This was Blitzoo’s own measure of positive vs. negative player feedback.

As you are building your game, you should always be asking yourself: what do I need to know? The answer will help you determine the metrics that are most important to your business.

SlotSpot Case Study: Maximizing revenue
When looking to maximize your revenue, the first step you should take is to identify your high value players. For the social casino genre, there are two types of high value players: whales and evangelists. Whales are the players that spend the most, typically spending over 10x more than the typical ARPPU rate of your game. These players can spend over $1,000 per game and make up a substantial portion of your game’s revenue. Evangelists are players that love your game. They invite their friends, are active on forums, and give you valuable feedback. Catering to these two groups is of utmost importance for any social casino game.

To appeal to these high value customers, you should segment them internally via your analytics and present them with unique offers. You should also track their playing habits and retention so that you can optimize your game to keep these players around. When dealing with their support or feature requests, take a little more time to write a custom response. Small tweaks like these can create the best experience for these players and keep them coming back.

Your second step when maximizing revenue is a no brainer: maximize your revenue from all of your players. Run A/B Tests and experiments on players, using the Lean Startup as a framework. Notice how in the graph above, the revenue spikes get larger and larger. This is because the A/B tests are improving the promotions’ effectiveness over time.

When running these experiments, be sure to segment each test by player type, whether it’s a whale, evangelist, first-time buyer or someone who has never purchased. Dave says that 90% of players on Facebook never pay, 5% will typically pay, and 5% will maybe pay. The key to maximizing the amount of players that do pay is by running these experiments. Test different offers, such as coin bundles, sales or referral promotions, to see which copy performs better. Be sure you test one thing with each test: the copy, the collateral, or the offer itself. However, never test yourself into having only one “optimal” offer. Dave’s advice is that a variety of offers always performs better than one “best” offer. Also, be sure to vary your delivery method of the offers, whether it’s from an interstitial ad, a banner ad, or an in-game graphic.

SlotSpot Case Study: Analytics as an immune system
The last key use for analytics is as an early warning system for bugs or problems with your game. Use your analytics tool to track errors, customer service requests, ARPPU, free-to-paid conversion, virality, average bet size, and anything else that is a mission-critical function of the game. If any of these numbers spike or drop dramatically, your canary is dead and it’s time to troubleshoot the coal mine. For example, Blitzoo had a problem where their overall revenue suddenly dropped. From their analytics, they could see that they had seen a significant increase in free-to-paid conversion rate, but an even larger decrease in ARPPU. It turned out that a promotion created by their marketing team was too aggressive, and undercut their price significantly. Once the error was spotted, fixing it was simple and the situation was resolved (damn it, marketing! :P).

Mobile: The Next Frontier
To conclude his presentation, Dave talked about Blitzoo’s upcoming transition to mobile and how they planned to adapt SlotSpot to the new space. He reiterated analytics’ importance on mobile because there are even more factors to a mobile player than a social player. There is no “single solution” for social sharing on mobile like there is on Facebook, so you need to incorporate and track a variety of them. Also, customer acquisition is done on a per-deal basis rather than run all through Facebook, so there’s a lot of segmentation there as well. Finally, customer onboarding is incredibly important to building a successful game on mobile and A/B tests are the key to optimizing your initial onboarding flow.

Dave pointed out that you simply cannot port a social network game to a mobile platform and expect the same results. For one, there’s a much shorter session length: mobile players spend 3 minutes per session on average while social players can spend up to 20 minutes. This requires a complete overhaul of game balance and the willingness to take an ax to your feature set. Offers are responded to differently on mobile, so you will need to start your in-game marketing testing all over again. Lastly, engagement is king on mobile. It’s much harder to keep players engaged with a mobile game and re-engage players that have been lost. Making all of these adjustments has been key to Blitzoo’s preparation for the mobile launch of their SlotSpots app.

Wrapping up
Thanks again to Blitzoo and Kontagent for throwing a great webinar. I’d recommend getting on their webinar circuit by signing up for their mailing list, each one I attend keeps getting better.

The social casino space is on fire right now, and one can only expect the fire to spread to mobile. If you’re looking to get into this space, move quickly and buckle up, because it’s about to take off. And while you’re busy fending off incumbents, trying to keep distance from the newcomer nipping at your heels, and avoiding the real-money giants, you might want to look into Betable. We’re the first and only platform that lets you add real-money gambling to games, and it might be the weapon you need to win in the social casino game space.

I recently started a weekly blog series on our company blog called “Game Face”. It is “our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that.” This is important for our company and I think it should be for your company.

PUTTING ON THE GAME FACE

At the beginning of 2012, I did an assessment of all the projects we open sourced in 2011. Late much? 22 projects wasn’t bad and I want to beat that number in 2012, but it got me thinking that this kind of information is timely and needs more regular attention. So, I started kicking around ideas for a blog series name. After a day, I settled on “Game Face”. “Game” obviously referred to the work we do in developing gaming-related libraries and middleware. “Face” would refer to the code-wise public face of the company and the developer(s) working on the individual libraries.

It will help you become a better writer: If you’re a developer and you’re scared about writing a weekly blog series, you shouldn’t be. The focus of the blog is technical in nature, so you don’t need to spend a lot of time on exposition. If you’re good with your commit messages or keeping a CHANGELOG, the blog posts write themselves. For example, from an item in a recent blog post, “This releases addresses the first future idea from the README when the gem was released over a year ago to add a method allowing for bulk insert of data into a leaderboard.” Cut and paste for the most part my friends. The intrepid developer might even automate the creation of the initial blog post that can be wordsmithed by someone you deem more fluent in languages other than C++ :)

It will help you become a better developer: We get feedback on our code each day from our co-workers. By opening up your code to the world, you get peer feedback beyond your immediate peanut gallery. You also get comments, questions and sometimes code where developers are using your code in new and interesting ways that you hadn’t thought of yet. Case-and-point: I recently integrated a patch to our leaderboard library with all the code and tests to allow for leaderboards in “reverse” (lowest-to-highest) sorted order. I just hadn’t come across that use case in the games we’ve worked on where I needed a leaderboard appropriate for a racing game. But someone else did. And now our library is better off because of it. Beyond peer feedback, or any feedback, opening up your code and talking about it helps you to think about the ramifications of changes when there are developers other than you or your company using your code.

It will help you as a company: By highlighting your company’s open source work, it may motivate your developers to want to take a stab at open source in the first place or to clean up a library for external publishing. You might also attract developers who put a high value on knowing their contributions will be recognized, whether they are internal or external, and that their contributions may see the light of day outside of your company’s hallowed halls. The last “Game Face” blog post I wrote highlighted a new library one of our developers had released that allowed you to parse Beersmith2 (beer brewing software) files in Ruby. I imagine many video game companies rely on open source to some degree, and so by publicly promoting your contributions to open source, it can also help you in the eyes of the open source community.

It will help you have awkward conversations: I can’t tell you what to open source and what not to open source. You’ll have to talk as a team, as a company, and possibly with your lawyers to understand what you can and cannot open source. In 2011, there was only one instance where I went to our CEO to get a check on, “Can I open source this?” We walked around the block and our conversation went basically as follows:

Me: “I’d like to open source a leaderboard library. Given this is a core thing we do, is that OK?”

CEO: “So, someone else could do this before us using a similar method?”

Me: “Yes.”

CEO: “Ship it!”

FIN

Hopefully I’ve made some compelling arguments that your company should highlight your internal and external open source development. Our weekly blog post series highlighting our internal and external open source work comes out on Friday and now I usually get one or two messages throughout the week in our group chat room asking, “What’s going in to Game Face this week?” It feels pretty good when I can say back, “Your face.” If your company does anything with open source, I’d love to know about it!

However, in putting together my latest endeavour (a hack-n-slash isometric multi-player RPG in HTML5 canvas), I’ve learned a few more things and want to share them.

TLDR; I’ve pushed the code I’m currently using to Github and it can be found here: https://github.com/robashton/swallow

Anyway – moving on we can discover some of these learnings:

Simply waiting for requested assets to load via HTTP on start-up isn’t always enough

In my earlier post, I suggested if you had a resource caretaker which you requested resources like textures, sounds, models, shaders etc from – then it could take care of loading the data and return promises rather than the real things, this looked something like this:

1
2
3
4
5
6
7
8
9
10
11

var modelOne = resources.find('models/hovercraft.json');
var modelTwo = resources.find('models/missile.json');
var explosionSound = resources.find('sounds/explosion.wav');
 
resources.on('fullyLoaded', function() {
  game.start();
 
  // use the resources
  modelOne.get();
  modelTwo.get();
});

And the world is a happy place to live in, we rely on those assets coming down via HTTP, rely on HTTP caching and everything else given to us and it works well.

However, we also rely on the initial state of the game indicating which resources it is going to need – and doesn’t account for other resources that might be requested once the game is under way (for example, explosions, other models/textures further in the world, sounds, etc).

This results in negative artifacts like ‘popping’, or sounds being played after the special effect has finished (or in bad cases, the player walking on an empty background!)

Therefore the answer is to pre-load, but how…?

The Application Cache

Now, I’m not going to give a full description of this, but essentially you can tell the browser “Hey, these are my resources, please download and cache them”, for example:

1
2
3
4
5
6
7
8
9

CACHE MANIFEST
# 2012-03-11:v1
 
CACHE:
/favicon.ico
game.html
models/hovercraft.json
models/missile.json
sounds/explosion.wav

We can generate this kind of application cache automatically as part of our deploy process by enumerating through the assets, and we can choose not to re-generate it if none of the files have changed.

When we re-generate it, we can set the timestamp (therefore indicating to the browser that because the manifest has changed it might like to go through those assets on the server and see which ones it needs to download).

We also have similar code to write against this, “Update the resources if necessary, wait for this process to complete”

1
2
3

appCache.addEventListener('updateready', function() {
  game.start();
}, false);

Ace.

This has its share of issues though – the biggest one is probably that it doesn’t really allow for any granularity in your app.

In this modern age of the internet, our users have short attention spans and a long initial load may well mean losing out on users – if you have different levels for example, you probably want to download each level and the assets for that level before you play that level – not at the beginning of the whole game.

In short, the Application Cache is great if you have a small self contained game (such as puzzle games), or if you want to fully support the game being playable offline immediately after download;  it does however come with its share of issues.

That brings us onto another option

Write your own Application Cache

We can easily emulate what the AppCache does for us by writing our own manifest definition and using that to pre-load files from the server. We can then rely on the browser-cache to carry on working the way it always has (so if files haven’t changed, don’t fetch them etc).

This is essentially like our first solution again, except we now have manifest files which describe what data needs pre-loading.

1
2
3
4
5
6
7
8
9
10
11
12

resources.preloadFromManifest('levels/levelOne.json');
var modelOne = resources.find('models/hovercraft.json');
var modelTwo = resources.find('models/missile.json');
var explosionSound = resources.find('sounds/explosion.wav');
 
resources.on('fullyLoaded', function() {
  game.start();
 
  // use the resources
  modelOne.get();
  modelTwo.get();
});

This still causes some issues – the browser may choose to not cache some objects, may decide that some things aren’t available offline, and weirdly give us some issues with sounds.

For example: One issue I discovered with sounds in my last Ludum Dare attempt was that you have to create a new Audio object every-time you play a sound – and the browser re-requests the asset from the URL you give it each time that happens!

Another problem with this approach is again it doesn’t scale too well – if you’ve got 500 textures, 500 HTTP requests is not really appropriate at with the current technology stack (Until SPDY or something similar is supported universally at least).

That brings us to where I am at the moment with my hack-n-slash multiplayer canvas RPG…

Bundle everything up into a single file!

How things come full circle – desktop game developers have been inventing and consuming package formats since time began, and now web-game developers can get in on that action too.

So hence writing a command line utility in NodeJS to scan a directory and package various files into a JSON file.

1
2
3
4

.json -> keep it as JSON
.png -> Base64 encode it
.wav -> Base64 encode it
.shader -> add it as a string

This works well, as on the client loading these assets means writing a small amount of code like so:

1
2

var image = new Image();
image.src = "data:image/png;base64," + imageResource.get();

In actuality, what we end up doing is loading an entire asset package, say ‘assets.json’ and writing the following code before loading the game.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$.getJSON('assets.json', function(rawData) {
  preloadAssets(rawData);
});
 
function preloadAssets(rawData) {
    forEachResourceInRawData(rawData, preloadItem);
};
 
function preloadItem(key, itemData) {
  var loader = findLoaderForFiletype(key);
  increaseAwaitCounter();
  loader(key, itemData, decreaseAwaitcounter);
};
 
resources.on('complete', startGame);

Where an implementation of a handler for a PNG might do this

1
2
3
4
5
6

function preloadPng(key, itemData, cb) {
    var preloadedImage = new Image();
    preloadedImage.src = "data:image/png;base64," + itemData;
    preloadedAssets[key] = preloadedImage;
    preloadedImage.onload = cb;
};

This means that once the assets file is loaded, the game can start and there are no delays in playing audio, displaying textures and so on and so forth.

Clearly this is massively simplified, because in the real world yet again what we actually do is.

Do the hybrid approach

• Load the assets for the current land
• Request an asset
• Is the asset in the preloaded package?
• Yes? -> Return a promise containing the asset
• No? -> Return a promise without that asset
• -> Make an HTTP request to get the asset
• -> Cache the asset

For example, my little RPG pre-loads most of the textures and models used across the land, but downloads the actual tile information (where is a tile, what is on that tile) as the player walks around.

This is similar to the streaming that takes places in any reasonable desktop game, and offers a good compromise in a connected multiplayer game (and would work well for a disconnected one too).

I’ll push out the client code I’m using as a library at some point, but that’s not really a suitable candidate for ‘frameworking’, because homogenising something like asset management on the client side can do more harm than good.

I’m happy to announce our first #AltDevPanel! Scheduled for this Saturday at 10PM PDT, we’ll be bringing together some masters of high performance programming to talk about their craft. Our scheduled panelists are Mike Acton, Tony Albrecht, John McCutchan, and Jaymin Kessler.

The optimization panel will focus on optimizations from a micro to a macro level, providing observations on what to do and what not to do when optimizing a code base. It will also delve into how to measure performance, and how to weigh the impact of a particular code change. And finally some discussion on how to practice optimizing so you can do it when it counts.

We’ll be accepting questions related to the panel’s topic prior to the panel. Questions can be posed in the comments below or via Google+ and Twitter using the hash tag #AltDevPanel.

The panel will be broadcast using Google’s On-Air functionality. This requires a Google+ account to participate as an audience member. If you don’t have a Google+ account or aren’t available at the time the panel is set to occur you can still watch the talk at a later time. All talks will be posted to YouTube, and the links will appear on the site shortly after.

We’d like to thank Google for allowing us early access to On-Air. We’d also like to thank the people that helped us with this process, Colt McAnlis and Travis Sanchez. Without their efforts this wouldn’t be possible, so thank you guys.

Panelist Bios

	Mike Acton Engine Director at Insomniac Games (Resistance, Ratchet and Clank for PS3.) Keeper of #AltDevBlogADay
	Tony Albrecht Tony Albrecht is the founder and director of Overbyte, a company that specialises in high performance programming solutions for game companies. He’s built game engines for a range of companies over the last decade or so and now spends his time helping other studios improve their software’s performance.
	John McCutchan John McCutchan is from Kitchener-Waterloo, Ontario, Canada. Wrote inotify for the Linux kernel. Got his M.Sc. in computer science from McMaster University. Lead of the Game Systems team in Developer Support, SCEA. Wrote Move.Me.
	Jaymin Kessler DoD disciple, his datas trifle. He shoot structs from his brain just like a rifle at Q-Games.

Players have all kinds of crazy crap.

When it comes to audio, players be crazy. Not meant as a dis. Just an observation. One person might have a high fidelity 7.1 surround sound system with discrete speakers placed perfectly according to THX standards. The next person might have the same system, but all of the speakers are stacked on top of each other like a tower in the center of their room. And the next person is listening through their old CRT television’s stock speakers. And the next guy is listening on amazing headphones. And the next person is listening on earbuds that they got from a plastic egg after depositing a nickel in one of those vending machines.

We never really know what the heck our player is doing when it comes to their sound system setup. The Volition offices are a microcosm of this. Some people here use professional speakers (Team Audio). Some people use headphones (Team Design). Some people use cheap PC speakers (Team Production). Some people use earbuds (Team We Hate Team Audio).

Again, not meant as a dis. All of these are, of course, perfectly fine means of getting sounds into ears. However, some issues can crop up as a result.

“I can’t hear Sound X. Turn it up.”

Story time! Also, disclaimer time! This is a dramatization. This didn’t happen here at Volition. But I know it has happened elsewhere, and it’s not some isolated incident. Team Audios world-wide will be more than happy to regale you with some variation of this story while sullenly drinking their whiskey and/or pink-colored bismuth.

Once upon a time there was an Audio Designer. Our dear Audio Designer had made Sound X and had implemented it into the game. Audio Designer was happy.

Two weeks later, Audio Designer is called over to Project Producer’s desk. Audio Designer was scared. Nervous. Vulnerable. Naked. Okay, not naked. No, you know what? Naked. May as well make this story extra weird.

Project Producer was frustrated because they could not hear Sound X very well, and demonstrated this through their cheap PC speakers. Naked Audio Designer responded that the reason that Sound X couldn’t be heard very well was because of the crappy sound system that Project Producer was using.

Project Producer frowned. Project Producer rebutted that perhaps Naked Audio Designer shouldn’t be testing sounds on top tier studio speakers and should instead be testing sounds on crappy PC speakers, or television speakers. And why wasn’t Naked Audio Designer wearing clothes? Well, that’s another story altogether, isn’t it? It is.

Well, Naked Audio Designer immediately went home and wept for two weeks in their crawl space and forgot to drink any water, and therefore died, and was of course eaten by rats.

So, why don’t we just test on TV speakers?

I find that the easiest way to answer that question is to refer to our dear friends in Team Art. While it’s possible that Team Art might have to use whatever monitor is in front of them, they tend to want to work on the highest possible quality, perfectly color calibrated monitor. When they work on these monitors, they can assume that the visual quality will be maintained to all the different types of screens.

We’re in a similar situation in audio. We do our work using high end studio speakers (usually called reference monitors, but I tend to use the word speaker to avoid confusion), sometimes with especially flat frequency responses (meaning the speaker itself isn’t changing the sound), and usually in calibrated listening spaces, because that way it should translate better to all the different kinds of speakers that our players have.

If a colleague from production or studio management wants your Team Audio to work with low quality speakers because that’s what most players will be using, point to this article if you want. Maybe then you’ll be able to scam them into getting you nice equipment. Wait, did I just ruin that strategy? Maybe I did. But I’m too lazy to go back and change the word scam to convince. Moving on.

But testing on TVs… that just seems smart.

You make a good point, Blog Heading.

Testing on TVs still seems like a good idea. I’m not trying to make a case against testing your mix on a ridiculously wide range of audio systems. You should probably still do that. And you should still test your game audio on a system that you’re most familiar with. Test it on the system that you play games on. Then capture some audio from your game, burn it to a disc, and listen to it in your car. Test it in the conference room where you do your show n’ tells. Test it on the Jumbotron in your living room. What, doesn’t everyone have a Jumbotron in their living room? Well, then do your best with what you have, I guess.

In my experience, the most important thing to test on a wider variety of sound systems is your mix. If you are struggling to hear the dialogue in your game on your television where you play most of your games, then that’s something that might be worth looking into. Does the music seem too loud when you listen on the headphones you’re most used to? Data point added! Just be careful. If you’re doing a final mix of the game, then you’re probably pretty close to submission. Choose your battles wisely!

Another trend in audio is to give our players a bunch of audio options. We give them discrete volume options to adjust sound, music, and voice, sometimes even more granular than that. We also occasionally give them overarching presets like Hi-Fi, Television, Headphones, etc, which can do things like control how much compression there is on the master mix or on sub-mixes, among other wizardry. This is awesome, but if we’re being thorough, it can put an increased burden on the audio team as well as audio QA.

No time. Who care about most?

Wow, Blog Heading, the way you wrote yourself is really convincing. You even deliberately left out a couple of words. Nice work.

I think the question that the Blog Heading is trying to ask is, when we’re low on time, which is typical in audio land, then who should we be designing audio for? Should it be for the player with the kick ass high end sound system? Or the player with the crappy earbuds?

Your team may have varying opinions on this. Some people or disciplines will suggest that you need to appease the lowest common denominator. Like when you make your PC game playable on an 80486SX, even though you may need to wait two years to play the same game on max settings. So, if that’s the case, then we need to design for earbuds gamer. Or at least crappy Labtec PC speakers gamer.

Well, my current opinion is the opposite. Go for the gamer with the higher end sound system. Why? That gamer probably gives a crap about the sound in the game.

I know, earbuds gamer might care, too. Maybe earbuds gamer just isn’t able to afford a nice sound system. Well, that’s okay, because fortunately for earbuds gamer, we always do our best to accommodate everyone, right? Right!

But the gamer with the higher end sound system is basically begging you to put that system to good use. Higher end sound system gamer probably wants you to do some serious face melting. Why else would they spend all that money? They probably want an aural experience more similar to the (good sounding) movie theaters.

This might be kind of controversial. I don’t really know. I haven’t tested the waters on this one, yet. I guess I should say that this is only my personal opinion, not that of any company that I work for, and it’s just an opinion, and I’m just a dude whose opinion changes, like, all the time. I’m not running for president, here. And if I was, I’d probably make a law that proclaims bologna as the official meat AND undergarment material of the country. Does the president make laws? I can’t remember, but I don’t think so. I think it’s congress, actually. Anyway, just remember, a vote for me is a vote for bologna.

However, over the course of being heavily involved in a variety of volunteer operations, it’s come to my attention that the general understanding of what it means to volunteer may vary from person to person. It’s not just raising your hand and feeling good. These two steps are indeed part of the process, but there’s a lot more to it than that!

The following are general guidelines for volunteering, which I’m hoping may serve as standards to help a variety of organizations, groups, and individuals by better educating budding volunteers on how they can most effectively serve their cause.

 Initial Steps

The first step to volunteering isn’t just saying you’ll do something. If you’re eager to get involved with an effort, please start by:

1. identifying available opportunities
2. evaluating how realistic it is for you to contribute, based on your existing workload, schedule, and abilities

You might begin by asking a professional organization like IGDA how you can help out, and in turn, receive a list of items that could currently use some attention.

Take a moment and actively match yourself to things you know you can accomplish. That isn’t to say you shouldn’t push yourself to grow, but you should be able to fulfill the basic need you’re stepping up for. If you volunteer for something that’s completely beyond your capabilities, the organizer is basically back to square one when this detail is discovered, which works against the overall intent of helping people.

Playing to your strengths and thoughtfully managing your time will overall aid you in becoming an outstanding volunteer.

Following Through

Similarly, it’s extremely important to finish what you’ve started. The bottom line is, someone needed help with X. Therefore, the most useful thing you can do is actually take X from a need to a finished objective.

Offering your time to help an organization, group, or individual should be considered a commitment. When you volunteer, people are now counting on you to pitch in with something!

For example, say I needed someone to make posters for an event. If no one came forward, I would be aware that I had something unasigned and I’d unconsciously operate with the understanding that I need to stretch my resources to cover it. However, if I have volunteers, my organizational thoughts change. I might start pouring more effort into my primary tasks, in the interest of making the event all that much better. Having more hands to help essentially means you can do more!

Unfortunately, when a volunteer bails last minute or otherwise falls short, an organizer suddenly has an unanticipated hole in their plans, and thus, must scramble to shuffle things around and make it right. Be aware that flaking out on something you committed to makes it harder for everyone else involved. Volunteers should strive to be a helpful and accountable.

That being said, it’s understandable that sometimes life happens and things don’t always go according to plan (emergencies, etc.) If something uncontrollable comes up, be sure to let your coordinator or organizer know as soon as possible.

Doing it Well

Keep in mind that volunteering generally results in some form of work–although it can often be quite enjoyable–and thus, your volunteering efforts should be held to a high quality standard. You want to be proud of your work, right? Unless the original objective stated was to “slap something rough together,” treat your task as you would a paid job. If you’re not sure what the expectation is, don’t be afraid to ask! Most coordinators are more than happy to detail out tasks and get you everything you need to accomplish the given goal.

Additionally, in the game development community, always demonstrating that you uphold high quality standards as a volunteer is a great way to build an awesome reputation. In all, it publicly demonstrates that you’re a hard worker. It’s no secret that this is a very close-knit industry and people talk. If they have great things to say, it’s highly beneficial to how others (including potential employers,) may regard you. In contrast, if they have nothing good to say, it can have the opposite effect.

Enjoying the Benefits

So it might sound like a great deal of effort, but in general, volunteering in the game development industry is great for both personal and professional growth.

• You get out there and meet tons of great new people in your field! This is exceptionally useful to networking efforts.
• You often acquire new skills along the way! For example, I learned the logistics behind running an IGF booth last year.
• You feel awesome for contributing to something larger than you could do on your own.
• You continue to nurture the game development community, which is carried entirely by volunteers who are dedicated to their craft and the constant improvement of it.

The Big Picture

My hope is that these tips will help you be the best volunteers you can be, and through teaching each other, we can continue to nurture the quality and reach of our community efforts.

For those of you already out there, doing all of these things and more: thank you so much! You are the amazing force that makes this field so inspiring to work in.

And to all aspiring super-volunteers in the making: go forth and be excellent!

So this week a year ago I was…

• Listening to this classic song (http://www.youtube.com/watch?v=fzMhh8zhTiY ) while improving my Starcraft 2 Laddering position of course.
• Working Hard: Quote “Stress + Overtime = Weight gain -> More Stress -> Stress + Overtime… (T_T)”
• Learning to use Blender: http://yfrog.com/h413067151p
• Bitching about it’s lack of ngons, which it now has in dev branch (^_^) Great job guys!
• Which led to this game asset: http://yfrog.com/h0oiyp, http://yfrog.com/h2zxzvfzj, http://yfrog.com/h2og8sbj

I was going to write another designer skill-up post regards art tools but then I wanted to  follow up the brilliant, “You Should be Drawing” post by Mike Jungbluth, with a piece on the Flour Sack doodle. How everyone should doodle old school animation at least once to get a feel for motion and weight in animation… which led to me just wanting to shout from the roof tops.

BE ASPIRATIONAL!

If you’re reading AltDev, or better yet contributing, you have at least made the first steps. I encourage you to draw up a bucket list or a dream list of stuff you want to do! Try Schemer!

Start your list with some traditional gamedev skills.

• Code
• Make “Hello World”
• Make Pong
• Make Particle Fountain
• Art
• Draw Flour Sack Animation
• Draw Human Hand
• Model Something on Your Desk
• Design
• Invent a Card Game using a normal playing deck
• Make a Boardgame
• Write a Roleplaying Module

Nothing is stopping you extending that list to include:

• Cook a Quiche
• Crochet a Scarf
• Model something out of clay
• Learn to Cat Yodel

There is no such thing as a useless skill! Aspire to be more, learn more, do more! Also being reminded about last year via timehop and making plans in schemer are not bad places to start.

Everyone I’ve ever met worth anything wanted to me worth more.

P.S. Sorry about the fluffy post but I got super fired up and needed to shout!
P.P.S Also there are similiar services to TimeHop & Schemer. They just happen to be the ones I’m using.
P.P.P.S Should I think things through more and be less impulsive… maybe
P.P.P.P.S I really will do a more technical, solid post next time I promise.
P.P.P.P.P.S: Can you make a recursive Post Script? I wonder…