A widely-understood idea in games is the idea of the ‘game world.’ It is the virtual space in which the game takes place; it is the venue, the stage and backdrop. It’s full of interesting systems and processes: visuals, sound, motion, gravity, autonomous agents…

Yet while this world is presented to the player as a single cohesive, orchestrated whole, the reality of virtuality isn’t so singular. These different processes are frequently happening in their own independent worlds – like parallel dimensions that are deliberately overlapped. For example, what appears to be a highly detailed, carefully sculpted object in the visual world, may only be a flat-faced cube in the physics world.

The visual world

The physics world (artist's impression)

We consider them the same object because they share certain pieces of information. The position of the sculpture and the cube, for example, may be shared, such that as the cube is moved around by physical processes, the player perceives the sculpture to be moving. The sculpture itself has no mass or velocity; indeed it exists in a world where nothing has mass or velocity, and no processes exist that move objects around. But similarly, while the cube has mass and velocity, it has no textures or shaders, and so there is no way to really describe how it “looks.”

So objects in the game will have different representations in different ‘worlds.’ There may even be some worlds in which they have no representation at all. Invisible walls, for example, have a representation in the physics world, but no representation in the visual world; while individual particles in a particle effect often exist only in the visual world, playing no part in physics, AI or sound. In reality, sound would bounce off the particles of dust from an explosion just as any other surface, but in our simulations the dust just doesn’t exist from the sound’s point of view.

Why do we do this? Performance and simplicity, mostly. By setting up different ‘worlds’ for the different processes in the game, we can present those processes with data that is more suited to them. We could present the sculpture to the physics system in all its finely-detailed glory, but computing the collisions of such a complex object is a lot harder than computing collisions for a simple cube – and, if we’re honest, treating the sculpture as if it were a cube produces physical interactions that look good enough. By designing our different worlds carefully, we can simplify some of the problems that we face, turning intractable problems into tractable ones.

Take AI movement, for example. Imagine that you’re trying to write AI behaviour that’s based in the physics world. It’s logically possible – you can work out, from the physical representations of objects, which places your character can and can not be, and thus you can find a path to a target that accounts for obstacles like walls… but it would be very, very slow.

So instead, the AI world is very simple: it’s just a flat, connected mesh of triangles – a “navigation mesh” – coupled with a law that says that all AI agents must be located on the surface of one of the triangles at any time. There are no walls in the AI world – only the edges of the world itself. And now that we only need to obey the law about staying on the mesh, pathfinding and movement become a lot simpler; movement is effectively 2D within the surface of a single triangle, with a little more complexity at triangle borders. (There are more complications in practice, of course, like dynamic obstacle avoidance, and characters that jump, but they become special cases – the vast majority of agent movement is simplified).

The AI world - basic navigation mesh, perhaps with a couple of "cover points" marked for agents to use

How is it that doing AI computations in a world without walls is able to correctly move characters in a world with walls? Because we, as the architects of these worlds, carefully designed them to overlap in particular ways. The AI world’s navigation mesh, for example, does not have any triangles that would overlap with the Physics world’s collision mesh. And the result? AI agents are prohibited – by their own world’s laws – from being in any position that would see them stuck in a wall, because such a position would not be on the surface of the AI world’s navigation mesh.

They appear to be exercising collision detection in the physics world – but in fact, their inability to walk through walls is almost coincidental.

When the worlds don’t overlap in the right way, or when one world lacks the laws required for a particular object to exist in it, we can get some odd results. For example, it’s common that the audio world in a game is a simple, infinite void, populated by a collection of point-based sound emitters and, somewhere, a ‘listener.’ While this is sufficient to model some of the behaviour we expect from sound – for example, as I walk closer to a sound source, the sound gets louder – it’s also lacking the ability to model sound occlusion. If I’m standing 20 metres from an explosion, it’ll sound the same regardless of whether those 20 metres are occupied by air or by concrete. (I understand that the Call of Duty games shipped with an implementation of sound occlusion, but beyond that I don’t see many people talking about it. Anyone know how their implementation works?)

Oddly, players are used to a certain amount of discrepancy between worlds – they don’t expect particles to get in the way of bullets, or for the player character to bob up and down while walking over a pile of gibs. But many other situations are considered problematic. For example, walking against a wall that results in the player’s mesh intersecting the wall, because the physics representation is too tight relative to the visual representation; or a protruding bit of floor that the player appears to be able to stand on but which they actually fall straight through because it’s absent from the physics world.

Ensuring that these worlds are correlated in a way that makes sense to the player is an important task. Much of the time I spent working in QA was dealing with failures of this type.

So, given that managing that correlation is important, how can we do it most effectively?

At the most basic level we can simply test the game as a whole, exploring the superposition of all universes and their corresponding processes. This is a natural part of game testing, but it’s slow and error-prone: Walking into every corner of every level to ensure that the player cannot “fall out of the world” is not a very fun task, I assure you.

The problem is that the information we can gather about the different worlds in their “natural” states is frequently limited. We can see several hundred meters ahead of us in the visual world, but we can only touch the objects immediately next to us in the physics world. So we can inspect the visuals of the game world en masse, but inspecting the physics requires touching everything in turn, kicking every rock to check whether it kicks back. It’s even worse with things like the AI world – you’d have to coax and push agents around the world to check that they don’t suddenly behave strangely in one place.

So the first obvious solution is to transpose one world into another: to visualise worlds that aren’t usually visual. Many game engines do this: rendering all the physics polygons, or the AI’s navigation mesh, as a translucent overlay over the regular visual world. Different colours are used to indicate different properties, such as “sticky” or “only traversable by crawling agents.” Transposing the other worlds into the visual world is usually the most convenient way to work, as we can gather a large amount of precise information quickly in that world, but it’s not the only transposition available: we could use audio, too. Imagine making physics surfaces close behind the camera emit a low hum, so you can tell when the camera is backing into something without having to look at it. There are some interesting possibilities here for enabling game development for the visually impaired, too.

The next solution is to try to ensure that all the worlds are being furnished with data from a common source. We can hand-craft collision and navigation meshes, but aside from the risk of human error, this is very prone to resynchronisation – someone changes the render mesh and doesn’t update the collision mesh. Instead, we can seek to generate our collision and navigation meshes from our render meshes. Tools exist for this already: libraries like QHull can be used to generate simple convex hulls for collision, or more advanced tools like Simplygon can produce more accurate low-polygon meshes suitable for use in the physics world. For navigation meshes, the Recast toolkit is excellent (and free!). Depending on your game’s needs, you’ll likely need to write some custom analysis and generation methods, but existing tech can be used to provide a lot of the infrastructure for that.

The most important thing, though, is to just be very clear about what the different worlds your game takes place in are, and which data and processes are present in each. Ensure that your processes don’t allow anyone to confuse a render mesh with a collision mesh. Ensure that your codebase doesn’t try to perform AI walkability tests to points that are on the render mesh but not on the AI mesh. And so on.

Understanding and embracing the segregation of processes into multiple, cohabiting game worlds can be a very useful tool for breaking down what your game is doing into manageable pieces. Once you’ve got it, you can zero in much more quickly on the causes of problems; and even, when faced with some new and difficult problem, spin off an entirely new world to contain, reduce, and resolve it.

I was walking in town today, anxious. The photography class I was headed to had started almost a half hour ago; I’d just had a slightly surreal, less-than-satisfactory experience trying to buy something; I was walking, rushing, pushing past students and tourists and everyone else as I traversed the streets of Oxford to get where I was going. To both further my isolation from the moving meat obstructing my path, and to give myself a beat to march to, I put on my headphones and set my iPhone to shuffle.

Two minutes later, I was listening to Muse’s Stockholm Syndrome, smashing my hand around in time to the beat and mouthing the words to the chorus. And in between the choruses, I couldn’t stop smiling.

Why is music so effective at altering my mood?

I mean, it’s weirdly effective. Few other things can bring about such a marked change so quickly. If I’m feeling a bit blurgh, I can watch a TV show, or play a game, or eat a nice meal, and I’ll feel better… but not that much better. It’ll relax me, distance me from my suffering and strife, and return me to a comfortable medium – but music energizes me, commands me, like nothing else.

It’s not like listening to a song that I’ve listened to many times before is teaching me anything new, or achieving anything in particular – yet it can elicit a reaction as strong as discovering that I’d been accepted to Oxford Uni. An odd equality, don’t you think?

I find it with games, very often, as well. I loved the Ace Attorney games, for example – but the parts of the game where I’d be most excited, where I’d shut out all distractions, live and die with every right and wrong answer… was whenever the “Cornered” theme was playing. Sure, it was cool to see the witness – who we all knew was secretly the killer, but hadn’t quite proved it yet – start to sweat, to watch their story break down and their web of lies unravel, but it just wasn’t the same when playing on mute.

Without the music, there was no urgency, no drama. I could understand the tension on an intellectual level, I knew what was happening and what it meant for the storyline, and so on. But with the music, I was tapping through the dialogue as quickly as the characters seemed, to me, to be speaking; gasping at each reveal or new ploy by the enemy, marveling at Phoenix’s quick moves to confound them (“…unless you have evidence that this man somehow flew through the air, which I very much doubt!” “I have evidence.” “WHA-!!”). I was aware how viscerally I was responding to the game, and I was aware that it was largely the music causing it, but it didn’t matter; in that moment, I wanted that response, I wanted to put myself in the game’s hands. Had the music risen in key partway through the sequence, I suspect I would have had an orgasm. Or a heart attack. Or both.

Other games have had effects of similar strength, if not always directed towards the same euphoria. Introversion’s Darwinia has some excellent tracks by Trash80/Tresk (such as Faces Of A Fashion and Excuses – the latter conveys a kind of isolated desolation that majorly frames the way I think about the game world). Terry Cavanagh’s VVVVVV has an excellent soundtrack, from which tracks like Positive Force and Potential For Anything tend to get me every time (Positive Force even moreso in its trance remix format). I should note, it’s not all chiptune and indie stuff; I will always love Marty O’Donnell’s Halo theme (especially when standing in a bus, which I am then inclined to imagine is some kind of military dropship), or Stuart Chatwood’s Tower of Dawn from the first of the new Prince of Persia games.

It’s not that listening to those tracks remind me of fun experiences – they do, but I reacted to them in the same way the very first time I had them, before I’d had the experiences to be reminded of. It’s qualities of the music itself, the structures and patterns and shapes and balances, that affect me. The slow build of a rising sequence, or the way a chaotic, swirling mass of mini-themes resolve gradually into a single magnificent musical megastructure – or, alternatively, the way that structure breaks back down into chaos. It may well not even be the entire song, but just some particular moment in it where something wonderful happens.

It’s almost never lyrics. Even overlooking the fact that I personally am very bad at hearing what lyrics are actually being sung, I’ve never understood why people are so obsessed with lyrics. Well, that’s not true – I think I understand it, but it seems like a misplaced focus. There are interesting things that can be done with the voice as an instrument, creating particular sounds and rhythms through choice words, but beyond that, isn’t it just poetry? To listen to the lyrics, rather than the music, seems to me like ordering a steak and then only eating the salad garnish. The meaning of lyrics can enhance the music, but in my experience, that’s rare; creators who focus on interesting lyrics tend to set them to boring music.

(Song titles are related. I think I do enjoy the VVVVVV soundtrack a little more as a result of the optimistic, forward-looking ideas conjured up by the track titles).

But what puzzles me is that music is the only thing that seems to have this kind of effect on me in such a cheap and immediate manner. Short of actual life-changing events (which are hard to set up, in any case), I don’t find myself overcome with emotion when walking through an art gallery, and I tend only to respond to pathos in fiction when it’s part of the payoff for something very long-established. Music is, I think, the only thing you could use if you needed to set my mood in 30 seconds. And if you picked the right music, you’d be guaranteed success.

Am I alone in this? I’m not sure. There are definitely people for whom music seems to do nothing. I wonder if there’s something else they respond to instead. Or maybe they’re just dead inside.

How Editor scripts are built

The different projects generated by Unity 3.4. The assets folder in each project is the same physical folder on disk; each project includes different files within it. Note that this only shows the C# projects - if you're using UnityScript, you get another four.

Because you don’t want all your editor customization to be included in the game that you ship, and because you don’t want your shipping game to have any dependencies on things in the Unity editor, Unity keeps runtime and editor code in separate assemblies. The compilation order is such that runtime code is compiled before editor code, so that your editor classes can safely refer to runtime components (otherwise it’d be difficult to edit them) – but it does mean that your runtime components can’t reference any of your editor code. You have to maintain a strict layering. Unity’s gotten a bit more explicit about this in 3.4 – now the project files it generates (for VS/MonoDevelop) clearly correspond to the four compilation stages it provides, so there’s no confusion about which files will get build at which times.

The documentation is a little unclear on one particular point. When I first started, I thought that I had to have a single ‘Editor’ folder at the top of my project, and that all editor classes had to go inside it. The system’s actually a bit more flexible than that; you can have as many ‘Editor’ folders as you want in your project, buried wherever you like inside your Assets folder, and all of them can contain editor code. So now it’s very common that I’ll have a folder for one particular feature (e.g. ‘AI’) that contains all the components for that feature, with an Editor folder alongside (e.g. ‘AI/Editor’) that contains the editor extensions for working with those components.

As far as Unity’s built-in types go, runtime types all live in the UnityEngine namespace (in the UnityEngine assembly), while editor types all live in the UnityEditor namespace (in the UnityEditor assembly).

The UnityEditor.Editor class

By far the most common kind of customization I set up is a custom inspector. Unity’s Inspector panel provides your window into a component’s state, but in its base form it only understands a limited set of types, and will only expose public fields (no properties). Custom inspectors give you the opportunity to completely control how users view and edit your components; for example, they let you display read-only properties, enforce value constraints, or just change the way an option is presented – for example, replacing a [0..1] float field with a percentage slider.

Inspectors in Unity are all subclasses of the Editor class, so that’s where you start. One thing I don’t like about the editor class, though, is the way it handles types: it has a ‘target’ member that refers to the object the inspector is editing, but it’s of the base ‘Object’ type, so you keep on having to cast it to a more useful type. To get around this I use a very simple generic class:


public class InspectorBase<T> : Editor where T : UnityEngine.Object
{
protected T Target { get { return (T) target; } }
}


Now, if I want to create an inspector for MyCustomComponent, I can derive the inspector from InspectorBase<MyCustomComponent>, and use the ‘Target’ member instead of the untyped ‘target’ member, and I don’t have to keep casting everywhere.

Note that you also need to attach the [CustomEditor] attribute to your inspector classes for Unity to actually pick them up and use them.

Editor GUI

Once you’ve created your custom inspector, the method you usually want to implement is OnInspectorGUI(). OnInspectorGUI() is responsible for specifying everything shown in the inspector, using Unity’s immediate-mode GUI system. Because this is editor code, we can use the types in the UnityEditor namespace, which includes EditorGUILayout. EditorGUILayout provides a bunch of really simple controls for use in the editor, over and above what Unity’s regular runtime GUI system offers. For example, say I want to show the user a field for entering a 3D position. I could use EditorGUILayout.Vector3Field():

Target.somePosition = EditorGUILayout.Vector3Field("Some position", Target.somePosition);

This results in a line in your inspector that looks like this:

The immediate-mode GUI system works such that if I change the values in the UI, Vector3Field will return the new values, and Target.somePosition gets updated. You’re free to manipulate the value – for example, clamping it to a range – before assigning it to the target; and you’re free to ignore the return value completely (which would effectively make the field read-only). The values don’t have to be coming to/from fields, either – you can expose properties in the inspector by adding lines for them in this way, or call a function to get the current value and another function to save it, whatever you like.

Of course, Unity does this stuff by default for any public members. If you just want to build on top of what Unity’s already showing, you don’t have to reimplement all those fields – Editor has a DrawDefaultInspector() method that tells Unity to draw all the controls it would usually draw, but when it’s finished, you’ve still got the opportunity to add a few extra fields and buttons yourself.

Speaking of buttons… EditorGUILayout is pretty comprehensive, but you might notice that there are some things missing – what if I want to put a “recalculate” button on my navigation mesh component, for example? The trick is that EditorGUILayout is still built on top of the regular runtime GUILayout, so everything in GUILayout is available to you as well.

As you make changes to the fields in the inspector, and assign new values to the fields of your target object, Unity detects that you’re changing the object and flags it as ‘dirty’ so that it will be written out to disk the next time you save the scene or project. This detection is limited: it only picks up direct assignment to public properties. If you’re changing the target object through properties or through calling methods on it, you may need to call EditorUtility.SetDirty yourself.

Extending the component context menu

It’s often useful to be able to manually trigger certain behavior when testing things out. You could do this by putting a button on the custom inspector that triggers the behavior:


if(GUILayout.Button("Explode now!")) Target.ExplodeNow();


but there’s an even easier way – one that doesn’t require a custom inspector at all. What you can do instead is use the UnityEngine.ContextMenu attribute:


/* In the target class... */
[ContextMenu("Explode now!")]
public void ExplodeNow() { ... }


Right-clicking in the component’s inspector – whether you’ve customized it or not – will then show you a context menu, with the extra item in there. Very handy for quickly rigging things up for testing.

Extending the main menus

Everything I’ve talked about so far has been about customization centered around a particular component. What about other kinds of extension, like general utilities?

The animation system in my game stores its assets in a folder structure, such that each folder corresponds to one entry in an enum. When I change the enum, it’s useful to be able to synchronize that folder structure, adding any missing folders and deleting any obsolete ones. So I’ve got this class, with a simple method:


public class AnimationSystem
{

}


But how and when do I call it? What I’ve done is to wire it up to a menu item in the Assets menu, using the MenuItem attribute:

[MenuItem("Assets/Sync folder structure")]
public static void SyncFolderStructure() { ... }


Clicking the menu item calls the function. Note that the function needs to be static – but the class it’s in can be anything, and can derive from anything (including nothing).

Wizards

Editor GUI elements don’t only have to live inside the Inspector. It’s also possible to create arbitrary editor windows, that can be moved and docked like any of Unity’s built-in windows, and that are populated using GUI commands just like in an inspector. One of the simplest ways of doing this is to use a ScriptableWizard, which is like a dialogue box – you display it, set some values in it, then hit a button to make it work its magic.

Unity's built-in Create Ragdoll wizard (GameObject->Create Other->Ragdoll). You drag and drop the relevant bones and bits into the slots in the window, and then hit the 'create' button to have it rig your ragdoll with rigidbodies and joints.

ScriptableWizard, by default, works almost like an inspector: any public fields in your derived class will be automatically shown in the wizard window. Your wizard might be as simple as a bunch of public fields, plus an OnWizardCreate() method, which Unity will call when the user hits the ‘Create’ button. Worth noting that you can change the text on that button, too, if ‘Apply’ or ‘OK’ or similar would be more intuitive.

The only other aspect of the wizard is deciding how the user will launch it; the usual approach is to use a menu item bound to a static function, as shown above:


[MenuItem("GameObject/Create Other/Explosion")]
public static void CreateExplosion()
{
ScriptableWizard.DisplayWizard<ExplosionWizard>("Create explosion");
}


Conclusion

There’s loads more – I’ve not even touched on custom asset types yet, or asset import postprocessors, or scene view customization. They’ll wait for another day…

The big, unannounced project I’m working on is reaching a point where most of the first-pass features are in, and now we need the game to run at an acceptable speed. It turns out that trying to render 8 million normal-mapped triangles 60 times a second on a GeForce 8800 is asking a little much of the poor thing. Who knew?

Rendering 8 million triangles at 60Hz on a GeForce 8800 requires 1.21 jiggawatts of power.

So, a large chunk of the past couple of weeks has been spent getting various performance-improving systems in place: a level-of-detail system, some simple area-based culling, and so on. Unity’s built-in profiling tools range from “rudimentary” to “undocumented,” so I’ve supplemented them with some of my own – the most useful of which, so far, has been a simple tool that generates a list of all the meshes presently being rendered in the scene, ordered by their total contribution to the polygon count. Very helpful for telling us where to focus our art optimization efforts.

One of the things my polygon-counter tool doesn’t do is generate percentages.

It would be easy to add, but I consciously decided against it, because it’s very easy to be misled by them. I’d have thought that most people in game development know how percentages work, but it seems to happen all the time when people look over my shoulder at the profiler graph: “Hey, physics is using 30% of your frame time – it was only using 10% when I looked a few hours ago. Did you break something?”

…

No, I optimized the renderer, dropping the total frame time from 20ms to 6.6ms. The physics is taking 2ms just like it was before. Kthx.

If you’re focusing your optimization on decreasing the percentage that a particular component takes up, then you can always achieve that by making everything else slower… whenever you’re dealing with a percentage, you need to remember to ask: A percentage of what? And whenever you’re comparing two percentages, you need to remember to ask: Are they percentages of the same thing?

Frames-per-second values are similarly deceptive, because the same piece of work will have a different effect on the number at different points. (Also, shouldn’t they be measured in Hertz (Hz)? Physics, people!)

Say I’m running at 30FPS in a development build, and I’ve written a new feature. I turn it on, and the framerate drops to 28FPS – that’s not so bad, right? Only 2 frames per second dropped? Why, that’s only 6% of the original framerate…

Later on in the project, though, when the art’s been optimized and we’re running at more like 80FPS, I turn the feature on again – and this time the framerate drops from 80FPS to 67FPS. Whoa! 13FPS drop! That’s more than six times what it was before. Is the code slower?

The code’s taking exactly the same amount of time (2.38ms per frame). Most processes in a game will take the same amount of time regardless of what your framerate is; what determines their time is how complex they are and how much data they’ve got to process, not how fast the overall system is. And because the FPS figure is the reciprocal of the frame time (i.e. it’s 1/{frame time in seconds}), adding or removing milliseconds at a low FPS will produce a much smaller change than adding or removing milliseconds at a high FPS.

If you want your game to run at a solid framerate of X frames per second, then you’ve got 1000/X seconds to burn each frame. Here’s a handy table for you:

No, wait, sorry, my bad. (Seriously, Google Images? That’s the handiest table you could give me?)

See how adding or removing a single millisecond to the per-frame workload makes a much bigger difference to a 100Hz framerate than to a 30Hz framerate?

Now, pick a target framerate, write down how many milliseconds per frame you have accordingly, and then forget about the framerate. Focus on concrete time. The framerate lies to you, but concrete’s always been supportive.

This is, of course, all quite apart from the fact that in modern multicore games, it may not even make sense to be talking about one single “framerate.” You’ve got to start separating it out into render frames, physics frames, input frames…

So, anyway, those are two numbers that deceive you: profiler percentages and framerates. Anyone got other frequently-misinterpreted, pathologically misleading, or just plain wrong numbers that they’d like to share with the class?

Whenever you’re creating a process that will take place over multiple frames – without multithreading – you need to find some way of breaking the work up into chunks that can be run one-per-frame. For any algorithm with a central loop, it’s fairly obvious: an A* pathfinder, for example, can be structured such that it maintains its node lists semi-permanently, processing only a handful of nodes from the open list each frame, instead of trying to do all the work in one go. There’s some balancing to be done to manage latency – after all, if you’re locking your framerate at 60 or 30 frames per second, then your process will only take 60 or 30 steps per second, and that might cause the process to just take too long overall. A neat design might offer the smallest possible unit of work at one level – e.g. process a single A* node – and layer on top a way of grouping work together into larger chunks – e.g. keep processing A* nodes for X milliseconds. (Some people call this ‘timeslicing’, though I don’t).

Still, allowing the work to be broken up in this way means you have to transfer state from one frame to the next. If you’re breaking an iterative algorithm up, then you’ve got to preserve all the state shared across iterations, as well as a means of tracking which iteration is to be performed next. That’s not usually too bad – the design of an ‘A* pathfinder class’ is fairly obvious – but there are other cases, too, that are less pleasant. Sometimes you’ll be facing long computations that are doing different kinds of work from frame to frame; the object capturing their state can end up with a big mess of semi-useful ‘locals,’ kept for passing data from one frame to the next. And if you’re dealing with a sparse process, you often end up having to implement a small state machine just to track when work should be done at all.

Wouldn’t it be neat if, instead of having to explicitly track all this state across multiple frames, and instead of having to multithread and manage synchronization and locking and so on, you could just write your function as a single chunk of code, and mark particular places where the function should ‘pause’ and carry on at a later time?

Unity – along with a number of other environments and languages – provides this in the form of Coroutines.

How do they look?

In “Unityscript” (Javascript):

1
2
3
4
5
6
7
8
9
10

function LongComputation()
{
    while(someCondition)
    {
        /* Do a chunk of work */
 
        // Pause here and carry on next frame
        yield;
    }
}

In C#:

1
2
3
4
5
6
7
8
9
10

IEnumerator LongComputation()
{
    while(someCondition)
    {
        /* Do a chunk of work */
 
        // Pause here and carry on next frame
        yield return null;
    }
}

How do they work?

Let me just say, quickly, that I don’t work for Unity Technologies. I’ve not seen the Unity source code. I’ve never seen the guts of Unity’s coroutine engine. However, if they’ve implemented it in a way that is radically different from what I’m about to describe, then I’ll be quite surprised. If anyone from UT wants to chime in and talk about how it actually works, then that’d be great.

The big clues are in the C# version. Firstly, note that the return type for the function is IEnumerator. And secondly, note that one of the statements is yield return. This means that yield must be a keyword, and as Unity’s C# support is vanilla C# 3.5, it must be a vanilla C# 3.5 keyword. Indeed, here it is in MSDN – talking about something called ‘iterator blocks.’ So what’s going on?

Firstly, there’s this IEnumerator type. The IEnumerator type acts like a cursor over a sequence, providing two significant members: Current, which is a property giving you the element the cursor is presently over, and MoveNext(), a function that moves to the next element in the sequence. Because IEnumerator is an interface, it doesn’t specify exactly how these members are implemented; MoveNext() could just add one to Current, or it could load the new value from a file, or it could download an image from the Internet and hash it and store the new hash in Current… or it could even do one thing for the first element in the sequence, and something entirely different for the second. You could even use it to generate an infinite sequence if you so desired. MoveNext() calculates the next value in the sequence (returning false if there are no more values), and Current retrieves the value it calculated.

Ordinarily, if you wanted to implement an interface, you’d have to write a class, implement the members, and so on. Iterator blocks are a convenient way of implementing IEnumerator without all that hassle – you just follow a few rules, and the IEnumerator implementation is generated automatically by the compiler.

An iterator block is a regular function that (a) returns IEnumerator, and (b) uses the yield keyword. So what does the yield keyword actually do? It declares what the next value in the sequence is – or that there are no more values. The point at which the code encounters a yield return X or yield break is the point at which IEnumerator.MoveNext() should stop; a yield return X causes MoveNext() to return true and Current to be assigned the value X, while a yield break causes MoveNext() to return false.

Now, here’s the trick. It doesn’t have to matter what the actual values returned by the sequence are. You can call MoveNext() repeatly, and ignore Current; the computations will still be performed. Each time MoveNext() is called, your iterator block runs to the next ‘yield’ statement, regardless of what expression it actually yields. So you can write something like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

IEnumerator TellMeASecret()
{
  PlayAnimation("LeanInConspiratorially");
  while(playingAnimation)
    yield return null;
 
  Say("I stole the cookie from the cookie jar!");
  while(speaking)
    yield return null;
 
  PlayAnimation("LeanOutRelieved");
  while(playingAnimation)
    yield return null;
}

and what you’ve actually written is an iterator block that generates a long sequence of null values, but what’s significant is the side-effects of the work it does to calculate them. You could run this coroutine using a simple loop like this:

1
2

IEnumerator e = TellMeASecret();
while(e.MoveNext()) { }

Or, more usefully, you could mix it in with other work:

1
2
3
4
5
6

IEnumerator e = TellMeASecret();
while(e.MoveNext()) 
{ 
  // If they press 'Escape', skip the cutscene
  if(Input.GetKeyDown(KeyCode.Escape)) { break; }
}

It’s all in the timing

As you’ve seen, each yield return statement must provide an expression (like null) so that the iterator block has something to actually assign to IEnumerator.Current. A long sequence of nulls isn’t exactly useful, but we’re more interested in the side-effects. Aren’t we?

There’s something handy we can do with that expression, actually. What if, instead of just yielding null and ignoring it, we yielded something that indicated when we expect to need to do more work? Often we’ll need to carry straight on the next frame, sure, but not always: there will be plenty of times where we want to carry on after an animation or sound has finished playing, or after a particular amount of time has passed. Those while(playingAnimation) yield return null; constructs are bit tedious, don’t you think?

Unity declares the YieldInstruction base type, and provides a few concrete derived types that indicate particular kinds of wait. You’ve got WaitForSeconds, which resumes the coroutine after the designated amount of time has passed. You’ve got WaitForEndOfFrame, which resumes the coroutine at a particular point later in the same frame. You’ve got the Coroutine type itself, which, when coroutine A yields coroutine B, pauses coroutine A until after coroutine B has finished.

What does this look like from a runtime point of view? As I said, I don’t work for Unity, so I’ve never seen their code; but I’d imagine it might look a little bit like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

List<IEnumerator> unblockedCoroutines;
List<IEnumerator> shouldRunNextFrame;
List<IEnumerator> shouldRunAtEndOfFrame;
SortedList<float, IEnumerator> shouldRunAfterTimes;
 
foreach(IEnumerator coroutine in unblockedCoroutines)
{
    if(!coroutine.MoveNext())
        // This coroutine has finished
        continue;
 
    if(!coroutine.Current is YieldInstruction)
    {
        // This coroutine yielded null, or some other value we don't understand; run it next frame.
        shouldRunNextFrame.Add(coroutine);
        continue;
    }
 
    if(coroutine.Current is WaitForSeconds)
    {
        WaitForSeconds wait = (WaitForSeconds)coroutine.Current;
        shouldRunAfterTimes.Add(Time.time + wait.duration, coroutine);
    }
    else if(coroutine.Current is WaitForEndOfFrame)
    {
        shouldRunAtEndOfFrame.Add(coroutine);
    }
    else /* similar stuff for other YieldInstruction subtypes */
}
 
unblockedCoroutines = shouldRunNextFrame;

It’s not difficult to imagine how more YieldInstruction subtypes could be added to handle other cases – engine-level support for signals, for example, could be added, with a WaitForSignal("SignalName") YieldInstruction supporting it. By adding more YieldInstructions, the coroutines themselves can become more expressive – yield return new WaitForSignal("GameOver") is nicer to read than while(!Signals.HasFired("GameOver")) yield return null, if you ask me, quite apart from the fact that doing it in the engine could be faster than doing it in script.

A couple of non-obvious ramifications

There’s a couple of useful things about all this that people sometimes miss that I thought I should point out.

Firstly, yield return is just yielding an expression – any expression – and YieldInstruction is a regular type. This means you can do things like:

1
2
3
4
5
6
7
8
9
10

YieldInstruction y;
 
if(something)
 y = null;
else if(somethingElse)
 y = new WaitForEndOfFrame();
else
 y = new WaitForSeconds(1.0f);
 
yield return y;

The specific lines yield return new WaitForSeconds(), yield return new WaitForEndOfFrame(), etc, are common, but they’re not actually special forms in their own right.

Secondly, because these coroutines are just iterator blocks, you can iterate over them yourself if you want – you don’t have to have the engine do it for you. I’ve used this for adding interrupt conditions to a coroutine before:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

IEnumerator DoSomething()
{
  /* ... */
}
 
IEnumerator DoSomethingUnlessInterrupted()
{
  IEnumerator e = DoSomething();
  bool interrupted = false;
  while(!interrupted)
  {
    e.MoveNext();
    yield return e.Current;
    interrupted = HasBeenInterrupted();
  }
}

Thirdly, the fact that you can yield on other coroutines can sort of allow you to implement your own YieldInstructions, albeit not as performantly as if they were implemented by the engine. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

IEnumerator UntilTrueCoroutine(Func fn)
{
   while(!fn()) yield return null;
}
 
Coroutine UntilTrue(Func fn)
{
  return StartCoroutine(UntilTrueCoroutine(fn));
}
 
IEnumerator SomeTask()
{
  /* ... */
  yield return UntilTrue(() => _lives < 3);
  /* ... */
}

however, I wouldn’t really recommend this – the cost of starting a Coroutine is a little heavy for my liking.

Conclusion

I hope this clarifies a little some of what’s really happening when you use a Coroutine in Unity. C#’s iterator blocks are a groovy little construct, and even if you’re not using Unity, maybe you’ll find it useful to take advantage of them in the same way.

A designer, a programmer, and a producer walk into a cocktail bar. The barman asks, “What can I get for you?”

The designer says, “Give me something fruity, with an umbrella in. No gin though, it makes me violent.”

The programmer says, “Mix two parts vodka with one part white rum, shake it with ice. Pour it into a glass with strawberry syrup, and top up with sprite. Stick an umbrella in the top and hand it to me.”

The producer, who only came along because they needed a third person to make up the joke, sighs and says, “I’ll have what these guys are having.”

The bartender begins making the drinks, “OK, three Equivalent Expressions, coming up…”

Anyway, folks…

There’s a fundamental difference between the kind of problems that designers and programmers spend their time solving, and as such, I think there’s a difference in their habitual approach to solving problems, and the way they structure and explain their solutions.

Designers are fundamentally interested in the properties of the end result – the visible ‘outer shell’ of the product. They don’t care about the technology inside it, beyond the observable effects (like it being fast enough, not crashing, etc). Their descriptions of the product, and how it should be, are thus all written in terms of these visible properties: which things the player experiences, when and where the player experiences them, and so on.

Programmers, by contrast, are focused on the internals of the product – they are tasked with filling in the gap between the outer shell provided by the designers, and the solid core of functionality provided by the platform. It’s like a translation process – translating high-level game concepts into low-level hardware operations – and so their explanations are in similar terms: which lower-level concepts are in use, how they are combined into large structures, and so on.

Speaking in tongues

This mismatch in modes of thinking requires extra effort to ‘bridge the gap.’ However, it’s not symmetrical.

As we’ve said, the entire focus of a programmer is on constructing these onion-layers of high-level to low-level concepts, composing and decomposing functionality to create the desired result. Connecting concepts between different layers of abstraction is programming. So when a designer talks to a programmer in the highest-level terms, mapping those terms down to the level that the programmer is working at is employing the programmer’s core skill.

The same is not so true for designers. Taking information about the implementation, and deducing from that the visible effects and constraints upon the outer shell, is not actually a part of design. Design is making decisions about the outer shell, within the constraints imposed by the implementation; figuring out what those constraints are is something different. Good designers can do it, of course, but it’s not their core skill.

To put this in slightly more concrete terms: When a designer tells a programmer that he wants a particular enemy to shout something when he sees the player, figuring out the points in the code that will need to be modified to support that is what programming’s all about. But when a programmer tells a designer that he’s added sound hooks to the base enemy class upon selecting a new object of interest, figuring out what this will actually allow the designer to do is not what design’s all about; it’s deciding which enemies should play sounds, and which sounds, that employs their core skill.

Another example, from outside the games industry, is CSS. Of all the designers I know who’ve done work with CSS, there are a few particular parts that they always have trouble with – display, position, and float. Together, these properties are extremely powerful for producing fluid layouts – but using them the way you want requires that you stop thinking like a visual designer, and start thinking like a web page renderer layout system. First this box will be laid out, then this one will be floated to the left inside it, then this other one will be positioned relative to it… it’s a procedural combination of pieces, compensating for the fact that CSS is lacking some useful layout concepts, like built-in support for columns. Knowledge of how to achieve things like an equal-height 3-column layout is passed around as a bit of arcane ‘magic’ that designers just learn and apply, understanding it only in as far as is necessary to use it.

So, I think that when it comes to interfacing design and programming, it would be most efficient to get the programmers to learn to ‘speak designer,’ rather than getting the designers to learn to ‘speak programmer.’ It’s a better match to their core skills, to their habitual modes of thinking.

Designing for designers

Most established programmers already do this in direct communication, I think. When explaining a new system to the design team, you don’t tell them all the class names and algorithms used; you tell them about the features and behaviors that they will use, phrased in their terms, the terms of the end user’s experience. Sure, sometimes you slip up – be it through absent-mindedly forgetting that you’re not talking to fellow programmers, or through enthusiasm about the elegance of what you built, or whatever – but even then, the designers usually speak enough Programmer to follow along.

Where I think this happens less is in tool design.

Internal tools frequently present their concepts and functionality in implementation terms, rather than in end-user-experience terms. Designers get settings with names like ‘Collision Detection mode’ and options like ‘Discrete’ and ‘Continuous,’ which only make sense if you know about how collision detection systems work. So what happens is they learn that they need to set it to ‘Continuous’ for fast-moving objects and ‘Discrete’ for slow-moving objects, and at that point the setting has basically become equivalent to a ‘Movement class’ setting with ‘Fast’ and ‘Slow’ options, except that it includes this extra pointless translation step that needs to be learnt and applied.

It’s not hard to imagine how the tools end up like this. Many of them start out almost as testing rigs for the programmers; the tool is there to allow the programmer to quickly configure and test the system’s features and interactions, so they write it in Programmer, and when they’ve done that it’s technically possible for the designer to set it up however they need it to be, so why make further changes? Yet while it may be technically possible for for the designers to set things up the way they’re needed, it’s not easy, which means that it requires training, takes longer, distracts the designer, and is more error-prone. These costs are often not taken fully into account. The tool has effectively been designed by a programmer, and if that’s OK, why do you have designers on your team at all?

“Well, what if someday there’s an object that is slow-moving but needs Continuous collision detection for some reason nobody can think of right now? Better to keep the flexibility, right? Also, while it might make it easier for the designer to set things up, it makes it harder for the programmer, because now they’ve got to keep remembering that ‘Fast movement mode’ means ‘Continuous collision detection.’ It’s not saved any work, it’s just moved it around.”

Firstly, bear in mind that remembering things like “Fast movement mode is implemented using continuous collision detection” is a programming core skill – just like remembering things such as “CPU Texture reads are implemented using readback-from-video-memory, which is slow, and so should be avoided.” It might be the same amount of work, but programmers have more practice at it.

Secondly, though: simplicity and flexibility aren’t mutually exclusive! Presenting designers with options “phrased in Designer” doesn’t have to mean taking away the options “phrased in Programmer.” The implementation-level options can be hidden away in ‘Advanced Settings’ sections, or in a ‘Programmer View’ mode. The designer-friendly options can be provided as presets – a quick means for designers to automatically set the implementation-level options, while still being able to keep thinking in terms of the end user experience, and without losing the flexibility of tweaking things more directly when required. These are solvable problems!

TL;DR version:

Programmers and designers have different habits for thinking about problems. When programmers create tools that designers will use, they should present those tools in ‘design language’ – the language of the end-user experience – rather than just exposing the flags and values that the implementation works with. It’s better for them to do that, than for the designers to learn how to manipulate the implementation, because figuring out how to set the implementation for a particular design goal is what the programmers are good at.

Blame @bobvodka if you didn’t think the opening joke worked. If he’s not escaped already, that is.

The truth is that Unity is an incredibly powerful, versatile tool; however, for a person used to a more traditional development process, it can be a bit difficult to grasp at first. I’ve been working extensively with Unity for about 8 months now, and I’m at a point where I’m really comfortable with it – to the extent that I start prototyping all new ideas in Unity, unless there’s a really good reason not to – and I’ve started extolling its virtues to others. Frequently, I see those people then run into exactly the same culture shock I first experienced with it. So, I thought I’d write down a few notes about the things I’ve learned – the things I wish someone had told me when I first started getting into it.

Misconception 1: It’s all about the Scene View

By far the most visually impressive and dominating part of Unity – especially if you load it up with the example project that ships with it – is the Scene View, presenting one or more 3D views of the game in ‘editing mode.’

This is all very bright and exciting, and tends to be the first thing people want to interact with. Anyone with a basic knowledge of 3D editing programs figures out how to fly the camera around pretty quickly, and starts clicking on things. It becomes very easy to start out by assuming that the scene view is at the center of your development process – that everything will be drag and drop, and that code is a rare and distant sideline.

In practice, while the scene view is important, it’s by no means the center of the development process. It’s highly significant when you’re doing ‘level editing’ type tasks – laying out the geometry that makes up your game world – and it also works as a nice intermediary environment for building up prefabs (more on those in a minute), but that is all. You’ll still be writing code in a code editor which you can then make use of in the scene view, just as a traditional project would write code that the designers could then make use of in their level editor. Though I’m working with Unity as a development platform, I still spend most of my time in Visual Studio.

Misconception 2: Every kind of game object will have a distinct, concrete class

Unity has been my first substantial experience with a component-based architecture.

Traditional games tend to have a concrete class for each kind of entity. Need a gun? Make a class for it, derived from ‘PhysicalObject.’ Need a special kind of gun that also fires grenades? Subclass your ‘gun’ class. One of the commercial games I worked on had an entity inheritance hierarchy that must have been at least 10 levels deep; each level added certain bits of functionality, like ‘is renderable’ and ‘plays pre-recorded animations’ and ‘generates physics impulses’ and ‘is considered friendly to the player.’ Inheritance-heavy architectures tend to be very popular with object-orientation newbies; they grok the meaning of an ‘is-a’ relationship, and then build massive towers out of it.

More savvy – or more jaded – software architects know the value of composition over inheritance. It’s a lot less fragile. Instead of saying that an enemy ‘is-a’ skinned, animated mesh, they say that an enemy ‘has-a’ skinned, animated mesh. Instead of the HugeSpiderEnemy class inheriting from SkinnedMesh, it has a SkinnedMesh member variable. Following this approach, these towering inheritance hierarchies turn into fairly simple classes that just contain a bunch of relevant components: a SkinnedMesh, an AnimationPlayer, a PhysicsInteractor, and so on, with a bit of glue in the class itself to keep all the different bits in sync. There’s often still a bit of inheritance, as a neat way to encapsulate the glue code without sacrificing type interchangeability, but it’s shallow. This makes it much easier to swap systems in and out, and removes the risk of changes in one system percolating down through everything via inheritance.

Unity takes this composition-based approach to its logical conclusion by having you build all your game objects entirely using composition – even the ‘glue code.’ Every game object consists purely of composed parts. The end result is that if you did still have concrete classes for each kind of game object, the only difference between them would be the components that make them up. As such, you only need one concrete class for a game object – Unity appropriately calls it GameObject – and it acts as little more than a container for components.

What about the ‘glue’ code, the stuff that would normally tie all the components together? You can build it as a component too. With the GameObject acting only as a component container, it’s up to the individual components to sort things out between themselves. The glue component isn’t one that you might expect to use on any objects other than this one – but the same was true of the dedicated class, too. Note too that component inheritance is still perfectly normal, so your ‘glue’ code components can participate in an inheritance hierarchy if appropriate.

Components make it very easy to build objects that interact with a wide number of systems, without introducing strong dependencies between these systems. Some care must be taken to ensure that component interactions are handled nicely, but no more than previous approaches.

Misconception 3: Prefabs are only for making repetitive levels

One of the ideas I encountered quite early on was the idea of the prefab. Prefabs are “reusable GameObjects,” according to the manual. When I first saw them, and saw that they were called prefabs, I thought: great! I can use those to quickly build up large scenes by effectively copying-and-pasting objects around. If I want to dot trees all over my terrain, I can use a bunch of tree prefabs. If I want buildings, or bits of buildings, I can create prefabs for those, too. Great! But that’s really just a time-saving tool for the artists and level designers, surely…

I’d missed the real point of prefabs. It’s true that you can use them for easily copying objects around, but the really vital thing is that you can insert copies of them at runtime, too. They’re not only about composing the scene in the editor. They’re also about composing the scene, from code, at runtime, too. (For the more pattern-oriented amongst you, what I’m driving at here is the Prototype pattern. Feel free to refresh your memory so you can read the next bit with a knowing smile).

Naturally there are many objects in your game that aren’t in the scene when you start. Bullets in motion. Gibs. Enemies that haven’t spawned yet. Sometimes, you could put these things into the scene and just mark them as inactive, but not always; you need the option to create things dynamically. Unity has that, of course: GameObject has a public constructor, so you can create the GameObject, set its name and layer, then use GameObject.AddComponent<>() to add all the components you need, then set up all the parameters, etc. There’s a twofold problem with this approach, though: firstly, it’s incredibly tedious, and secondly, it makes it really difficult for non-coders to change the way things are set up. If an artist or designer wants bullets to play a different sound or use a slightly different color tracer, they need a coder to go in and alter the spawn code.

This is where prefabs come in. Instead of building a new object from scratch, what you can do is to create a clone of the prefab. Cloning an object duplicates all of its components, along with all of its children and their components, so you don’t have to attach anything or set anything else up; the prefab acts like a prototype for the kind of object you want, a ‘reference copy.’ Furthermore, as the prefab is just a GameObject (or tree of GameObjects), it can be edited by artists and designers in exactly the same way as anything else in the scene; if the artist wants to make your plasma bullets glow more blue, he can click the prefab, find the relevant component, adjust the value, save the prefab, and run the game – no code changes required.

As noted before, this is actually a well-established software pattern, known as the Prototype pattern.

Misconception 4: You need to write debugging visualizers

It’s quite common to want to annotate the screen with extra information while your game is in development. Take AI as an example: it’s often very useful to be able to see the path that your agent is trying to follow, or the state he’s presently in, while the game is running. So you write a little bit of code that dumps the information into the top left corner of the screen using the GUI renderer, and watch it change as you go. That’s what I was doing to start with, even in Unity – the ease of access to the GUI system from any component made it very easy – but I don’t anymore, because I’ve realized that Unity already has a method for inspecting such data, and it’s far superior: You can just use the editing tools as if the game weren’t running.

Say you’re in the middle of the game, and one of the AIs is doing something funny. You can pause the game (if you like), pull up the hierarchy window, and click on him to view his inspector. Maybe you’ve got hundreds of AIs and it’s not clear which one is him; in that case, you can switch to the Scene view, navigate to where you paused the game, and click on him to select him. The inspector panel will then let you look into the present state of all components, in exactly the same way as it will let you look at the initial state of them when the game’s not running.

Maybe the default information displayed isn’t enough. In that case you’ve got two choices: firstly, you can try toggling the inspector into ‘debug’ mode, which causes various private members to show up (usually desirable when you’re trying to inspect internal state); and secondly, you can write a custom inspector class for the component you’re interested in, one that displays exactly the information you want in exactly the way you want it. There’s a simple function call – DrawDefaultInspector() – that you can use to make sure that all Unity’s existing information continues to be displayed, and then you can augment that information with any calculated or private values you might want. (Notably, Unity’s default inspector does not show properties, only fields, so I often find myself just adding a couple of entries to show the present values of properties while the game is running).

You can even pull the Game tab out from the Unity interface and drop it onto a second monitor. Now you can play the game on one monitor, and watch the scene view, hierarchy, and inspectors all on the other. Pausing the game isn’t even necessary; all the views can update in real-time, while the game is running, quite happily.

Misconception 5: There’s no spatial indexing; you have to search the whole scene for everything

Unity supports finding objects in the scene based on their name, their tag, and the components attached to them, but it’s slow. The physics engine – Nvidia’s PhysX – doubtless does some scene management to accelerate collision tests, and the fruits of this are exposed to you as raycasts/linecasts/spherecasts in the Physics class, but these only apply to objects that the physics engine knows about; you can’t use it to retrieve non-physics objects like particle effects, for example.

There’s nothing stopping you from maintaining your own lists and caches, though. If you’re going to need to run through a list of all objects that have a component of type X, then type X can, quite happily, keep a private static list of all its instances, and provide a couple of functions for searching that list much more quickly than the general scenegraph functions would be. You need to be careful about exactly when you add and remove objects from this list – using OnEnable()/OnDisable() is likely to be a better plan than using Awake()/OnDestroy(), for example, because that way you won’t have any problems with things like prefabs or inactive objects accidentally ending up on the list – but it’s still fairly straightforward to do.

Another approach I’ve found quite fruitful is to use tracking volumes. A tracking volume is a collision volume, marked as a trigger (i.e. look for collisions, but don’t block any movement), with a simple script attached. When an object enters the volume, the script adds it to a list; when it leaves, it removes it; and when any other code wants to find out which objects are presently in the volume, the script can simply provide the list. This is exploiting the fact that the physics engine is doing highly optimized spatial checks anyway, so we might as well cache the results, instead of putting the physics engine on the spot right when we want the data. The tracking volume script can be completely generic for use with any object and volume setup, and you can augment it with useful features like filtering based on object type (‘only track objects that have a Flammable component’) or layers (‘only track objects in the Pickup layer’).

Misconception 6: Coroutines Are Useless

There’s a lot of work that a game needs to perform every frame – obviously there’s all the core things like rendering, physics, animation, etc. There’s also quite a lot of gameplay logic – responding to player input to initiate actions, for example. We want to minimize latency so we do it as quickly as possible, and that means keeping an eye out for it every frame.

There are some bits of logic that don’t really need to be checked for every frame, though. Take the win conditions: you want to check that the player’s in a particular area, that he’s killed particular enemies, collected certain items, and so on. Would it really kill the experience if there was a half-second delay between killing the final enemy and the win sequence kicking off? Frequently, no. But we tend to check every frame anyway, because adding the code into the per-frame update is easier than trying to schedule checks less frequently.

Unity’s coroutine support makes for a really nice alternative to this. Coroutines are used to implement cooperative multitasking. You can set up a long-running process – an infinite loop, for example – but stop it from blocking other code by having it ‘yield’ at suitable points. The scheduler is then very simple: it just cycles through all the coroutines, running each one in turn until it yields, then moving on to the next.

Unity’s scheduler is a bit smarter than this, though, because when a coroutine yields, it can emit a value. In Unity, this value is treated as a note to the scheduler about when the coroutine should next be run. You can yield a null to indicate that you want to continue on the next frame; you can yield a WaitForSeconds(0.5f) object to indicate that you don’t need to run again until half a second has passed; you can yield an animation object to indicate that you don’t need to run again until the animation has finished playing; and so on. (Edit: I misremembered; yielding AnimationStates isn’t actually supported. See the comments for some ways to get around this).

This makes it very easy to turn an every-frame check into an every-three-seconds check: write the check in an infinite loop, with a yield-wait-for-three-seconds. Start the coroutine when the component is enabled, and stop it when it’s disabled (or if you’re mad like me, hook it up to a property setter, so that setting a CheckX flag property automatically starts and stops DoCheckXCoroutine). The value that you yield is evaluated at runtime, so you can do whatever you like to decide how long to yield for – maybe check more frequently when the player is getting close, or make the check interval a designer-configurable variable.

Conclusion

There’s probably more I could say, but I’ll leave it there for now, because I should really be working. I encourage you to give Unity a try, if you haven’t already.

The theme of the contest this time was “It’s dangerous to go alone! Take this.” I considered that if it was dangerous to go alone, then it must be safer to go together. I didn’t think it would be feasible to develop a full multiplayer game in 48 hours, but I could develop a game that was ostensible singleplayer while having some kind of interaction between all the people who are playing it. Kinda like an MMO, but where all the other players are invisible. (I considered trying to explain this within the game in terms of quantum interference and Everett MWI or something, but I didn’t get time).

Naturally, in order to have players interact, I’d need some kind of server that could co-ordinate that interaction. Initially I thought I might be able to use IRC, but, alas, I wanted the game to run in the browser, and there are security implications that make connecting to IRC from inside the browser window impossible, at least without some kind of proxy server. If I wanted to run in the browser and also have multiplayer elements, I’d need to run my own server somewhere.

What’s the simplest way to get custom code running somewhere Internet-accessible for the duration of the competition? I couldn’t run it from home – my home connection’s upload bandwidth is terrible. I didn’t really want to sign up for a full shell host; most of them have a minimum term of one month, and I only really need half of that.

Choosing EC2

I decided to try running my server on Amazon’s Web Services infrastructure, specifically their Elastic Compute Cloud (EC2). The multiplayer server is super-simple – a single instance that stores everything in memory – so I didn’t need most of the things that EC2 have on offer, but the ability to spin up a fresh Linux install in seconds, put my server software on it, and just turn it off when I’m done was very attractive.

I’m also within my first year of AWS usage, which means I qualify for their free usage tier: a 1.0 GHz Xeon processor, 613MB of memory, 10GB of storage, and 15GB of bandwidth each way in and out, free for a year. Hard to beat free.

I can’t write this post without mentioning the recent outage they had. Simply put, it doesn’t worry me. Amazon’s analysis of what went wrong is very thorough and shows their competency; shit happens, and when it does, the important thing is how it’s handled. That this kind of chain reaction can happen is interesting, but I’m confident that they’ll make the changes they need to make in order to interrupt the chain should anything like it happen again. Remember, Amazon dogfood this stuff; problems in their cloud don’t just affect their cloud customers, they affect Amazon.com itself.

Creating the Instance

The first step was to create the instance (the ‘virtual machine’ that I was going to use). From Amazon’s EC2 management page, I hit the ‘Launch Instance’ button, then picked the “Basic 32-bit Amazon Linux AMI” from the list. The page walked me through configuring how big I wanted the instance to be (“Micro” size to get the free tier) and which ‘zone’ I wanted it to be hosted in (I didn’t care). There were a couple more pages of config where I could turn on things like advanced monitoring and enter custom metadata, but I didn’t need any of that.

Next it had me create a key pair for the machine. All the Amazon machine images are preconfigured with key-pair based SSH, and it’s really a much nicer method than using passwords anyway, so I had it create a key pair for me and downloaded the private key to my own machine.

Then it asked me to pick a Security Group. Security Groups are used for configuring the firewall; you might have one security group that allows SSH and HTTP, another that allows FTP, a third that allows only SSH, and so on. I put the machine in a ‘default’ group. Finally the wizard had me quickly review my settings; after a quick check, I hit OK, and the machine began booting.

It was ready within about 10 seconds.

Getting Connected

Now my machine was running, I wanted to connect to it and start working.

The first thing was to enable SSH on the firewall. The default security group settings are such that machines within a security group can connect to each other on any port, but there’s no access from the outside world by default. I went to the security group page, picked my ‘default’ group, and went to the ‘inbound’ tab. I picked ‘SSH’ from the ‘create a new rule’ dropdown, and left the source at 0.0.0.0/0 (i.e. any source). Hit Add Rule, hit Apply Changes, and the firewall was instantly reconfigured.

Now I just needed to get key-pair authentication working. My SSH client of choice on Windows is, of course, PuTTY; the private key file Amazon had given me isn’t something that PuTTY can read by default, but PuTTY companion program PuTTYgen can import it and convert it to PuTTY’s native format. Then I just had to set up the connection: the required settings in PuTTY are a bit hidden, but I set the auto-login username on the Connection->Data page to ‘ec2-user’, and pointed the private key file on Connection->SSH->Auth at my private key file. Then I just had to punch in the IP address of my instance – visible on EC2′s management page – and I was connected.

This entire process, from deciding to use EC2 to being connected to my very own instance over SSH, took about 15 minutes.

Installing Mono

I wanted to write my server in C#, just like the rest of my game, so I’d need the ability to compile and execute C# code on my server. I also wanted to use some of the server-side tools that Unity provides for dealing with things like cross-domain security policy, and they too are written in C#. Even if I compiled everything on my own machine, I’d still need the Mono runtime to be present in order to run them.

I was all set for a difficult and time-consuming process of building things from source or something, but then I found a perfect set of instructions on StackOverflow. Other than needing to prefix a few of the commands with ‘sudo,’ they worked flawlessly.

…and we’re away

That’s basically all I needed to do, beyond writing the server itself. I needed to open a couple more ports in my Security Group – one for the game server itself, and port 843 for the crossdomain.xml policy server – but that was basically it.

I’ve not set up any persistent storage for my instance. Other than the mono install and the server software, I’ve not changed the base AMI, and it’d take me 10 minutes to reapply those changes. In any case, I’m not planning on shutting down the instance until after the contest, at which point it won’t matter.

If you want to run a custom game server and you don’t want to do it from home, AWS may well be worth checking out: it’s extremely easy to get going, free for most of the things that a small game would need (for a year, at least) and scales nicely if you’re looking at doing something bigger.

It is the difference between actors acting out an event, and the lone playwright standing on a bare stage recounting the event to the audience.

From an information-content point of view, the latter still gets everything across – but the former would be more fun to watch. Why is that, though? What’s so fun about seeing the events acted out?

In many cases, if not all, fun comes from learning (as Raph Koster writes). The process of understanding – of gaining a new explanation of how something is done or why something works, and thereby solving a problem that was posed by the lack of that explanation – is enjoyable.

So maybe that’s why Showing is more fun than Telling – because it offers more to learn, more opportunities to undergo that process of understanding.

Telling

The Colonel was very brave.

When somebody Tells you something, you’ve got two options: take it or leave it. You can’t do more than that, because you’ve got nothing else to go on. It’s their way – their summary of events – or the highway. If you’ve got questions or objections, there is no further evidence available with which to reason. (We’re talking about a broadcast medium here, like a book or a film or a game. Obviously in a normal conversation, you can ask questions – but increasingly it stops being what I’d call Telling).

So the amount you can learn is very limited – the speaker, usually the narrator, has done most of the inference for you, so what you see is what you get. It’s efficient for the author to write, and efficient for the audience to consume, but if there are any doubts or questions lingering in the audience’s mind afterwards, it does nothing about that. Telling tells you the way things are, and hopes that that’s good enough for you.

Older games made heavy use of Telling for things like the game’s storyline; for example, the ‘mission briefing’ format, where the game’s story would be presented as screens of text between each level, is textbook Telling. At the same time, many games also explained their game mechanics by Telling the player how they worked in the manual.

Showing

Lying prone in the mud amidst the gunfire, the Colonel gritted his teeth, and reached down past his blood-stained uniform to pull a cigarette out from the packet. The wound hurt like hell, but he was damned if he was going back now – not while his boys were still in the enemy camp.

On the other hand, when somebody Shows you something, you are being given more detailed information, and being allowed to decide for yourself which things are significant, and which things feature into your ongoing understanding of how the characters think and how the world of the story works. You have (or make) guesses about how things work and why people do things, and as the Show gives you more and more information, you reject and refine those guesses. A Show can take a told idea – such as ‘the Colonel is brave’ – and let you see exactly why the narrator considers him brave, and when he is brave, and what his motivation is for being brave, and so on. If you doubted that he was brave – perhaps because he did other things that it didn’t seem so brave – then the more detailed account given by the Show might help you see him in a new light, and understand how his brave nature could fit with those other things. Showing allows you to not only discover the way things are, but also how and why they are that way, which is a much more substantial piece of knowledge that can be used to answer many questions and assuage many doubts.

The original Half-Life was greatly praised for its innovative approach to storytelling: it was almost all Show and no Tell. There was no opening screen full of backstory text, no long cutscenes with characters ‘filling you in’ on what had happened with no regard for what a person would actually say in their situation. The shocking moment when you first meet the U.S. Marines and discover that they’re out to get you too isn’t highlighted with someone saying “The Marines are under orders to kill you too” (at least not immediately). So there’s a wonderful few moments of confusion – was it an accident? Did they mistake you for an alien? No, these guys are too good for that. If they were shooting at you it must have been on purpose. Why would they do that? And then the truth dawns on you…

Many games, now, follow in Half-Life‘s footsteps. Cutscenes and flashbacks replace the briefing screens of old. The location design includes information that story elements can be deduced from, like computer terminals and newspaper clippings – or, more subtly, big welcoming banners displayed on the outskirts of ruined cities, human skeletons on the tables of alien medical labs, and so on.

Showing is still limited. While the author isn’t overtly giving you their own interpretation of events as with Telling, it still influences things, as they’re still in control of which details you are Shown and which you are not. So, they’ll anticipate the questions you’ll have, and try to include enough details that you can work out the answers; but they’ll often fail to anticipate your questions entirely correctly, and they might not provide satisfactory answers – or you might miss the answer that they intended you to find. If that happens, then there’s no recourse, as with Telling.

It’s the learning equivalent of a rail shooter: the author sets up the questions and answers for you in the hope that it will satisfy you, and they usually cover most of what you wanted to know, but there’s often a few unresolved issues – plot holes, actions that seemed out of character, and so on. Not to mention the “Don’t go in there!” effect: Who hasn’t watched a movie, seen the characters do something stupid, and thought, “Aargh! Why they hell didn’t they just do it this other way?”

That’s where games come in. Because games, as an interactive medium, add a third mode of exposition: Play.

Playing

When a creator lets you play with something, lets you engage with a mechanic and poke it and turn it over in your hands, then you gain the freedom to explore counterfactuals (“what if” questions). In a film or a book, if the characters take the left-hand door, then they will always take the left-hand door, no matter how many times you rewatch or reread; but in a game, you can take the right-hand door instead. Instead of being nice, you can be nasty; instead of being cautious, you can be gung-ho; and you can see what happens. For the most part, you can decide which questions you want to explore, and which ideas you want to put to the test, at your convenience. Through Play, you can discover not just the way it is, and how it comes to be that way, but also why it couldn’t be any other way – why the ideas that are simply Shown in other media are hard to vary without making them worse. (Or, sometimes, that it should be another way, as characters in films and books often don’t make the best decisions).

Every game, as you’d expect, contains some element of Play. The most remarkable ones are the ones that let you Play with unusual things: Façade, for example, lets you Play with human relationships; Dead Rising includes a mechanic that lets you Play a little with photography and capturing interesting and well-framed shots; and a game like Mirror’s Edge lets you Play with free-running and acrobatics. There are, of course, a huge number of games that let you Play with things like combat tactics, too.

Just as with Showing, however, there are limits. Often in a game you can’t take the right-hand door instead of the left-hand door, because the developers set the game world up that way, barring the door and not modelling the room behind it – but really, this just means that “which door gets taken” isn’t something that the developers are letting you Play with in that game. They let you Play with other things instead, like whether you throw a flashbang grenade into a room before you enter it, or whether you use a health pack now or later, or what happens if you press the button marked ‘self destruct,’ and so on.

Play is also subject to the developer’s interpretation, just as much as Showing or Telling, because the developer is free to distort and focus their modelling of real-world phenomena. A game about running for a political office, for example, might base its mechanics around making rousing speeches and designing good policy – or it might base it around bribing electoral officials and leaking scandalous stories about one’s rivals…

Who cares?

OK, so Play can be used for exposition purposes. So what? Isn’t Showing and Telling good enough already?

Firstly, consider that none of these exposition methods apply solely to game stories: exposition can be used to convey any idea, from “he is Luke’s father” to “you should hit the weak point for massive damage.” Any idea that you want the player to acquire will appear somewhere in this framework.

Secondly, while Showing and Telling are often perfectly fine and easy, Play allows you to convey deeper ideas, and it allows you to convey them more persuasively. For a person to be persuaded that an idea is true (or at least, the closest thing to true they’ve seen so far), any alternative ideas they can think of need to be dismissed as inferior – inferior because they don’t work, or they don’t fit the evidence, or they’re pointlessly complicated, or they’re too arbitrary, or whatever.

While Showing and Telling can convey an idea, they’re one-shot weapons: the creator decides what idea he wants to get across, and commits it to a medium that is ultimately read-only, in that, once published, his work will only ever communicate that idea. The audience consume that idea, perhaps with varying interpretations (i.e. my idea of ‘brave’ might be a bit different to your idea of ‘brave’), but if they don’t buy it, the author has no second chance to convince them.

When the audience is invited to Play with an idea, however, the author’s got a much more populated arsenal: the game can accept an entire range of approaches that the player might try out, developing them to their conclusions. If the player is Playing the game a particular way, and they think of a better way, then they can try that out. They can put all of their alternative ideas to the test, and discover what the actual problems are with each one in turn. When they’ve finished Playing the game, not only do they understand the idea that you set out to communicate, but they’ve also dismissed many of the alternative ideas they thought of, too.

Sid Meier famously said that “a good game is a series of interesting choices.” For something to be a choice, there must be multiple alternatives available to the player – and for the choice to be interesting, the alternatives must all be viable, to the extent that it could be worth trying any one of them. Considered against the role of Play in allowing the player to learn what not to do, it’s clear to see why Meier is right: if you don’t offer choices then you are merely Telling or Showing, rather than Playing; and if the choices are not interesting, it is because the player doesn’t consider the presented alternatives to be superior and in need of testing.

Practical ramifications

I invite you to consider which forms of exposition you’re using for which aspects of your game, and to think: are there things that you’re presently Showing or Telling that you could be Playing instead?

If you’re making a game about a demon hunter, for example, how are you communicating that the player has killed a demon? I’m going to guess that you’re Showing it – the player swings their sword, the demon spits up blood, and so on. The player will probably buy that – after all, it’s a simple Show that conforms to all the standard gaming imagery – but, if you wanted to let them Play with it, why not let them choose exactly where to stab and slice the defeated demon, based on a simple model of demon anatomy?

The best strategy is the one that leaves no living demons, while still taking the least time (because other demons are around and the player shouldn’t linger). As the player works their way towards that strategy, they’ll make mistakes, sometimes wasting time chopping up a thoroughly dead demon, other times finding themselves face to face with a pissed-off and wounded but not dead demon… but when they succeed, they won’t just be taking your word for it that they killed the demon, they’ll feel like they did it with their own two hands, because they’ve developed expert knowledge on the subject. It helps transform the player from a generic fighter into a real butcher.

Let the writing of your game – not just the story, but the themes and concepts you want to convey – inform not just the art and sound direction of your game, but the core gameplay design itself. Let it inform your balancing and tuning. Bake the message you want to send into the player’s own actions.

Play with Play. You’ll have more fun that way.

Why is that?

The Concept of Concept

Perhaps it’s just that the games were really strange ideas? I mean, there are some pretty wacked ideas out there. There’s this one game where you’re rolling a ball around the place, and stuff sticks to it, and you have to make the ball bigger and bigger… oh, you own that one? OK, then how about… this game where you’re a plumber, and little mushrooms and turtles are attacking you so you have to jump on them, and you explore castles and deserts and stuff, and rescue a princess or something… what? One of the biggest and most successful game franchises in history? Hmm.

People actually bought this.

Maybe weird isn’t a bad thing then. Maybe it’s the less weird, more realistic games that fail? Oh yeah, I heard about this one game that is basically just a ‘ordinary guy’ simulator. You have a little guy, and you make him go to work, and fall in love, and do housework, and all that stuff. Who’d want to play that? I mean, I do enough of those things in real life. What do you mean, it’s the highest-grossing game franchise in PC gaming history? Man, these consumers. They’ll buy anything, huh?

Game ideas can be as strange as you like, or can be so based in real life as to be a simulation of it. They can succeed either way. So, if it’s not because of the concept, why do some game projects fail?

Delta Concept Over Delta Time

I propose that you can start a game project with pretty much any concept, and end up with a hit. It’s not the concept that matters. It’s the process. The right process will take any concept and turn it into a hit – will ‘polish a turd,’ as they put it. It takes time – perhaps more than you have – and the worse your initial concept is then the more time it will take, but you can always get there in the end. The wrong process will founder and flail and flounder and fail.

Everybody knows what bits of this process looks like:

• It’s agile, and it’s agile for everybody, not just for programmers.

  Agile is usually pitched as a programming methodology, and it’s buzzwordy, but all it really means is “able to respond quickly to changes that are needed.” That’s an approach that applies to all parts of development.

  

  This cat is prepared to respond quickly to changes that are needed. Are you?

  I’m not sure that there’s much discussion of agile development outside of programming, at least by that name, but I imagine that many people have discovered things about it. For example: on my project, each character animation file has a full copy of the character’s geometry and rig. This makes changes to the geometry and rig quite painful, because each animation file needs to be updated so that things don’t get out of sync. I’ve been considering trying to find an alternative approach that uses Containers or XRef Objects to share one copy of the geometry and rig across all the animation files, making it much easier to respond to changes in the base skin – much more agile. Have any readers tried this?

  (If you’ve got other tips on how to be agile in parts of the project beyond programming, share them in the comments!)

• It doesn’t get attached to anything.

  This is almost a repetition of the previous point – being attached to things makes you less agile – but it bears repeating because people often underestimate the emotional impact of change. The artists spent two months working on a particular setting, and now you’re starting to think that the game would be better off without it – how do you think they will take that?

  You have to be prepared to “shoot your baby in the crib,” as they say.

  

  Sometimes, it's for the best.

  Don’t fall for the Sunk Cost Fallacy. Whether it’s a particularly clever bit of code, a particularly beautiful piece of artwork, or a particularly intriguing design concept – it may not be appropriate for the game. And that, at the end of the day, is what matters: that the game is a coherent, balanced, self-consistent whole, not just a collection of beautiful but ill-fitting parts.

  If you have to tell someone that their work is going to be cut, it can help if you can help them understand why it’s being cut. The better you can communicate your vision for how things should be, the easier it will be for them to agree with you. Of course, they might disagree that your vision is better, in which case you should listen carefully – they might be right, after all.

  Also, never throw beautiful things away entirely: maybe they’re not right for this project, but if you shelve them for now, they might come in very handy on the next project. It’s senseless to simply destroy work that you’ve already paid to develop, and the idea that the work is being shelved for future use, instead of being wasted time, can make it a lot easier on the creator.

• It actively looks for criticism.

  Everyone agrees that QA and play-testing is useful, but most projects do it infrequently, or put it off until really late in the project. Why? I think there are two major factors:

  The first is that the importance of criticism isn’t well understood. “The focus is on getting the game right – we can worry about the ways in which it’s wrong after we’ve done that.” But is it really always that easy to identify what’s right? A lot of the time, it’s fine, but every project I’ve worked on has seen disagreements within the development team about how particular things should be done – from the use of set-pieces, to the way a particular explosion effect should look. Lots of predictions are made, and lots of options are presented, with a fair case being made for each one. If, rather than worrying about getting things right, you focus on not getting them wrong, then you will still approach the best option, but you won’t spend any time agonizing over what that option is when really any of them would be perfectly valid.

  The second is that people are afraid of criticism because it means they might need to change the way things are going. And this, of course, is extremely silly. If you don’t test to find problems, the problems won’t just go away! It’s better to know about a problem – to know about a criticism – and then choose not to act on it, than it is to not know.

  

  Not an effective problem-solving strategy.

• Embrace what makes your idea different – and, conversely, don’t put too much effort into trying to beat other games at things that aren’t central to what you’re doing. If what makes your game different is its innovative approach to strategy and macro-management, then it’s fine that you display health bars in the same way as every other game. Nobody buys an RTS game because “it has a particularly innovative approach to displaying health bars.”

  Games, while young, have developed a lot of traditions, and those traditions represent the progressive refinement of an idea over multiple iterations. There are ideas that are cliche, sure, but they’re cliche because they were used in many games, and they were used in many games because they worked in many games. If you want to change one of these traditions, you should make sure you understand how your change will improve it; being different just for the sake of being different, without understanding how you’re making things better, makes it very likely that you’ll be making things worse.

The End Boss

There’s one colossal obstacle to all this. The experienced developers amongst you probably already know what it is. It’s the commercial realities of a project – the iron grip of the publisher.

The publisher, all too often, will not understand if you want to spend time and money making your project more agile. They do fall for the sunk cost fallacy, and will actively thwart your attempts to change the direction of the project if it’s too far away from their original plan, regardless of whether the direction is better. They will not fund testing and QA at the times that you need it, because the idea that testing and QA should be a continuous process is a comparatively new – and expensive – idea.

And worst of all, they will put all the emphasis on the original concept. Regardless of your team’s ability to take a concept and make it great, they won’t trust that. They’ll insist that you get the concept right on day one, so they can hold you to it. And when you don’t quite get the concept right on day one – because nobody’s perfect – they’ll keep insisting that you polish it and polish it, regardless how much of a turd it is.

The first publisher to understand that the success of a project depends not on where you start, but on how quickly and effectively you can improve, will produce a lot of hit games, and make a lot of money.

The real key to polishing a turd is being given the freedom to throw the turd away if you need to.

Until now, that is. Subversion is no longer my first choice, and I reckon that six months from now, I’ll never need to use it on a project again. If you’re using Subversion, I think you should probably stop using it too.

So, what’s wrong with Subversion?

Committing = publishing

The first big problem with Subversion is that there’s no separation between creating and publishing a commit. If your changes aren’t present in the centrally accessible repository, then they’ve not been committed. This makes commits pretty slow.

It also makes them susceptible to network errors. If I’m committing some large assets to a repository hosted in the cloud, it’s quite often been the case that the connection has just died somewhere along the line. If, while the upload’s been happening, I’ve made more changes to my working copy, then it’s difficult to repeat the commit: I have to carefully pick out my new changes and keep them separate from my old changes.

Furthermore, because my commits are published as soon as they’re made, I have to be very careful about what I commit. As soon as the commit is published, other people may check it out and try building it. So, I mustn’t make any commits that would break the build and stop other people from working. (Private branches remedy this somewhat, but this opens up a whole separate set of problems around merging, that I’ll come to later).

Lastly, if I can’t access the repository server – if it’s gone down, or if I’m working remotely – then I can’t commit at all.

Merges

While Subversion evangelists always emphasized the ease with which you could create branches, they were a bit more quiet when it came to merging those branches. If you’re going to make serious use of branches on a project, there are certain things that the SCM needs to be able to support:

• Files post-merge must correctly track history back along all merged branches.
• Tracking which commits have been merged from one codeline to another, so that you don’t risk double-merging.
• It should be easy to pull commits from one codeline to another.

Since version 1.5, Subversion technically supports all of these… but it’s all a bit fragile. For any folder that you’ve merged things into, an svn:mergeinfo property is set, recording the revision numbers of all the things that have been merged in (so that they’re not accidentally remerged, and so that future merges know where to pick up from). This more or less works – provided you don’t want to do things like merge code from other repositories – but there are many reports of it being somewhat prone to error. If your mergeinfo property has been corrupted somehow, it might be extremely hard to spot – and it’s not possible to regenerate the property by simply examining the codelines and commits.

File system operations

If you want to reorganize files in your working copy, you have to make sure that you involve Subversion in the appropriate manner, otherwise you risk doing things like interrupting your file history or committing changes to the wrong places. Filesystem operations like move and copy need to be ‘svn move’ and ‘svn copy.’

It’s not too hard to remember to do this if you’re working by hand, but increasingly we expect our tools to manage our files for us, and they don’t invite Subversion into the conversation.

What I’m using instead

I’ve switched to Git. It solves all of the above problems, as well as just generally being much, much faster.

When I first heard about Git, I didn’t really get it. Distributed version control seemed like a really bad idea – who’s copy of the code is authoritative? If bugs are logged against a particular build, how do you track down the code that was used to create it? The loss of things like sequential revision numbers seemed like a big drawback. And branching wasn’t something I ever used in Subversion, so I didn’t see why I should care about it in Git.

Why it’s better

It took me a while to realize the best way to think about this, which is: Screw the distributed bit. If you want to have a central, authoritative repository, you can still do that. It’s just that it becomes a matter of project policy, rather than a technical requirement; a particular repository is central and authoritative because you put it in the center and treat it as authoritative (or ‘bless’ it, in the terminology). You can mimic your present Subversion setup completely. What’s disconcerting is that you have to choose to do so, rather than simply having it fostered upon you.

Futhermore, Git explicitly models something that Subversion seems to miss: Every developer’s working copy is a private branch. The branch is created when they check out the code, it diverges from the central repository as others publish their changes, and it’s merged when the developer updates or commits their working copy. By giving the developer an entire repository instance on their own machine to store that branch in, it can actually be treated as a real branch, with multiple commits, a proper name, and so on. Some people ask: what if developers don’t push their changes to anyone else? To which I respond: What if they don’t check in their changes at all?

A common criticism leveled against Git was its usability – Git originated as a set of command-line tools for the Linux kernel hackers, and that heritage persisted. This argument was reasonable a year ago, I think, but the tools have matured now. I’m using Git Extensions as a general Windows-based Git UI, and it includes Visual Studio integration; on Mac, Tower is an excellent choice (Edit: And XCode 4, out today, includes Git integration out of the box). It’s true that I have dropped down to the CLI for some more involved tinkering – mostly relating to importing my Subversion repositories – but I don’t use it at all on a day-to-day basis.

There’s a little bit more terminology to learn – staging, pushing and pulling, rebasing, and so on – but it’s not that big a deal, and most of it’s intuitive. This isn’t a severe disadvantage over Subversion, anyway; I’ve seen claims that non-technical people can’t handle the jargon, but if you think that non-technical people aren’t already learning what terms like ‘checkout’ and ‘commit’ mean then you’re kidding yourself. Git’s UNIX-philosophy layers-of-hundreds-of-tiny-programs approach makes it very easy to build your own scripts and automation over the top to make things as easy as possible for your team, without forcing them to adapt a particular workflow.

Why it’s not

There are a couple of things I can see might be problems for people. I think they’re problems that can be circumvented or solved, either through project policy, or by waiting for Git to mature a little more, and they’re not deal breakers for me – but they’re worth considering.

Firstly, every developer will have a complete copy of the entire repository on their machine. Given how cheap storage is these days, I don’t think this is a very common issue, but I can imagine that some extremely large projects might find it prohibitive, especially if they’re storing lots of large assets. To give you some idea, one of my projects takes up about 900MB on disk (code + assets), and the git database adds another 430MB on top of that, for a total checkout size of 1.3GB; before I moved it to Git, its Subversion checkout was 1.85GB. As far as speed is concerned, though, Git’s speed barely drops for very large projects.

Secondly, because Git tracks files by their content rather than by their path information, it’s sometimes difficult to keep tracking history for binary files. Many changes that seem small – even changing a single pixel in a texture – can result in the content of the file changing dramatically when compressed, and if too much of the file has changed, Git might not recognize it as being the same file any more. It tries to be smart about this, and if you don’t change the filename then you’re usually OK, but occasionally it can break.

Thirdly, on the topic of binary files: while diffing and merging binary files is impossible just as much as under any SCM system, people tend to go for the next best thing, which is locking files to make sure that at least people won’t waste time making changes that can’t be merged – but that’s not something you can do out of the box with Git, as there’s no authority that can tell you which files are and are not locked. Most of the time this won’t be a big deal – most of the teams I know have fairly strong ownership of art assets, so you wouldn’t really expect two different people to be changing the same model – but it’s likely to bite you at the worst possible times, such as when everybody is rushing to get something done for a milestone.

Per-file locking is part of a more general issue, which is that Git doesn’t have much in the way of an access control mechanism: you can grant read access without granting write access, but either the entire repository is accessible, or none of it is. Sometimes it’s convenient to be able to set up a project in which certain directories (for example, ‘shared’ or ‘engine’) can only be committed to by particular people.

Personally, I’ve found it’s not too hard to work around this by simply using multiple repositories: I’ve got multiple ‘section’ repositories that have particular permissions set up, and then a central ‘master’ repository that pulls branches from each section. Git’s got no problem that the branches I’m pulling in came from entirely separate repositories – another result of the distributed approach – and they merge fine, provided I make sure that the directory structures in each section are set up appropriately. It’s a little less neat than approaches I’ve previously used that employed svn:externals, but it’s not been a big problem. Git does also have support for ‘submodules,’ but after reading the documentation several times I’ve still not been able to figure out whether they’re quite suitable for my setup – if anyone has insight into how they can be used for this stuff, I’d be interested to hear it.

These features are presently lacking, but Git does have support for hooks at various points in the SCM workflow; the ‘update’ and ‘pre-commit’ hooks look quite promising to me as a way of rejecting commits that break either locking policies or ACL requirements. Hopefully we’ll soon see somebody provide readily usable scripts that use these hooks to add in the missing features.

(Given these problems, there’s a fair case to be made that while Git is fine for source code, it’s not the right place to keep assets – you might be better off using a system more deliberately designed for assets, like Alienbrain. Personally it’s not affecting me – our art is contractor-developed, so it tends to change infrequently. Using text-based asset formats where possible does help, though).

Conclusion

In the month or so that I’ve been using Git, I’ve generally been impressed. My workflow has improved; I’m now making lots of tiny, logically self-contained commits, instead of the huge blob changes I was committing before, and I’m staging, committing, and then continuing to work without waiting for any tedious uploading. It’s encouraged me to experiment with trying out new features and new ways of doing things, using local branches to quickly spin off a safe environment in which to play – as well as ensuring that I’ve got somewhere I can keep the results of the failed attempts if I want to refer back to them later.

I can imagine that I might find myself using Subversion occasionally in the next few months, as third-party application vendors work to bring their Git integration support up to the same standard as their Subversion integration. I don’t think it’ll be long now, though. Subversion as a concept, as an architecture for an approach to source control, has been refuted.

Subversion is dead. It’s just taking a bit of time for the rigor mortis to kick in.

(Update: Very interesting feedback from lots of people on this. Read the comments below; you may also want to check out the discussions over at Hacker News and reddit for more).