One particularly tricky bug from that time stuck in my memory.

It was Wednesday when I got the email from the project’s technical lead.

Our memory profiling system had been spitting out a warning on one of our target platforms that indicated a memory trample. Both the technical lead, and the programmer who implemented the system were snowed under, so I was asked to take a look and find out what the problem was.

Now, at this stage there were two possibilities: either (a) there was a memory trample, but for whatever reason we could only see it on one platform, or (b) the memory profiler code is not working correctly on that platform.

My first step was to put a breakpoint on the line of code that spits out the warnings and run the code in a debugger on the target platform.

Execution stops, and I examine the code in some more detail. The code that fires the warning works as follows: When a pointer to memory is about to be freed it is passed to the memory profiler, which calls a “GetMemSize” function to determine the size of the allocation, then fetches a value from the last 4 bytes of the block and checks if it is the expected value. If the value read isn’t as expected a warning is printed.

The callstack does not make me happy. The code freeing the memory is my code – it’s my generic array class (like std::vector, but awesome). This is code I’ve brought with me and has been in production use on shipping projects for years. It is unlikely, in my opinion, that this code is broken, but not impossible – so I keep looking.

I look at the code using the array – is it doing anything untoward? The array class has assertions on the element access to trap bounds errors, but the iterators do not. The code using the array class does not (that I can see) use iterators. It uses: push_back, pop_back and operator[]. All calls with debug guards to trap misuse.

So I take a step back. The profiling code itself is quite new, so I look at it in more detail. I check where the trapped value is supposed to be set, and find something that might be a lead – a complex nest of preprocessor conditional compilation.

Our memory manager is layered – Micro-Allocator, Turbo-Allocator and then the system (vendor/OS) allocator. The memory profiler is currently turned off for allocations that go via the micro and turbo allocators, so maybe there is a mistake in the way the #ifdefs have been structured that means the value is being checked, but not set up at all.

This seems like a good lead at first, and I follow it for a while, but it turns out that the #ifdefs are all set up correctly. The allocation in question is going via the system allocator, and the profiler is inserting it’s pad value at the end correctly.

Back to square one.

The problem is happening at start up, which is deterministic, and the OS uses a fixed memory model that means allocations always end up at the exact same address run after run. Time for some hardware breakpoints.

I run the game again, find out the address of the value the system is complaining about, and set up a breakpoint-on-write, then restart the run.

I filter my way through the breakpoints hits, checking if what is writing there should be, and find nothing. Nothing relevant at all. Nothing writes to that byte. It is unused. The value is not touched.

What on earth?

So I backtrack. Again. Double check my numbers. Did I have the right address? Yes. Break point the set up function. Breakpoint the creation of the array. Single-step though lots of code. The allocator is setting up the boundary value – but not in the same place as it is being fetched from!

So, a moment of clarity – it isn’t the value at the end of the array getting stomped, but the address the code is getting for the end of the array that is wrong!

Perhaps it’s the value at the start of the array getting stamped?

New hardware breakpoint, at the start of the array this time. Run. Watch were we stop. The breakpoint hits in system malloc and system free. It makes sense. The callstack looks right. Hits in malloc, then free, malloc, free, malloc, free.

Nothing that shouldn’t touch that memory was touching it. I see a bunch of mallocs and frees and then the system falls over.

What on earth is going on?

And then I notice it. The system that is using the array is an XML parser. There isn’t one array – there are loads of them.

This is the mind boggling part: The moment the call to free stamps on the size value of array ‘A’ was not the moment that the memory owned by ‘A’ was freed. It was when some other array was freeing memory.

So: We have a two blocks of memory allocated by the operating system. At the start of each one, we have a 4-byte int for the total size of the block. At the end of each one we have the guard value. We know the location of the guard value by checking the size of the memory block, but when we free ‘B’ the size of ‘A’ changes.

Nice.

Lets have a look at GetMemSize. For the platfrom in question GetMemSize(ptr) returns ptr[-1]-5.

Note the vendor in question does not provide a memstat/memsize function so this is the result of deduction and reverse engineering.

And it’s wrong. The version of the function for another platform by the same vendor has a hack of a work around that does:

1

if (memSize % 2 != 0) memSize = memSize + 1;

Oh Hello.

So, it seems like the system memory manager uses those first 4 bytes for more than just the size. After all, the memory allocations are all 4 byte aligned, so those first 2 bits aren’t really needed, and it seems like they are used for something else, perhaps to do with the availablity of neighboring blocks for use with realloc?

Who knows.

But the fix now is clear, mask off the bottom bit, and we are good.

1

UInt32 memSize = (Ptr[-1]-4)&~1;

Perhaps I should’ve masked off the bottom 2 bits?

Who knows, it all seemed to work after that. And the game shipped in December without major bugs.

This story is a rewrite of one I originally posted here: http://www.hulver.com/scoop/story/2009/3/20/133915/125

You can read the original article (Using Typedefs) if you want. I’d like now to try to distil why I think this advice still holds up, 10 years later, even if you aren’t using templates, the STL or C++.

Programming is a form of communication. As a programmer you write for two audiences: You write for the compiler, and you write for the future other programmer.

For most programmers who are game developers, that future other programmer is someone already on your team, or as yet hired future team member.

Even for the solo programmer, the indie developer, working from home, will on occasion return to code written many years ago, and use it as the basis of new work.

Who knows, if that solo project goes viral, one day you might be hiring people to port it to the latest console or handheld device.

Any professional writer will tell you, one of the many important principles of good writing is to consider your audience. This applies to programming as well.

Now, I’m not going to say that the human audience comes first. A programmers primary job is to get the computer to Do The Right Things, and sacrificing how clearly you are instructing the computer in order to better communicate with the human reader may well be a questionable trade off, especially when performance is an important factor in the engineering goals.

But even with the primary function of writing code that works dealt with, there is often room to improve how literate your programming is, and one of the most powerful tools as your disposal as a programmer is giving things good names.

A good type name can tell the reader not just what a type is for, but also what it is not for. Consider a Name vs a String, or a NameList vs a StringContainer.

So, please, think carefully about how you name your
types, your functions, methods, and variables, because it’s not just the compiler reading the code.

One day it might be me.

There are however, some problems that C/C++ simply can’t solve well. Sometimes the best tool for the job isn’t going to be your favourite tool, and rather than try to twist the tool you like to the job you need to do, you should leave your comfort zone and try something new.

This article explores once such problem, and what happened when I left my comfort zone.

This problem came up on a C++ mailing list I subscribe too:

I’m working on a problem where integer multiplicative overflow is plausible and I want to check for it.

The poster had already constructed a solution that looked like this:

1

overflow = (a!=0 && (a*b)/a!=b)

Given that mathematically: (a*b)/b == a, the poster was concerned that the optimiser would spot that and eliminate the test. He also wondered if there were better ways of doing it.

The question prompted a fair amount of discussion. The general consensus was that the test worked, and that the compiler would not optimise it away.

Some other solutions were proposed, none of them unambiguously better.

But it got me thinking. Given that every CPU architecture that I have any experience with has an overflow bit, shouldn’t we be able to check that? A divide is relatively expensive, and the test includes a compare and a branch to set a boolean that will then most certainly be used in a compare and a branch.

Wouldn’t it be better to just check the overflow bit?

So I started with this as my baseline:

1
2
3
4
5
6
7

// returns true (non-zero) if the result of a*b is within the valid range of an integer
// the value of the multiplication is stored in the address pointed to by result
BOOL SafeMultiply( int a, int b, int* result ) {
    int c = a * b;
    *result = c;
    return (a==0 || c/a==b);
}

Which can be used like this:

1
2
3
4
5
6

if (SafeMultiply(a,b,&c)) {
   // do something with c
}
else {
   // report an error   
}

I then spent about an hour relearning some assembly code. I knew it could be done, but the specific opcodes, mnemonics, and inline-asm syntax were fuzzy. It’s been probably 16 years since I actually wrote any assembler. Debugging it, sure, looking at the generated code, yes. But writing assembler just hasn’t come up in my work for a very long time.

I wrote the first version on my lunch break at work, so I used MSVC, and thus MASM syntax inline assembler.What I wrote looked something like this:

1
2
3
4
5
6
7
8
9
10
11
12

int SafeMultiply( int a, int b, int* result )
{
    char good; 
    __asm{
        mov eax, a;
        imul eax, b;
        setno good;
        mov ebx, result;
        mov dword ptr [ebx], eax;
    }
    return good; 
}

MASM made it easy. I was pleased. I should have stopped there. What happened next, however, was that I decided to write this article for #AltDevBlogADay.

When I write for #AltDevBlogADay, I do it at home. On my laptop. Which doesn’t have MSVC. The #AltDevBlogADay readers deserve, I thought, more than just MASM. They deserve a GNUC compatible version. They deserve that, and a comparison of the code generated, with inline assembler and pure C compiled with -O3.

What I learnt, my friends, is that the GNUC inline assembler syntax is evil. Pure evil. Evil genius, perhaps. Remarkably powerful in ways I hadn’t even considered possible. But still evil.

It is not for the faint-hearted, and might not be the optimal solution, but here it is:

1
2
3
4
5
6
7
8
9
10
11

int SafeMultiply( int a, int b, int* result )
{
    char good; 
    __asm__(
        "imull %2, %3;\n\t"
        "setno %0;\n\t"
        "movl %3, %1;"
            : "=r" (good), "=r" (*result) : "r" (a), "r" (b) 
    );   
    return good;
}

Note, that I’ve shown here each version on it’s own. In reality it’s all mashed together with appropriate _MSC_VER and __GNUC__ #ifdefs.

I shan’t waste your time trying to explain the syntax, if you’d like to get into that (and I recommend you do, it is fascinating), then the two pages I give at the bottom of this article could be a good place to start.

Anyway, I promised a look at the generated code. This was compiled on OS X with GCC, using -O3 and the -save-temps command line. I have stripped out the function call header code, and parred it down to only what is relevant: the different bodies of the functions.

The C implementation generates this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

	movl	%esi, %ecx
	imull	%edi, %ecx
	movl	%ecx, (%rdx)
	movl	$1, %eax
	testl	%edi, %edi
	je	L4
	movl	%ecx, %edx
	movl	%ecx, %eax
	sarl	$31, %edx
	idivl	%edi
	cmpl	%esi, %eax
	sete	%al
	movzbl	%al, %eax
L4:
	leave
	ret

And the inline assembler (GNUC version), generates this:

1
2
3
4
5
6
7

	imull	%edi, %esi;
	setno	%dil;
	movl	%esi, %eax;
	movl	%eax, (%rdx)
	movsbl	%dil,%eax
	leave
	ret

Now I’m no expert, and I’ve not timed it, but I can see that the latter is smaller, and I think its a safe bet that it’s also faster.

Thanks for reading!

• A Brief Tutorial on GCC inline asm (x86 biased)
• GCC-Inline-Assembly-HOWTO

With thanks to everyone who gave me feedback on my initial draft, and especially to Fabian Giesen and Don Olmstead for their constructive critisism, and for the title suggestion.

Since I’m now working with Unreal Script some of the time, and the plug-in is open source, I decided to see if it would be straightforward to extend that functionality. My assumption was that it would be pretty easy, and in the end it was.

But that isn’t really what I want to write about now. It is what that work led me too that I think is interesting.

See, turning on the syntax highlighting for Unreal Script wasn’t too hard, but as it stood – as it still stands as I write this – the keyword colouring code in MetalScroll still only recognises C++ keywords, because when I dug deeper what I discovered was fascinating and took me off on a total tangent.

The initial work was done on office time – it was a worthwhile effort to boost productivity of multiple teams where I work by improving a popular tool, but then I dug a little deeper: How hard would it be to make it use an Unreal Script specific keyword list when highlighting Unreal Script? All I have to do is find the list of keywords, and swap it out for a different list depending on filetype. Right?

So away I go reading the code: RenderText calls IsCppKeyword, which has a list — but — hang on — it does what? But what if, oh, wait, so … but what if … oh, then what does that hash do? Oh. That’s neat. It’s got to be fast. Smart guy. So did he, really, make that by hand? I scroll up. Ah. So what’s this: /* C code produced by gperf version 3.0.3 */

Interesting.

Google: gperf.

Result: It’s a GNU tool for generating hash functions.

GNU gperf is a perfect hash function generator. For a given list of strings, it produces a hash function and hash table, in form of C or C++ code, for looking up a value depending on the input string. The hash function is perfect, which means that the hash table has no collisions, and the hash table lookup needs a single string comparison only.

That, right there, is gold. Right away I think back, all my years in the industry, and almost all the problems I’ve ever seen with asset names and hash functions, would have been solved. Like magic. Beautiful. If you can feed it a full list of your asset id names, it can spit out a hash function with guaranteed no clashes.

If you use asset names and hash them, and I’m betting a lot of people do, because I’ve seen it a ton of times, then you could to be using this.

But I didn’t stop there.

That’s a pretty tight way of spotting keywords. A fast hash, and a string compare.

But we still need to look at some of the characters in the input string more than once. There is an extra 1 to 3 reads from the input string to build the hash, before the character-by-character string compare.

If we are generating code for a fixed list of keywords (which is reasonable) then that could be improved on, I’m sure.

So I thought about it, and came up with this: If we treat the keyword list as a possibility tree, where each character narrows down which possible keywords it could be, we can stop as soon as we know it’s not a keyword, and if we are still “in” the tree when we hit the end (of both the keyword and the input string) we have a winner.

I started thinking through ways of building the tree structure with look up tables in static data, and then it dawned on me: since it’s generating code, the tree can be directly encoded in the structure of the source code. We can let the compiler do something C and C++ compilers tend to be very good it: Compiling switch statements.

So that evening I went home and I wrote this: gen-is-keyword-fn, and stuck it up on google-code.

It’s not finished yet – I’ve been absurdly busy recently (this post is being hastily rattled out at the 11th hour) – but it shows some initial promise. I haven’t timed yet how well it performs against a gperf based keyword identifier, since I haven’t yet implemented the generator for functions that accept keyword candidates of a known length, but initial testing vs a textbook binary search of a sorted list, give some very promising results: Specifically, with the switch-case coming in at 10%-20% of the execution time of the binary_search.

So, writing code that writes code can pay off. And is fun.

Which is point of my story, I think. For now.

How do I stop this code generating a:

MyClass(123) : warning: ‘result’ may be used uninitialized in this function

Without pragma’ing the warning off ?


T Remove(int key)
{
  // !!! We need a pragma here or something to tell the compiler to ignore the
  // fact that "result" is not initialized.

  return result;
}

To which I answered:

Burn it with fire.

Then, more seriously: As the code stands you can’t fix this. It is a fundamental error in the interface design. Presumably this is a member of a container class template that deals with value types.

In my opinion the method in question should be changed to one or other of:

// returns true on success, contents of result will be set to the value
// returns false if key does not exist, the contents of result
bool Remove(u32 key, T* result);

Or

// where, if key does not exist, return value "default"
T Remove(u32 key, const T& default);

Or if Remove is widely used and changing the signature is too big a change, then consider have the “default” value specified during construction of the object.

Of the three possible solutions, I prefer the first.

Note the use of mutexes implies this is used in a multithreaded context, so double-checking “exists” before “remove” is unreliable (as well as inefficient), and so there is no way to check if the Remove operation was successful when using this container with built in types (int, float, etc) with the current interface methods.

I created an AI & tournament framework for it called “C++FIGHT!” around 10 years ago and used it as a sandbox for exploring turn-based game AI techniques, such as MiniMax, AlphaBeta, and heuristic tuning with genetic algorithms.

I had briefly experimented with implementing it as a playable game in Java about 8 years ago, but lost interest in Java as a language and never got further than sprites that could be moved around with the mouse.

So when, after finishing Syd Sym, I decided that I wanted to try making a browser game (but not with flash) as my next hobby project, it seemed like the natural thing to pick up unfinished business and finally give the AI I’d made for Pennywise some human opponents to play against. The realisation that the art requirements would be minimal was the icing on the cake.

The First Month
Initially the progress was slow. I spent long evenings reading poorly written HTML, CSS and Javascript tutorials, and figuring out very simple things, like how to implement drag and drop. I made a deliberate choice not to use <canvas>  but to work within the HTML document structure (I figured I’d learn more that way, and I think I was right). So I created a simple document structure that mirrored the games conceptual structure, a <div> for the game “table” containing a <div> for each player, and a <div> for the pool of change in the middle. I set up the HTML so they basically stacked on top of each other, and I placed a <div> for each type of coin in each of the player <divs>.

I then deleted huge chunks of it, wrote a javascript port of the game state objects from cppFIGHT and generated the HTML in the onload event handler for the page.

After which I researched how web apps normally implement drag and drop and used what I had learnt about onmousedown, onmousemove and onmouseup to implement my own drag drop system for picking up, moving coins and detecting which container div they had been dropped onto.

Each time a coin was dropped, I deduced how that related to a move in the game, applied an appropriate change to the game state, and regenerated the innerHTML from that.

At that stage I was becoming confident I’d be able to turn out a working game in a reasonable amount of time, so I roped an artist friend of mine into creating some original art for the game.

Enthused by the promise of real professional art for the game, I finally got around to creating a mercurial repository for it on my laptop.

The Second Month
With the basic framework for interacting in place and the promise of art in the not too distant future, I dived into the next major piece of the puzzle: the opponent player. As previously mentioned, I had already written some simple AIs for this game in C++, but a major unknown for me was Javascript performance. Would I be able to do a fully fledged AI, or would that result in the dreaded “A script on this page may be busy, or it may have stopped responding” message? I started by implementing a very simple non-searching opponent player strategy from the collection I had on the cppFIGHT repository, which worked, but with the game state updating the HTML immediately, tracking what was happening in the game was hard.

I spent about a week implementing a system for moving the coins (liner interpolation) to show the opponents moves, tidying up the code, testing and bug fixing before implementing any other AIs.

I then implemented 2 more rule-based AIs, the ones that eventually became the “Shipman” (easy) and “Ensign” (Normal) difficulties, along with support for AI vs AI matches and support for the human player to play 1st or 2nd, at either top or bottom position on the table.

1st mock up

To implement the scaling, the coin <div> lost it’s background style and gained an <img> tag, the click detection had to be modified to traverse the object graph and the new width (scale) had to be applied to the coin as it moved. Determining the correct scale to create a perspective illusion was simply a matter mapping the coins position on the trapezoid of the table:

1
2
3
4
5
6
7
8

function GetScaleOnTableAt( height )
{
	// where height = 0, result = TABLE_VERTICAL_RATIO
	// and where height = TABLE_HEIGHT, result = 1
	var v = 1.0 - TABLE_VERTICAL_RATIO;
	var f = height / TABLE_HEIGHT;
	return TABLE_VERTICAL_RATIO + f*v;
}

With the perspective effect proven, I decided that rather than rebuild the innerHTML for each game-state changing action, it would be better to manipulate the DOM to keep the HTML and game-state in sync. This gave a consistency of layout across turns that I think makes the game feel more “solid” and “physical” but bugs where the DOM didn’t match the game state were a major source of headaches for the remainder of the project.

Up to this point I’d done all of the development work using Safari as my test browser, so it was a huge surprise and a great relief to discover that I only had 2 minor issues to fix when I tested it in Firefox. One extra null check in the mouse event hander and a minor change to the CSS to anchor the img style with {top:0;left:0};.

Once the game worked in both Safari and Firefox it became clear that differences in browser rendering and Javascript performance meant a slightly more sophisticated approach to animation timing would be required to keep the movement speed of objects consistent. Changing the movement offset code (simplified here for brevity) from this:

1
2
3
4
5
6

	var l = Math.sqrt(dx*dx + dy*dy)
	var ox = dx / l;
	var oy = dy / l;
 
    	element.style.left = (left-ox) + 'px';
    	element.style.top = (top-oy) + 'px';

To this:

1
2
3
4
5
6
7
8
9
10

	var t = new Date().getTime();
	var elapsed = (t - oldTime) / 10;
	oldTime = t;
 
	var l = Math.sqrt(dx*dx + dy*dy);
	var ox = elapsed*dx / l;
	var oy = elapsed*dy / l;
 
    	element.style.left = (left-ox) + 'px';
    	element.style.top = (top-oy) + 'px';

Over the next week I tested a lot and fixed minor bugs as I found them, but my activity on the project was tailing off as I waited for the art to arrive.

Around halfway through the second month Roger sent me his first style-concept sketch:

Art Style / Concept Sketch

Around the same time I decided to implement support for the player to move back to the select game menu using the history.pushState / window.onpopstate API. This didn’t go well and ended up being replaced with the argument passing via the URL you see in the game now.

In the last week of the second month I split the opponent player code off into it’s own ai.js file, and implemented an AI based on the MiniMax with AlphaBeta prune that I had implemented for cppFIGHT. I first implemented the MiniMax with game state copy-and-modify (without a prune), and timed it in each of Safari, Chrome, and Firefox.

Mini Max implemented in Javascript, unpruned search to a depth of 4, timing results: Firefox = 224ms, Chrome = 306ms, Safari = 793ms.

I then tested a speculative optimisation where the game state is not copied, but after each recursive step the move (a modification to the game state) that was begin evaluated was un-done. This proved to be more efficient in all 3 browsers, so I kept the change.

Replaced creation of new game state per move with apply/undo logic: Firefox: 148ms, Chrome: 175ms, Safari: 424ms

Surprising twist in Javascript performance test, bump search depth to 5 and this happens: Chrome: 883ms, Firefox: 982ms, Safari: 2567ms.

I then implemented the alpha beta prune, which gave an order-of-magnitude performance improvement and allowed the search to look enough moves ahead to provide an interesting challenge to most players.

So, is it turns out an alpha beta prune is SUPER EFFECTIVE for this game. AI Decisions that took 2.5s now take 5ms. \o/

The Third (And Final) Month
It was at this stage I received the finished art and discovered I was on the receiving end of Roger’s wicked sense of humour. I was the pirate. Nice. I swallowed my pride, accepted looking silly on the internet is part of life now and got on with finishing the game.

Sorting out the layout to fit the new art, CSS changes and testing on multiple browsers took the best part of a week. At this stage of the project I was desperate to finish before I went on holiday and worked every evening. The biggest chunk of work came from the different sized coins in the final art, up to this point I had assumed all the coin types would be the same size on screen and had written most of the code with that assumption.

The following week was spent on fine tuning appearance and AI difficulty. Discovering the use of CSS and tiny divs to give non-rectangular text flow (for the scroll) was a great moment:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<style> .lW { float: left; clear: left; height: 16px; } </style>
<script>
function InjectScrollFlowElements()
{
	var scrollDiv = document.getElementById('scroll');
	var innerHTML = scrollDiv.innerHTML;
	var prefix = "";
	var th = 280;
	for(h=0;h<th;h+=16)
	{
		prefix += "<div class='lW' style='width:" + Math.floor(21*Math.sin(h/th*Math.PI)) + "px;'></div>";
	}
	scrollDiv.innerHTML = prefix + innerHTML;
}
</style>

The last week before the “public release” of the game was mostly spent testing it and asking friends to try it out. The final code fix was a redesign and rewrite of how placing coins on top of each other, so that the hidden coin cannot be picked up, would be prevented by the game. The final Hg commit was a set of single pixel CSS layout adjustments.

After announcing the game on twitter I went on holiday for 10 days.

It being an HTML/CSS/JS game, the full source code is of course available if you are interested. I have left it uncompressed and unobsficated. The whole thing comes in at less than 2000 lines of (poorly written) code, including white space.

Postscript
Since returning from my holiday I’ve added a “Tweet” button and a “Like” button, I also centred the whole game game in the browser window.

Using the “HTML5″ toolset was an interesting learning experience. Javascript is a more powerful language than I was expecting, though I did miss certain C++ features, especially compile time diagnostics and strong typing!

I hope you’ve enjoyed reading this and I hope you enjoy the game. If you have any feedback on either, please feel free to comment below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

struct PascalString
{
    size_t mLength;
    char* mBytes;
 
    PascalString(const char* c_string)
        : mLength( strlen(c_string) )
        , mBytes( new char[mLength] )
    {
        memcpy(mBytes, c_string, mLength );
    }
 
    ~PascalString() {
        delete[] mBytes;
    }
 
    // more stuff ...
};

The essence of this class is that it is a data structure that stores a pointer to a block of memory, and a count of how many elements there are in that data block. It also has construction and destruction behaviour implemented, so that the book keeping work of acquiring and releasing of the memory is handled automatically for users of the class.

An example of how this type might be used follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

ostream& operator<< (ostream& out, const PascalString& s)
{
    for (int i=0;i!=s.mLength;++i)
        out << s.mBytes[i];
    return out;
}
 
int main (int argc, const char * argv[])
{
    PascalString* p = new PascalString("Hello, World!\n");
 
    std::cout << *p;
 
    delete p;
    return 0;
}

Note that the object is created on the heap, and destroyed manually, with the “new” and “delete” operators. While for such a simple case p could have been a value object placed on the stack, the use pattern shown is more typical of production code, since the lifetime of game objects is usually more complex than this, and the allocations made are entirely relevant to the discussion!

So what’s wrong with this code? Nothing, really. It’s perfectly fine. If you are happy with the trade offs that have been made. The compromise implicit in the design.

This class trades space and time for simplicity of use and flexibility. This basic structure can be fleshed out to be a fully fledged value type. We could, from this starting point, implement assignment, concatenation, and so on and so forth.

These are perfectly fine trade offs, until they’re not. Optimisation is often about fitting the functionality implemented as tightly to the requirements as is feasible. If your data holding object is only created and interrogated, not modified (a situation that vastly more common in games development than is typically assumed), if the amount of data is known at the time the object is created, then the flexible array feature of C99 gives us a language level tool for creating code that more tightly fits that requirement.

Section 6.7.2.1, paragraph 16 of the C99 standard describes flexible arrays as follows:

As a special case, the last element of a structure with more than one named member may have an incomplete array type; this is called a flexible array member. In most situations, the flexible array member is ignored. In particular, the size of the structure is as if the flexible array member were omitted except that it may have more trailing padding than the omission would imply. However, when a . (or ->) operator has a left operand that is (a pointer to) a structure with a flexible array member and the right operand names that member, it behaves as if that member were replaced with the longest array (with the same element type) that would not make the structure larger than the object being accessed; the offset of the array shall remain that of the flexible array member, even if this would differ from that of the replacement array. If this array would have no elements, it behaves as if it had one element but the behavior is undefined if any attempt is made to access that element or to generate a pointer one past it.

So instead of having a pointer as the last member of a struct, we can declare a flexible array member, like so:

1
2
3
4
5

struct PascalString
{
    size_t mLength;
    char mBytes[];
    // ...

The array at the end of the structure allows you to allocate a variable-sized block of memory in which to store the struct, and the array elements, thus avoiding the run-time execution cost of an extra pointer dereference.

If you apply the sizeof operator to this structure, the array adds nothing to it’s size (alignment padding not withstanding). To get the true size of an object of this type, you need to consider the array size as well.

So now to complete the picture, we need to provide a new way to create and destroy objects of this type. Since the amount of memory required to allocate the struct itself now depends on the size of the data stored, and the compiler can’t rely on sizeof semantics, some of our construction logic has to be pulled outside the constructor, and the use of “new” and “delete” is no longer appropriate (you can make them private, to ensure they are never used “accidentally”):

5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

    // ...
    static PascalString* Create(const char* c_string) {
        // measure:
        size_t length = strlen(c_string);
        // allocate enough memory for the "struct" and the data it manages, in one block:
        PascalString* result = (PascalString*)malloc(sizeof(PascalString) + length);
        // initialise the members 
        result->mLength = length;
        memcpy(result->mBytes, c_string, result->mLength );
        return result;
    }
 
    static void Destroy(PascalString* p_string) {
        free(p_string);
    }
};

Thus creation is changed for the user like so:

10
11

    // PascalString* p = new PascalString("Hello, World!\n");
    PascalString* p = PascalString::Create("Hello, World!\n");

And destruction becomes:

14
15

    // delete p;
    PascalString::Destroy( p );

If you already hide your object lifetime management behind some kind of factory interface then there should be very little code to change. If on the other hand, your game is littered with calls to new and delete, then it’ll be more work to apply this optimisation.

So what benefits can we expect to see from a change like this? To demonstrate, I created a simple allocation tracker. For the trivial Hello World example, I got the following results: Unoptimised, Cumulative & Peak: 2 Allocations, 32 Bytes.
With Flexible Array Optimisation, Cumulative & Peak 1 Allocation, 32 Bytes.

So as expected, we have halved the number of allocations required to create objects of this type on the heap. Given sizeof(PascalString) is 16 bytes unoptimised, and 8 bytes with flexible array, and that the string “Hello, World!\n” is 14 bytes long, we have also learnt that the allocator for my operating system appears to allocate in 16 byte blocks.

In order to give you a clearer picture I also wrote a test function that creates 1000 objects of various sizes and puts them into an array. In this test the unoptimised version used 1032192 bytes, and the flexible array version used 1016192, a saving of roughly 2%. This measurement doesn’t include any savings in memory from the reduced allocation count, which depends very much on how your memory manager is implemented. Allocator book keeping can be significant when working with small objects. Given the per-allocation overhead can be anywhere between 2 – 16 bytes (or more, but lets be reasonable here), the true savings in memory use are closer to 3%, or 32k.

Now just think what you could do with that extra 32k!

The cumulative time savings from reduced allocation count, and the improved data locality of keeping the count adjacent to the data in memory are left as an exercise for the reader.

Disclaimer: While the technique described in this article has been successfully applied as an optimisation in a production environment, and was a critical part of getting a game running inside the fixed memory limits of a console platform, please keep in mind the effectiveness of this approach is entirely dependant on the existing code base and data set. Your milage may vary.

Given that I’m letting my developer account expire, you may have already concluded that the game was not a run away smash hit, and you’d be right. The paid version is bought, on average, just under once per week, and the free version (with iAds) is downloaded around 100 times per week. The iAds earn me around $6 a month. After developer fees and incidental costs, the game made me a grand total of £58.78 gross profit last year.

But the App Store is supposed to be a gold mine! Free money for developers! What went wrong? Why aren’t I writing this on a sold-gold laptop and wearing a money hat?

1. I was too slow. I missed the gold-rush. The development itself wasn’t particularly drawn out, but I really didn’t even start until the iOS game scene was already well established. At the time I released Syd Sym, I am told, there were literally hundreds of new apps released every single day.
2. I massively underestimated the effort required to market a game on iOS. As a behind the scenes game programmer I had no industry press contacts to lean on (and still don’t really). Getting reviews was excruciating. I sent out hundreds of emails, and was largely ignored, an experience I found almost monumentally frustrating. I doubt most of my press releases, review requests, tweets, forums posts, and begging emails were even read. I do of course understand why, 100s of new games a week, most of them rubbish. Why expect mine to be any different? I actually still don’t really know what I could or should have done differently to get any attention, but there you go.
3. The first review had a damning headline, and stayed at the top of the google search for the game for a long time. I’m not sure that really was something I did wrong (Because I released too soon? Bad press-releations? Poor SEO skills?) but it still sucked.
4. The game was, in all honesty, a poor fit for the way people play iPhone games. It simply takes too long to finish “one game”. It may be crude, but if you can’t play one “round” of an iOS game in the time it takes to poop, then you won’t be a hit.
5. What the game was was too open, and thus confusing for many players. I had done my best to make the game “easy to play, hard to master”, but since a play through of the game takes a long time relative to what’s expected on iOS, I think a lot of people only saw the “easy to play” bit (passing levels), and missed the deeper strategies required to master the high score tables.
6. The collect’em’up replay value aspect of the game was too subtle, and since early versions of the game only told you how many combos you had collected, with no way to see which ones you’d got already, this aspect of the game was overlooked.
7. The tutorial was too hard. Many players could not get past the last level of the tutorial, and thus failed to unlock the full game. Even after realising this, I failed to settle on a solution I liked, so the problem was never fixed.
8. I localised too soon, leaning on friends around the world to translate the game text left me in an awkward situation with putting out updates that added too or modified in game text, which discouraged me from iterating on in game messages, help text and tutorials, and left me locked in to design and UI choices I had made early on for fear of having to remove a localisation.
9. I bought Facebook Ads. At first it seemed like a hilarious free win, since you pay for clicks not page views, and I got thousands of page views for almost nothing. But actually, in retrospect, I’m fairly sure that it was a stupid waste of money, and didn’t result in a single additional sale.
10. I made two versions, a paid download and a free version.  In retrospect I probably should have made it free, with in-app-purchises (or maybe just accepted it as loss-making hobby project, and kept it 100% free).

An utter disaster, right? Not really, in all honesty, I am glad I did it, and may make another iOS game in the future. So what went right? This list is, I must admit, more “why I’m glad I made it” than “the things I did right in making it”, but I hope it’s useful or interesting anyway:

1. I kept my costs absolutely rock bottom, and didn’t quit the day job. Even with the lacklustre reception, I avoided losing my shirt by keeping it a hobby project.
2. I had fun, it was interesting, and entertaining, and the time spent developing Syd Sym pretty much weened me off of my WoW habit.
3. I got interviewed by Joystiq.
4. The game got more positive reviews than negative reviews.
5. I made something that someone really liked. I know this, because the top high score on game centre requires frankly surprising level of dedication to achieve.

Over all, although I can identify more “bad” things than “good” things, it was a worthwhile experience, and a game I’m glad I made. I am not sworn off of iOS or mobile at all, but I’m trying to explore and learn new things, so for now (in my hobby-time) I’ll be focusing on my writing here, and finishing my browser hosted html/javascript game.

MetalScroll

http://code.google.com/p/metalscroll/
MetalScroll is a free Visual Studio add-in which replaces the editor scrollbar with a graphic representation of the code. When working on my Laptop, (a PowerBook running OS X), using XCode or TextWrangler, this is without a doubt, the single thing I miss the most.

Beyond Compare

http://www.scootersoftware.com/
Beyond Compare is a file diff/merge tool for windows that I discovered whilst working on the GTA “Double Pack”, work that, for various reasons, involved a lot a diffing and merging of sections of large codebases. Despite the abundance of free and commercial tools that provide equivalent functionality, many of which I’ve tried at various times, Beyond Compare has been my 1st choice for heavy duty diff/and merge jobs for over 5 years. Highly recommended. Also pure gold for log file post-mortum work, handling even very large files without a hiccup.

Code Collaborator

http://smartbearsoftware.co.uk/codecollab.php
Code Collaborator is a web based tool for performing and managing code reviews. If you have ever thought code reviews sound like a good idea, but that the gritty details of managing the whole process get in the way, then this tool is well worth investigating.

Why not post a comment telling us about the tools that make your life easier.

It’s a tough world out there. People are not nice.

You’ll feel as if you’re waging war on the set of The Muppet Show.

If you think this sounds a bit repetitive, you’re right…

The texture work is blurred and repetitive. The game is saturated by grays. The particle system is lacking. The lighting effects are underdeveloped.

I’ve been playing it for over an hour nonstop and I just don’t get it.

In times like this there aren’t just a lack of words at the situation, but a lack of faith in people, because whatever shit-hole idea you think you can attach to a license in order to gain revenue, it’s nothing compared to what game devs that will do anything for a dollar are capable of.

All quotes from reviews of games I’ve worked on. Were those comments fair? Some of them, to an extent. In their own way. As hard as it can be to swallow, there are lessons to be learnt from all criticism. Even if you don’t always agree with it.

A Duck

If you can’t look back at your past self, and shake your head in disapproval of your own naïvety… If you can’t look back on your older work, and know with confidence what you did wrong… and what you’d do differently next time, then you are not growing. You are not advancing in your art. If you don’t swim, eventually you sink.

Recognising your own mistakes, and learning from them, is how one improves. Criticism provides vital feedback, points to consider. You don’t have to take it personally, but do take it to heart so you can learn from it.

One of the differences, I think, that sets the triple-A studios apart is their ability to self criticise, and their refusal to accept anything less than the best for their games. To survive working in a top studio, to be able to make the best of the best games, means having the strength to accept criticism as an important part of a positive feedback loop. Get rid of the crap, and what you’ve got left at the end, if there is anything left, must be good.

But if you can’t handle being told when your shit stinks, you are in the wrong industry. Round these parts you could be pooping rainbows and someone is going to tell you that your rainbow excretions are the worst they have ever seen.

So suck it up. Take it in. But don’t let it get you down. Criticism, like water off a ducks back. But you know what? Ducks also swim on water. Without it, they’d just be waddling around. The ducks need the water, and we need to accept that criticism is an important part of the process.

Now, before I sign off there is one last thing I want to talk about: Metacritic.

Last week metacritic showcased a new feature, they added a developer’s career score. Then retracted it the next day^[1]. I was not, and am still not, so sure how to feel about that. Many developers will work on games that review badly at some point in their career, and often a game will review badly for reasons completely out of the individual’s control. Given the weight that the industry already places on metacritic scores, is it reasonable to meta-score individual developers? Probably not. After all, the industry is already struggling with the growing disconnect between what the core gaming reviewers want from a game, and what the average (read: casual/social) gamer wants. Adding a further incentive to pander the critics, or encouraging people to “Alan Smithee” their involvement with a project they’re not sure will review well, is probably not a healthy move for the future of games.

If this subject interests you, you may also enjoy reading Nicholas Lovell’s “What Jacqueline Howetts Professional Self-Immolation Can Teach Us All“.

Well, it probably is, and you probably could. But what does “senior” mean in a job title these days? What do employers mean, what do they expect, when they say that they’re hiring a senior developer?

This post was inspired by a question posted to a mailing list, that asked: “Senior X Developer, what does Company Y expect?”. Also, two of the guys I work with recently got the “senior” promotion. Congratulations to both of them. But anyway, I’d like to try to answer that question for X = Game. I’ll keep the general stuff at the start, so designers and artists can stop reading when I get to the programming specific bits.

Please keep in mind, what is written here is my personal opinion only. I have been on both sides of the interview process, at a few different organisations, but my observations here do not represent the official policies of any of my employers, past or present.

Now, any company worth it’s salt will be looking for the magic combination, so eloquently summarised by Spolsky in his Guerrilla Guide to Interviewing: Smart and Get’s Things Done. That, as they say, is the magic ingredient, but it’s not what makes someone senior. It may be what keeps them in a job long enough to become senior, but it’s not a defining characteristic of seniority.

In the games industry, at least, what is expected of a senior developer is experience. The real question of course, is how experienced. According to the 2010 Tiga Industry Salary Survey, the average length of service for a game developer is between 7 and 10 years. While it’s not impossible to gain the title of senior after as little as 2 years, and one completed project, 4 or more years industry experience, with 2 or more shipped games is more in line what what seems to be expected.

Beyond the requirement of time done, the question then becomes, what will you have been expected to learn in that time, because the value of time accumulated doing the job comes from the lessons learnt, and experience gained.

A senior developer needs to talk-the-talk and walk-the-walk. This means, beyond doing their job, completing the tasks assigned to them by their lead, a senior is usually expected to have a basic understanding of how developer/publisher relationships work, and how the the roles of art, design, programming, & management typically work together to produce a finished game.

A senior developer should have had some experience of a full development cycle, from pre-production, through production, alpha, beta, and delivering a final finished master for publication.

A senior developer (of any discipline) will be expected to have a grasp of what it means to work with a version control system, usually one of P4, Subversion, or AlienBrain.

A senior hire may also be expected to have had some experience with planning & scheduling (at the least, task estimation), producing documentation, and mentoring or lead work.

For programmers (designers and artists can skip to the end now, if they like), seniority tends to imply a greater breadth of experience (unless the title gives specific sub domain, like Senior Physics Programmer, or Senior AI Programmer – that usually just means you can get that specific job done without any hand holding from your lead). A senior programmer will usually be expected to have worked on more than one platform (hardware or middleware), or have shipped games in more than one genre. More than one programming language may also be expected (think, C++ plus one of C#, Java[script], Lua, Python, Action Script, Objective C, a HLSL or other GPU/SPU programming).

So, if you can comfortably tick off most of those expectations, then you are probably ready. It’s time to do it. Go and tell your boss it’s time for that Senior title, or apply for that new job. You deserve it. Do it now. Do it while it still means something.

Photograph (c) Étienne Ljóni Poisson

Expect The Unexpected

Before you start, you need to think carefully about what you are trying to do. What problem is it you need to solve? Is it just one problem? Or is it several different problems? How are they different? How are they similar? I have found that there are five broad types of errors that programming teams should be aware off so that they can be actively detected and handled.

Programming Mistakes

Nobody is perfect. Everybody makes mistakes. Even programmers¹. We make assumptions about how things work, and write code based on those assumptions. When the assumptions are wrong, the game doesn’t work as intended. In order to ship a game that doesn’t crash, we must find the mistakes, and fix them. The team also needs a regularly updated version of the work in progress so that the content team can work. Ideally all programming mistakes will be eliminated before a build is even distributed to the designers and artists. Let me be clear on one point: A “broken” build distributed to  the whole team can halt productive work in its tracks and cost your employer tens of, if not hundreds of, thousands of pounds². When an error is detected that will crash or otherwise break the game, a programmer will need to investigate, debug and fix the problem.

Game Data Problems

For the game to load and run, the data assets (art, levels, scripts, and so on) must be present and in the formats expected by the game. During development game data problems should be detected and reported clearly so that developers (artists, level designers, etc) can fix them independently. Having the build crash because an asset is broken most likely means that a content creator has to stop what they are doing and ask a programmer for help. The programmer may well have to get a copy of the artist’s local work, and debug it at their own machine. This is not a cost effective way to work. What’s worse though, is that an undetected error in an asset that has been checked into the build can be just as devastating to the teams productivity as broken code build. However, error messages of the kind we need during development are not appropriate for a shipping build³.

Save File and User Created Content Hack Protection

Any data loaded by the system should not be trusted, be it save game files or user created content. If you want to get certified for publication on any of the closed console platforms, or indeed, simply wish to prevent save-edit cheating and exploits in general, you need to create a game-save & load system that is robust against malicious modification. This must work and be present in a shipping build.

Bad Disk Errors

File IO errors must not be ignored, and indeed, must be handled in very specific ways on the console platforms for a game to be certified for publication. In addition to checking success and failure, all data loaded should be validated to ensure it is as expected and will not crash the engine. As with save hack protection, loading IO handling must be present in your shipping build.

Lost Network Connections

Network IO errors must be handled gracefully, and as with save game data, it cannot be assumed that network packets are arriving as they were sent, so must be validated and checked to prevent cheating. This is especially important for multiplayer PC games operating over the internet.

The Tools Of The Trade

Please note, I will not be discussing development processes that are designed to improve quality and reduce mistakes. While techniques such as Code Review, Test Driven Development, and/or Pair Programming can help with the early detection and prevention of programming mistakes, and in improving code quality, they will not be covered here.

Assertions

Assertions both document and check at runtime the assumptions made by a programmer.  They are placed in the code at the point where, should those assumptions be incorrect, the program will fail. As such, they should be simple, fast and free from side effects. Usually they check function pre & post conditions, and class invariants. It is usual for game engines to define their own assertion macro(s). The canonical assertion macro accepts one argument, and checks if it is true, then calls an error handler if not. Mature engines can end up with some remarkably complex assertion failure behaviour. They can stack trace, screen dump, attach to a bug database over the network or simply stop in the debugger. The assertion error handling behavior can change with the requirements of the team, and can have (or not have) whatever you think is useful⁴. The macro usually looks something like this:

#ifdef MY_ASSERT_ENABLED
#define MY_ASSERT( c ) \
{if (!(c)){MyAssertHandler(#c,__FILE__,__LINE__);}}
#else
#define MY_ASSERT( c ) ;
#endif

Note that assertions are usually turned off in shipping builds, and may also be turned off in builds intended for artist & level designers use. As such, assertion statements should never include code that actually does something.

Unit Tests

Unit tests, like assertions, check that the code does what the programmer expects it to do. Also like assertions, unit tests function as a form of documentation, showing by example the correct use of interfaces and data structures. Unlike assertions, however, unit tests are not part of the running game executable, and are instead compiled into a “test harness” which is then executed as part of the build process. Unit tests provide a certain amount of guaranteed code coverage, and reduce the risk involved in modifying the code.

This decoupling of test and execute is elegant, and can ensure that corner cases that may be not be checked by assertions during a normal run, are definitely checked every time the game is built.

On the other hand, unit tests do not catch incorrect use of an API, or anything that the test authors didn’t think of that might go wrong at runtime.

The difference between unit tests and assertions can be summed up by saying: Unit Tests test that your code does what you expect it to do when it is used as you designed it to be used. Whereas Assertions check that your code is being used as you designed it to be used and that the rest of the application is doing the things you expected it to do.

Data Validation and Reporting

Data validation and error reporting is simply a matter of programming. That said, having a clear architecture decision made about where in the code the validation checks should appear, and how they should be reported is vital to avoiding a proliferation of redundant checking in code. You should check the data that comes into the engine, and you should have a strategy in place for reacting to the unexpected in an appropriate manner.

Defensive Programming

Defensive programming is the practice of assuming that the things that can (and can’t) go wrong will go wrong, and attempting to carry on regardless. At the core of the defensive programming philosophy is the idea that it is reasonable to sacrifice performance for the sake of stability. This is an anathema to many games programmers, and I would strongly advise against belt-and-braces code that combines assertions with defensive programming, often seen like so:

assert( pObj );
if ( pObj ) {
// ...
}

In this case the assertion is redundant, since the null pointer case is in fact handled, so including an assertion to check for it should not be necessary. Assertions and Defensive Programming do not go hand in hand. The more you have of one, the less you should see of the other.

That said, coding defensively should not be rejected out of hand, especially where the productivity of the content creators may be negatively affected. Implementing coherent strategy for defensive handling data errors that allows the game to continue after a data error has been reported is usually time well spent.

Advice

So, having outlined the different problems, and the usual tools, what now? If you’ve got this far without falling asleep you are probably expecting some advice from me. So here are my top 10 tips for dealing with errors:

1. Think about the problems you are trying to solve for your project, and the consequences of each solution on your team and your customers. This point overrides anything else I might suggest.
2. Accept that there are several types of error that you are interested in detecting and reporting. Treat each one separately. Do not try to create a one-size-fits-all solution.
3. Use Assertions and Unit Tests to detect programming errors. A bug caught by an assertion or a failed unit test is a bug you’ve caught early, and a core dump you are not downloading from your QA outsource provider at 1am.
4. Use assertions to check that your code is being used as you designed it to be used and that the rest of the application is doing the things you expect it to be doing.
5. Use Unit Tests to test that your code does what you want it to do and to provide examples of how the interfaces you have written are intended to be used.
6. Create separate build configurations for each of: Debugging, Content Checking and Shipping. Programmers should use a debugging build, content creators should use a content checking build and the shipping build is what you intend on giving to your customer(s).
7. Create a mechanism for reporting problems with textures, meshes and other loaded data, that can be understood by an artist, and make sure that it is used consistently. Do not use assertions to check loaded content and do not provide data-loading APIs that allows errors to be ignored.
8. Try to avoid content errors from being fatal. If a texture or mesh is in some way un-loadable, have the loading code return a default object after reporting the error.
9. Content Checking, Bad Disk and Save Hack detection may be very similar in the things they check, but how the build responds to problems once detected should differ greatly, depending on the context. Where the content error messages are not appropriate for a shipping build, the content checking error handler in a shipping build can route to the bad-disk handler for an extra layer of game hack protection.
10. Trust no data that can be tampered with, use encryption and/or checksums to prevent and detect tampering (the platform SDK likely provides functions specifically for this), but even after validating checksums, sanity check everything and have the code handle gracefully (ignore or report) anything that could break your game.

Related Articles On #AltDevBlogADay

If this topic interests you, you may also enjoy:

• Do or do not by Paul Evans.
• Elegance in Failure: How and when to crash horribly by Ben Carter.
• Debugging Tools by Justin Liew.
• The Pain Of Debuggery by Tony Albrecht.
• Tips and Tricks for Debugging Optimized Code by Max Burke.
• Why Names Matter by Ben Carter.

With Thanks

To Bjoern Knafla, Richard Fine and Andrea Frogley for proof reading and wonderful editorial feedback.

Fin

If you have any questions about this article, please feel free to post them in the comments section below, or ask me on twitter @codemonkey_uk.

Footnotes
1 But don’t go telling any artists I said that.
2 Or dollars. Or euros. Or Yen. Or whatever. Lots of money.
3 The exception to this being PC games that actively support a modding community.
4 Except a “Skip” or “Ignore” button – if you add that, then you’re not thinking. An assertion states what the programmer knows to be true. If an assertion fails then all bets are off and the toys should leave the pram.