Background

Let me start with some info on Pop Corny. Pop Corny was designed to be a simple and fun casual game, covering the need for short game session experiences that mobile phone games require.

I self funded, developed, published and initially launched the game as a paid application at the $0.99 tier. Later on, it switched to free and monetizing with IAPs, but bare with me for the details of this decision.

Below are some more technical details of the game:

• Development time: 8 months
• Number of developers: 1
• Outside contractors for sound and most graphics
• Build using a fork of Sylphis3D Game Engine
• Support of all iPhone, iPod touch and iPad devices
• Source code (gamecode + game engine code excluding third party libraries):
  
-----------------------------------------
Language files blank comment code
-----------------------------------------
Lua 52 2631 1026 10415
C/C++ Header 77 1209 1111 3718
C++ 32 731 882 2552
XML 1 1 0 788
Python 9 170 136 630
Bourne Shell 13 70 167 284
Objective C 2 14 14 25
-----------------------------------------
SUM: 186 4826 3336 18412
-----------------------------------------

• Game code accounted for 5983 lines of the above LUA code
• The total source dataset reached up to 0.4 GB
• Resulting “compiled” dataset at 10MB
• Initial price $0.99 (50 days) then dropped to free
• Total sales in two months: ~150000
• IAP conversion rate: ~2.0%
• Ratings: 4.5/5 (from 600 ratings, 70% of which are 5 star)

Promotion

As I said the game was funded and published without any help from the outside. This easily translates to: no money for marketing. Therefore I had to find ways to make the best I could on the cheap. One of the major aids on that seemed to be the famous launch day.

Everyone developing on the AppStore will tell you that the launch day is one of the most important events in effectively publishing your game. You must do the best in your powers not to fail that day. And this is true. I had published 3 more applications on the AppStore prior to this game and the launch day will give you eye balls costing thousands of dollars for free. This is through the AppStore’s new releases list.

My approach was to combine that visibility with all the “social” publicity I could harvest, doing my best to leap as high as I could on the charts. Then possibly some game journalists would notice and write about it, thus keeping the fire burning. To do that I created a trailer video for the game and one week earlier I created an event on Facebook for the “unveiling of the video trailer”. I invited everyone I could and one week later (on January 31th) I presented the video. That day I also posted the video on every related forum I could. This caused some fuzz but mostly in Greece where most of my facebook friends live in.

The release day was set on February 3rd. I already had approval from Apple and I was already sending out promo codes to game journalists before that. For this I was carefully compiling a list of game review websites and journalists from early on the development cycle. I had these ordered by alexa ranking by review scores and game types. When I got the game approved by Apple, I send out all 40 promo codes I had available, and contacted 100 more without a promo code.

Early on the development cycle I took another big decision based on marketing. That was to support OpenFeint in addition to GameCenter. OpenFeint has its own top games, featured games and new releases, which can give some extra exposure to your new and unknown game. I decided that I needed that extra eyeballs, so I supported both social gaming networks.

In short what I did prior to the release day were these:

• Created a “coming soon” video trailer
• Created Facebook page and Twitter page for the game
• Organized a Facebook event for the unveiling of the game video
• Send out all 40 promo codes to game journalists (with all media packs, bells and whistles)
• Wrote posts in “new releases” parts of forums
• Prepared a cross promotion among my other apps
• Supported OpenFeint

Then all I had to do was wait for the launch day and the sweet success… or not…

Apple spilling the milk

With everything set and the game automatically scheduled to launch at the world’s AppStores on February 3rd, an unexpected nightmare started for me on February 2nd (since we live in a globe and in New Zealand it was already “tomorrow”). Of course I was viewing the New Zealand’s AppStore waiting for Pop Corny to appear, and it did. Oh! The Joy! After all these months of hard work it was there… taking its chance! Couldn’t be happier and more stressed at the same time… however the moments of joy ended shortly after I looked closer. Fear filled my heart when I noticed next to my game’s icon a release day of “Jan 31th”. I rushed to the new releases list like a maniac looking for Pop Corny which was nowhere to be found. What happened? Had the app gone live 3 days ago and I didn’t notice? Hardly… the app was not live on any other AppStore. I manually checked every other AppStore… not released…

AppStore entry showing
the wrong release day

I was optimistic, these distributed systems are slow to update and initially things like that might happen. However, Feb 3rd started to dawn in more countries and the same story was repeating. Pop Corny was appearing as a 3 days old app. 3 days when you launch on the AppStore is an eternity. I was out of the front page, before I even got the time to get on it. I tried to contact Apple through the support system but that is easier said than done.

It was then I realised how small and expendable I was. There is absolutely no way to solve a situation like this with a system that works by choosing among 15 nested problem categories before you finally get a text box to write down your problem. No way to resolve it with an one day round trip for getting a reply that was canned and had nothing to do with what I described. No way when you can’t -at least- follow up on the answer you get. I tried again and again for the next days by submitting from the start my problem rephrased in the best way I would… fail.

It was almost 7pm US time when auto-magicaly the release day of Pop Corny rolled to the correct Feb 3rd on all AppStores. However it was too late. Some AppStores were about to go on Feb 5th by then. The icing on the cake however was the reply to my last plea at Apple that told me that they don’t see any problem with the date as it is Feb 3rd. The irony!

The problem also managed to mess up OpenFeint’s releases system. OpenFeint, once you have the app approved, scans the AppStore to see when your game releases. I guess that was the reason Pop Corny didn’t got on OpenFeint’s lists. One week later I contacted them about the problem and they said that somehow the system skipped my app, and that they had to add it manually. They couldn’t find an explanation, but I could…

This way I had zero exposure from the AppStore new releases, resulting in next to zero sales. It was a disaster. Apple should really make up to me for this someday by featuring it for a day or two… I can wait.

Getting a grip and keep walking

The above disaster affected all the AppStores in the world. It resulted in Pop Corny selling 10 units on release day in the US. I know crap-ware that were released the same day and sold more (based on their top paid ranking). It was so, because no one knew of my new released game.

In Greece, however, it was another story. Here is most of my “social network”. People knew about Pop Corny and they were expecting it. Several major local iPhone blogs wrote about it and the game took off. In less than 24 hours Pop Corny was number one top paid application. That was among all applications, not just games. 5 stars reviews started coming and it was heaven. A few days at the top and it was also on the local TV media.

This contrast between Greece and all other countries was to me a great illustration of how publicity is king. You must bring your game in front of the players face. And that is hard. It is the hardest part for a small developer. No one will look at you when you are small. Getting big seems like a self fulfilling prophecy.

From the 150 emails I send to reviewers I only got replies from 10 people. 8 of them kindly suggested I pay some fee and the rest two just told me that they were too busy to bother. By monitoring the downloads on the media pack I could see that just two of them bothered to download it. It is clear that something is wrong here. It seems that the only way to get covered these days is through dollars.

Being the number one top paid application in Greece translates to 200 sales a day (yes it is a small store). This is 100 euros a day after Apple’s cut. Does not sound that bad, but given the universal decay law of the AppStore (I will talk about it next), it was not enough. Meanwhile on all other AppStores the app was doing poorly.

I knew that it was out of my reach to reverse this situation, so I decided to step back, stop desperately trying to push it, and focus on fixing some bugs, implement some new features until the day the game would go free. Going free was in my mind since the development days. The game was actually designed to work as a free game from development day one but the final decision to go paid was taken a few days before the release.

I saw this as a second chance. My means of getting more eyeballs on the game. But I knew that going free is not easy. It is a whole different arena. Going free will give you a big sales boost mainly because of applications like AppShoper. These kind of apps scan the AppStore every day and report to its users which apps has gone free, resulting in many sales. However many is not enough when competing with free applications. You need LOTS more.

There are services out there like OpenFeint’s “Free Game Of The Day” that provide you with the extra visibility you need for the day you go free. They feature one game every day, and promote it through their website and their iPhone app. I contacted them, they liked the game and I took the first slot that was available, that happened to be Thursday, Match 22nd (yeap the day Angry Birds Space came out… no fear!). They are very well organized and the process of submitting artwork was very streamlined and efficient. I submitted the appropriate banners and texts and there was nothing more left to do but wait for the day.

The day came and Pop Corny became free. The effect of this promotion was overwhelming. Pop Corny started to climb the charts all over the world at incredible speeds. It was not much later that it was ranking in the top 10 of more than 20 AppStores, and being number one of its category in 10 AppStores.

At startup the game checks for a .plist file on my server that contains possible news to present to the user. This allowed me to measure the rate of downloads in realtime. It reached a whopping 48 downloads/min. A few hours in the promotion 148apps.com featured it and even more downloads happened. This continued for the whole day (which is actually 48 hours to cover all the globe) and I had the opportunity to correlate the spikes I saw on the news file consumption with mentions on twitter, and posts on blogs, etc. A very educative experience.

With the game on so many charts it would be stupid to just turn to paid after the promotion day, switch charts, and lose the exposure. Also the revenue from the free version was already a multiple of the revenue with the paid. So I took the decision to keep it free. I believe I did the right thing. The game did 70000 downloads in the weekend following the promotion day, received very good reviews from the users and the revenue from the IAP was climbing. I was totally satisfied.

The universal decay law of the AppStore

One of the biggest myths busted by my experience with launching Pop Corny was that visibility on the AppStore is king. It seems like everyone believes that being on the top 10 is the way to sell. That this will translate to visibility and sales in a feedback process that will sustain enough to make a profit. You must do all you can to enter the charts as a way to make your game sell more. For me now, this is the biggest misconception about the AppStore.
With the huge amount of applications in the AppStore and the new ones that enter everyday, the charts only act as a capacitor. They smooth out spikes. Most sales today start outside the AppStore and not inside. The people that will buy your application because they saw it on the charts are not enough to sustain it on its position, let alone to lift it. Therefore we witness “the universal decay law of the AppStore” as I call it. Any application that is not receiving some external to the AppStore’s charts force will follow the graph below:

Let that be paid or free. The only thing that changes is the absolute amount of downloads, which is orders of magnitude larger when free. How fast is the decay depends on the competition, how much you icon and descriptions attracts buyers and what rating you have.

Bottom line is that the charts cannot sustain you these days. It probably did some years ago, but not anymore. You should never think of the charts as your means to sales. Use it just as an indicator of how well you are doing. The game is played out of this field. Get reviews, tweets, mentions, and of course get featured by Apple.

Conclusion

The AppStore is not an easy place to publish in 2012. I was not expecting it to be easy, since Pop Corny was not my first application published there. However I would really love not to have been squashed by that bug with the release day. At least it would make me feel much better if I knew I could efficiently communicate the problem to Apple.

In the end however I think that Pop Corny did very well and will do better with time. I learned a lot through the process, it has almost covered its development costs and also gave me the joy of having hundred of thousands of people playing my game. With a zero marketing budget and being my first game, it certainly exceeded my expectations. It proved that you can still have a chance on the AppStore without some big dollar marketing campaign. However it is getting really hard to do so.

One of the hats I had to wear for the development of “Pop Corny” was that of the game designer. The closer I had ever come to game design before this, was playing games with a little more inquiring spirit than most players do. This can definitely be interpreted as a bad hat fit. It was clear that in order to have a successful game, I had to find clever ways to improve the fit. This of course could be done by adjusting the head (becoming a better game designer) or by adjusting the hat (adjusting the problem itself to something that I would handle). It was obvious that I had to do the first as much as possible, but without the later I was not going to go far.

Adjusting the head

So I went out and absorbed every single drop of knowledge I could on game design. Read blog articles, watched video presentations, hanged posters of Johnathan Blow over my desk, etc :) All these helped, but I knew that I had to bring the battle to my battlefield in order to win. This meant that I had to base my game on a simple (yet effective) gameplay idea, and build on that using analytic methodology where I can be good at. My guidelines were:

• single touch, simple and effective game mechanic that would make great gameplay for touch devices
• The player should be able to start a game whenever he had some time to “kill”, whether that is 1 minute or 10 minutes or half an hour, and enjoy the game equally.
• The gameplay should not require too much “brainpower” from the player and depend more on trained reflexes and getting used to, while giving you the joy of acquiring true skills.

The process of meeting the above requirements can’t be fully transcribed, as it was mostly intuition based. It was clear that the game had to be an endless runner, which basically means that the player can never finish the game, and getting better would mean better score and more play. This would allow for my “short fun” direction.

I struggled a lot more on what an effective game mechanic should be to produce a great gameplay for touch devices. The simplest of all mechanics on a touch screen is the “tap”. I wasn’t feeling right about it, as it is overused and you probably end up with an experience not so original. Then I looked at swipes. Swipes are simple, but at the same time elaborate enough to resemble real life “game” actions. For example a swipe can be a throwing action in real life. So I centered my thinking around throwing stuff with your fingers. I did some prototypes of throwing circles and hitting rectangles and it was fun. Through an experimentation process I ended up with the “sling” throwing mechanic. You basically swipe in the opposite direction of the intented. This was a revelation. The inverted swipe posed a slight challenge to the players brain, that as one started to get it, it gave a very cool feeling of accomplisment. I went out and gave it to friends so I could test reactions. As people where getting it you could see fingers pointing all over the screen… fights over the who plays next, etc… I was convinced.

Mr. Pop Corny is a popcorn loving monster

The idea of the pop corn, the pop corn loving monster, etc came from my sister during a brainstorming session. So we had a theme. Next we started thinking of ways to enrich the experience and keep the players’ interest for longer time. One of the elements that worked was the “box progression”. I got the idea originally from Tiny Wings, and the way you upgrade nests, but found out that it is a common thing in the iPhone games field. So the user has to complete certain objectives to upgrade his popcorn box and get bigger score multipliers. We also used this progression to teach the player how to get bigger scores, and survive longer. After that we added an extra task of collect drachma coins for the player (an inside joke for my country possibly defaulting to its national currency drachma). He could then use the drachma coins to upgrade bonuses and buy new pop corn boxes that gave extra powers. So the game became more complex and hard to balance, but this was mainly a problem I could cope with as an engineer. I was in my field.

Do it like an engineer

Questions like “how much should a bomb cost?” (a bomb is used once and pops all corn on the screen at once) are hard to answer without the right tools. What data is necessary to give a good answer? For the above question you need to know the answer to at least “How many coins does the player get in one game session?” and “How many bombs in a single session does not put the game out of balance?”. An experienced game designer might be able to answer the original question by instinct and get it right. However, I had to do what I can do good. Break down, measure, answer, build and repeat until the required feeling is achieved.

What I did was log all the key elements of game sessions by beta testers, and interpret them with the so handy SciPy and related NumPy modules for python. These modules allow you to easily read csv files, process the data with almost every statistical analysis method in the world, produce beautiful graphs, etc. For example lets look below at the number of sessions per coin collected range. These are the samples that were collected for the same player level in the game.

These follow the normal distribution around the value of 80. This means that most of the times a user will collect 80 coins per game. However there might be caveats if we are not careful enough. For example lets look at the distribution below:

If we try to fit a Gaussian function in the above data we will have huge errors in our estimations. Before we go and hit least squares and optimization algorithms, we would always try to think through of what is generating the data. Here the false assumption about our data comes from the fact that we consider them all to be coming from the same “generator”. That “generator“ is the player interacting with the game in order to collect as many coins as one can. However because the game is a beta, and users are beta-testing it, the generators are two. The first is the one I described before, but there is also the sessions that the user tries to test some other aspects of the game and does not collect coins! So what we really have here is two different data sets mixed to one. One from the players trying to collect coins that produces a distribution of values around 80 (the red graph) and one from the players that don’t collect because are testing something and produce a distribution of values around 0 (the green graph). Adding the red with the green gives the original observed blue. The graph bellow illustrates it:

So we need to fit the sum of two Gaussians in our data with unknowns the two mean values and the two variances. This way we get the correct value of 80 for the mean coins collected per game session, which would otherwise be detected as 60 or something. So if you want the player to be able to afford 3 bombs every 4 games you set the bomb to cost 100 coins and you are set.

Conclusion

This was my approach to designing my game, and it seems to have worked out quite well, with the game receiving great acceptance. The game got to #1 Top Paid Game in the Greek AppStore within hours of its release and #2 Top Paid across all app categories, all 5 star reviews. On most reviews the addictive gameplay was mentioned, which is so comforting when I know that was not my best field. It turns out you can replace part of intuition and experience with analytic tools.

With my game entering the final phase of development, I think its time for a pre-mortem. We all often get to read post-mortems but we seldom see the state of the developer when heading for the release. There is lot that can be discussed there, before the after release discussions kicks in to overshadow everything. So here I would like to share some random “wisdom” I gained walking this path.

First of all let me provide some background. I started going by coding the game engine in August 2010. This was not starting from scratch as I started by modifying my old game engine Sylphis3D (you can poke around the source here: http://devnet.sylphis3d.com). This was all done in my spare time, with the only goal to see if it would be possible to run in on iPhone. It took a little while to succeed, an endeavor that required lots of code chopping. You see Sylphis3D is a fairly heavy engine for iPhone with its stencil shadows, per-pixel-lighting, normal mapping, etc.

It was around January 2011 when the engine was pretty much ticking with a new low-end renderer, when I got the idea of creating a simple 2D game with it. The game project was started and canceled a few months later because of problems with the graphics artist’s “productivity”. It was clearly my mistake of overlooking several indications of problems coming along. Which is my first lesson learned thought out this project. Never overlook problems and expect them to solve themselves automatically. It is not going to happen. It took a lot of time but it was not all to waste, since the work on the game engine was there to stay. And this involved huge rewrites that would anyway take lots of time. The changes included the migration to a different scripting language, the integration of two 2D physics engines (ended up with Box2D), new renderer, and lots of other minor things. As you understand there is little left that still relates to the original Sylphis3D game engine I wrote years ago.

Later came the idea of a different game that started from a single gameplay element I thought of and that we later brainstormed along with my sister to a full themed game. This game was supposed to be done in 2-3 months, and here we are a few days in 2012 and the release will happen later this month… so classic. It turns out you can’t really estimate the human factor in months. Crippling bugs will happen, artists will not deliver on time, the (awesome) music composer and sound effects engineer might get caught in an album release, and so on. To all that add the fact that I always seem to forget the time cost of communication in all these.

Through all that we finally managed to be able to make a pre Christmas release, but at the last moment I pushed the big red cancel button. I thought that it might be better to release the game after Christmas despite the huge sales spike the AppStore sees. It occurred to me that with all the big names dropping prices, it would be pointless for a poor, self funded, indie game, to even consider a chance. I hope it was a wise decision. It certainly gives us more time to do tunings to the game progression with more beta testers logging many more hours of gameplay.

Below are the commandments to self for building an iPhone engine and game, feel free to embrace any of them:

• Go Lua. The iPhone can’t really handle a heavy scripting language like Python for example.
• Even with Lua you should:
  • use incremental garbage collection, or even better a near-realtime gc
  • disable the use of undeclared globals or suffer pain debugging typos
  • create an object graph walker that will walk all globals, locals, upvalues, parameters, in order to be able to understand how your game is using memory and find leaks
  • create a custom memory allocator that can also gather statistics of where your memory is allocated. Lua makes tones of small allocations and memory fragmentation will hurt you bad
  • create a remote debugger for script code
  • always precompile the lua scripts in the release build, his will pose far less stress to the memory allocator.
• Always have in-place a logging facility to be able to replay game sessions
• Don’t relay on the XCode build system to build/install/debug your game. Create scripts that deploy the game to the iPhone simulator (with rsync for example) and bring the turnaround time to 1-2 seconds when only script code is changed.
• Build a solid asset pipeline. It should be possible to build the whole assets directory from the raw source files with a single command.
• Use testflightapp.com. This will make the beta distribution as smooth as it gets, coupled with console and event logging send to you for every single session played.
• Create a logging facility for key gameplay events (for example how much time does it take the player to clear the first level?). This should log to a remote server so you can collect data from your beta testers, and statistically analyse it later to extract useful information.
• Always test the game at the lowest hardware you support.
• Always try to watch new players play your game as you must keep in touch with the newbie perspective of your game. If you implement achievements for example at the end of your gamedev cycle you might find out that you create them to be a bit hard to achieve for a new player.
• Premature optimization is the root of all evil, but you better assign a day a week or so for checking your “optimization corruption” and fixing it. Even if you don’t “fix” it, it will at least provide you with a more clear insight of how things are going which is a very good thing to have at the back of your head when coding.
• And last but not least, don’t go out making a game without having read all the #altdevblogaday posts!

I hope you will find the above canned experience useful for your development. I tried to provide them in almost single sentences for the purpose of forcing myself to extract the essence of my thoughts. I would really like to hear your single sentence kind of wisdom.

(edit: The game is now released and you can get it here)

The system I use for such cases is a full replay system. To be able to reproduce a bug, you must be able to replay the exact game. And by game I mean the whole session from execution start, all the way to the point of crash. To do so you must think of the game engine as a deterministic black box that has specific undeterministic inputs. If these underterministic inputs are logged, we can reproduce the exact execution any number of times we like.

The key here is to understand these inputs. The most obvious input is the user input, either that being the controller, or the mouse, or the keyboard. However these are not the only inputs that add randomness to the system. There are some more things that need to be consider as inputs in order to get to a full deterministic behaviour. If your game uses random numbers, the random number generator is an input. You must either log it, or use the same seed as you did in the crash. If your game queries the system time (of course it does!) then you have an other input! We are basically looking for ever point that the game engine reaches out of itself and get data, or it is fed data.

So the inputs are:

• User controller input
• Random number generator
• System time

You mush hook a logging mechanism to all these inputs and record every single event. When it is time to replay a sequence, you start the game with a log replay mechanism at these hooks, in a way that all calls are answered with the same values as in the original play. This way you will follow the exact execution path that was followed when recording, but this time its like you know the future, and you can stop at any time before the crash to investigate. All these of course hold true in a single threaded environment. You must not allow the undeterminism of the OS’s scheduler get in the way. Tests should be done in a single thread lock down mode. If the crash is not reproducible at all in single thread runs, it is probably a multi-threaded race problem in the first place, and all I wish you is god help you! ;)

One of the major conversion reasons to DOD -that is an almost “hyped” term these days- is that with DOD you get to focus on arranging the data in cache efficient ways. However we mostly focus on the data cache, and we tend to forget about the instruction cache.

The role of the instruction cache is to accelerate the fetching of instructions from the main memory. This is exactly like the way the data cache is used to accelerate the fetching of data. After all your code is data! The difference is that it is much harder to optimize for the instruction cache than it is to optimize for the data cache. This is logical if you consider that we can freely arrange data in any way we can device. We don’t have that freedom with code layout. This makes most of us turn away from the problem and ignore it, while focusing on the data cache and hope all will be fine. Others don’t bother at all or simply get as far as minimizing code size. However it turns out that doing cache misses for instruction fetching is as bad as regular cache misses.

Don’t get me wrong, minimizing code size helps a lot. And yes; inlining every function will eventually hurt you. So let’s suppose we have carefully minimized code size, what else is there that will affect the instruction cache performance? I always like to start with an example so lets take a look at the code below:

void updateEntities(entity_t *entities, unsigned int numOfEntities) {
   for(int i = 0 ; i < numOfEntities ; i++){
      entity_t *p = &entities[i];
      switch(p->type){ 
      case 0:
        updateEntityA(p);
        break;
      case 1:
        updateEntityB(p);
        break;
      }
   }
}

Consider now that we have two kinds of entities, each with its own update function. The above code runs over an array of entities and depending on the entity type it calls the appropriate update function. Assuming that the entities of the two types are randomly distributed in the array, we can easily see that the execution will alternate in the functions updateEntityA() and updateEntityB(). Now suppose that only one function can fit in the instruction cache at once. This would mean that for every entity we will be loading instructions from the main memory with 50% probability. If that sounds extreme, try thinking of it with ten different update functions alternating at random and running on a device like the iPhone. We can easily see that we can do better that that. What would happen if we grouped together entities of the same type? We could do that by having one array for every entity type, or by having a single array that we keep sorted on the entity type. This will change the code access patterns from:

updateEntityA(), updateEntityB(), updateEntityA(), updateEntityC(), updateEntityB()…. … ..

to:

updateEntityA(),….,updateEntityB(),….,updateEntityC(),….., … …

Now the CPU will be able to update all entities of type A with the code resident in the instruction cache, then all entities of type B with the code in the instruction cache, and so on. We effectively reduced the cache misses from the order of the number of entities, to the order of the number of entity types.

The point to be taken from the above is that there is performance to be gained by organizing data in a way that when processed the code will follow similar paths; whether that is by following the same branches consistently, or by dispatching the same virtual method of a parent class with various custom implementations as children. Critical for performance code paths should be tested and optimized for the instruction cache along with the optimization for the data cache.

Optimizing for the instruction cache can be a difficult balancing act, and with the fact that we don’t have the ability to exactly layout the code, we must try to understand what the compiler is doing under the hood and do lots of trials and benchmarks. Inspecting the machine code emitted by the compiler is also crucial. Unroll loops and inline functions with caution. I usually do it incrementally while monitoring the performance. However keeping the code small is not going to help you if you run all over your codebase doing “random” stuff. Keep it in order, and you shall receive!

• business plans
• marketing plans
• legal issues
• accounting
• management

Please note that I intentionally left out all the creative talents. I assume that these are there! Programming, art, music, game design, etc are assumed to be cultured by the passion over the years. But all the above are well known, and many megabytes of digital ink where wasted about it on articles all over the internet. There is something more there, though. Something more elusive. Something that most of us get wrong. Something that when doesn’t result in hurting the game, it at least hurts the developer. That elusive matter is time management.

It is true that when you mix passion with all the above it can be hard to keep schedules. Time becomes the ultimate enemy. The thing that gets between you and your goal. Somehow time is never enough. You find yourself struggling between coding, managing, promoting, building frameworks, engines, websites… and it is then that you forget the value of time off. Personally, I can’t remember the last time I “virtually” finished work on Friday afternoon and got back to it on Monday morning. And what does that give me? Two days per week. So a six month project can be done in five. If you stop and think about it, does it make sense? I don’t think it does, but the distance from theory to practice is small in theory and large in practice! It turns out that giving yourself time off is one of the hardest things to do, requiring lots of discipline. Realising that finishing a project one month earlier is not going to happen by working weekends, is harder that it looks. Realising that that is going to hurt the project and you is ever harder.

We have to understand that time off is part of the process. It is not a waste of time. It is a required ingredient in the recipe for success. Especially for us doing creative work. We can never be creative when we are consumed in a process. Creativity requires disengaging, stepping away and looking at things from completely wild directions. You can’t do that when you try to squeeze that code optimization at 3:00am in the morning because the next day you must manage the advertising documents. Sometimes you have to force yourself to stop working on the project and do something completely different. Do something completely different, let yourself escape. And “No”, going sailing while tapping the refresh on your twitter app does not count as escaping!

This post will go live on Sunday. I really hope it gets less that 10 views because on Sunday all of us have a life beyond gamedev. Even if that life is in football (which I hate!). I for sure will try to read only about Barcelona, Liverpool and other FCs this Sunday. Nothing about gamedev and preferably on a newspaper (in real paper) while lying on a sandy beach.

In one of my previous posts I talked about how you can make the Lua garbage collector (GC) more predictable in running time. This is a virtue that is highly valued in a GC used in games where you don’t have the luxury of going over you frame time. In that post I described a solution to the problem which works fine most of the time, leaving little space for garbage collection times that will hurt the framerate. However I ended that post with a promise to provide a better solution and in this post, I deliver.

The ideal situation would be to have the GC run for a specific amount of time. This way the game engine will be able to assign exact CPU time to the GC based on the situation. For example one strategy would be to give a constant amount of time to the GC per frame. Lets say 2ms every frame. Or it can be more clever and take into consideration other parameters, like the amount of time it took to do the actual frame. Is there enough time left for this frame? If there is, spend some for GC, if not, hold it for the next frame when things might not be too tight. Other parameters can be memory thresholds, memory warnings, etc.

All of the above depend on a GC that can be instructed to run for an exact amount of time. This kind of GC is what we call a realtime GC. And Lua does not have one. However it turns out that we can get very close to realtime with minor changes to the Lua GC.

The patch below modifies the behavior of the GC in the way we need it:

--- a/src/lgc.c
+++ b/src/lgc.c
@@ -609,15 +617,14 @@ static l_mem singlestep (lua_State *L) {
 
 void luaC_step (lua_State *L) {
   global_State *g = G(L);
-  l_mem lim = (GCSTEPSIZE/100) * g->gcstepmul;
-  if (lim == 0)
-    lim = (MAX_LUMEM-1)/2;  /* no limit */
   g->gcdept += g->totalbytes - g->GCthreshold;
+  double start = getTime();
+  double end = start + (double)g->gcstepmul / 1000.0;  
   do {
-    lim -= singlestep(L);
+    singlestep(L);
     if (g->gcstate == GCSpause)
       break;
-  } while (lim > 0);
+  } while (getTime() < end) {
     if (g->gcdept GCthreshold = g->totalbytes + GCSTEPSIZE;  /* - lim/g->gcstepmul;*/

The only missing part from the patch above is the getTime() that can be something like this:

double getTime() {
    struct timeval tp;
    gettimeofday(&tp, NULL);
    return (tp.tv_sec) + tp.tv_usec/1000000.0;
}

I guess however that everyone will want to use their own time function.

The patch modifies the code so that is stops based on a time limit and not based on a calculated target memory amount to be freed. The simplicity of the patch also comes from the fact that we “reuse” the STEPMUL parameter that is no longer used to carry the aggressiveness of the GC. We now use it to hold the exact duration we want the GC to run in milliseconds. So the usage will be this:

lua_gc(L, LUA_GCSETSTEPMUL, gcMilliSeconds);
lua_gc(L, LUA_GCSTEP, 0);

The above code will run the GC for gcMilliSeconds ms. This way you will never be out of your frame time budget, because the garbage collection took a little longer to execute. Problem solved!

Doing too many small allocations and releases on the heap can create memory fragmentation, along with all the evil that comes with this. The common approach is to create a specialized memory allocator that serves small and constant in size blocks of memory to the scripting language, taken from a bigger chunk of memory reserved from the system. This is a common in all “realtime” and intencive applications like games, and something I did many times to gain performance.

Can’t beat the standard malloc

What I discovered with my latest attempt was that it has gotten quite hard to beat the GNU implementation of malloc(). Something that used to be easy in the past when you focused on a specialized case (e.g. small blocks of memory). Not that you can’t do better if you try hard, but at this point the malloc() implementation is already super-fast for 99.9% of applications on the desktop. Rest asured that you will not be able to do much better. However that is not the case for embedded devices that don’t share the same virtual memory benefits as the desktop computers.

My hand tuned specialized memory allocator for small blocks of memory ( <= 256bytes ) was not able to be more that 1% faster than the native malloc() on the OS X 10.6. And its hard to fragment 8GB of RAM and see huge problems. However on the iPhone the same allocator was twice as fast as the native malloc() ! Since the target was from the begining the iPhone that seemed like big win! However when I set up a small benchmark in the scripting environment that did some allocations of game engine objects and released then again in various patterns, the results were disappointing. The gain from using my specialized (and twice as fast) allocator resulted in improvement of about 5% in execution speed in a memory intensive benchmark. And at some tests even slower! That was odd and most of all not good!

Why I was failing

After some inspections and tests that made the case of me doing something really stupid less probable, I narrowed down the cause.

In most cases of using a scripting language you have some classes defined in C++ that you instatiate in the scripting language. Take for example a 3D vector class “CVector3″ defined in C++. When you instatiate this in the script language you get two allocations. One in the scripting language that allocates the “proxy” object and one in the C++ environment. When giving a new allocator to the scripting language to do its allocations you only “optimize” the first allocation. The one in C++ still goes through the system default allocator.

And since you optimize half of the allocations you expect to have half the performance boost… well… wrong. It turns out that you can even be slower this way. The secret here is the CPU cache. By doing the above, you have two memory blocks that are usually accessed together, but are far apart in memory. This can really hurt performance badly on a device with slow memory and small cache, like the iPhone.

The solution

The solution was of course to use the same allocator on the C++ side by overriding the “new” operator of the class. This made the blocks of memory allocated on the script side to be close to the block allocated on the C++ side. This way access to the object only involves accessing one part of the memory and giving nice cache hits. Performance up by 30%, which was nice and expected.

One other interesting thing that I found from this is that, on the iPhone, if I just override the “new” operator of a class and make it allocate the memory with plain malloc() and don’t use my allocator at all, the system is again faster!

This is probably from the fact that “new” does not go through plain malloc() (didn’t bother to check) as the scripting language environment does. So the allocated blocks end up in different arenas at different parts of the memory, with the result of losing performance for the same reason as above!

So, keep your related allocations close together when crossing the language barrier!

Today the developer can choose from a large variety of scripting languages, or even go crazy and implement one on his own. It happens that the most favored language for game developers is Lua. Its easy to understand why Lua is the favorite but other options are used as well. For example Python and the lately upcoming force of  Javascript.

Here I would like to share some of my experience of moving a game engine from Python to Lua.

Transition

My first experience with embedding a scripting language was with Python. More specifically with Stackless Python. When I started implementing my game engine I chose Stackless Python mainly because it was my language of choice for all development when not doing it in C++, and due to the support for tasklets. However the last year I got into porting the engine into the iOS platform and actually “pulling” it down in graphical features in order for it to run efficiently on the iPhone. One of the major changes was the transition from Python to Lua.

Better Memory Management

Python is a very powerful language will many bells and whistles and has served me well on the “full” version of Sylphis3D. However on the mobile platform the language was making things hard on performance. One of the major problems I was facing was the memory management. For those that don’t know, Python uses reference counting for memory management, coupled with a garbage collector (GC). What reference counting means, is that Python keeps track of the number of references an object has, and when that goes down to zero the object is no longer needed. That works well until you have two objects referencing each other, or forming a cycle of references. The reference count will never go to zero for those objects even if the cycle is not connected to the main object reference graph. So this is work for the GC who finds these lost clusters of objects by waking all reachable objects.

The problem with games is that we have to be fast. Fast and predictable. If you want the game to run at 60 frames per second, it means that you must do all the work in under 16.7ms. Whenever you miss that deadline -even by little- you have to wait for the next screen refresh. The worst thing that can happen is to cross that time limit sporadically and in a random pattern. This will result in motion shuttering, and it is most likely to give the user a better experience by going down to 30 FPS and sticking to it. So how does Python’s memory management system fits into the above picture? – On mobile… not so well…

The reference counting is completely predictable and consistent. An object will be released exactly the moment it is no longer needed. We have predictable and immediate release of resources. This is ideal for game development as long as you don’t create cycles. You have to be careful not to create cycles and if you do, make sure you break them. This however requires the scripter (which can be a non programmer) to be aware of implementation details. The ugly truth is that reference cycles will happen, and when they do it’s hard to find them. You must have a garbage collector, and Python does. However Python’s GC has a mind of his own and of course does not know of our 16.7ms time limit.

Python allows you to tune the GC, you can turn it on and off, but you can’t make it keep the processing low. When it will decide to run you are most likely to skip frames. Very bad.

On a desktop however you can easily get away with the following things combined:

1. Reference counting
2. Relaxed memory amount constrains
3. Explicitly calling the GC at key moments

Actually this was what I was doing on the desktop version. It turns out that since most of the deallocation of memory will be handled by the reference counting system, and that system is not too constrained on memory for possible reference cycles to be a problem, we can get away with manually calling the GC at key moments (e.g. before level changing). You might get a bit heavier on memory usage, but this is small price to pay for having a predictable frame length.

But on the iPhone you don’t get relaxed memory amount constrains. Memory is tight. There is no page file. You either turn the GC on and get frame skips when it runs, or you run it at least when you get a memory warning. In practise this doesn’t work out well. Not as well as I want it to at least. Which lead us to the decision to try out Lua.

Lua needs no introduction to game developers as it is well known for being lightweight, easy to embed and… featuring an incremental GC! This was the major push for adopting Lua in the mobile version of the engine. Having an incremental GC basically means that you can spread a full garbage collection cycle over many frames. This way you can keep it under a specific time budget. All you have to do is disable the automatic GC like this:

1

lua_gc(L, LUA_GCCOLLECT, 0);

and then do a:

1

lua_gc(L, LUA_GCSTEP, 200);

at the end of every frame to do a single garbage collection step. Depending of how big the parameter you pass to LUA_GCSTEP you will need more or less “steps” to complete a full garbage collection cycle. The choice of the step parameter however is left for the village magician to determine! To quote the Lua documentation:

…larger values mean more steps in a non-specified way.

So you have to play around with the variable to find the sweet-spot that does not use up too much of your precious frame time, and also keeps the garbage generated from the frame to rational amounts. This can be hard, and at the end we get something close to what I had with Python and full GC on, as at some point even a single GC step will spike up to 7ms. And that is logical since the Lua GC has far more work to do that the Python’s one, as it is not backed by reference counting.

But the swich to Lua was not in vain since the problem can almost completely go away by dropping the GCSTEP parameter down to zero and calling the GC repeatedly until a specific time frame expires. Something that looks like this:

1
2
3
4

double endTime = getTime() + 0.001;
do {
  lua_gc(L, LUA_GCSTEP, 0);
} while( getTime() < endTime );

With the above code we almost get it to run as we want it. However notice that the above code does not turn the Lua GC into a realtime GC. The above will run the GC for a minimum of 1ms. Every frame the GC will takeup _at least_ 1ms, even when there is no need to. That doesn’t feel really good doing on a device running on batteries. Apart from that we have an implementation that will hardly skip a frame due to the GC. And that is a win over the Python version.

Closing words

In general the overall transition can be called a success and it was worth the effort. I now have an engine that has much smaller footprint in memory size, more consistent behaviour and I got to keep all the good things. Here someone can raise several arguments like the loss of bitwise operations, native integer number support, even the count from one in arrays. But in the end as it turns out, it is not that much of a problem after all. The true power of Lua lies in its flexibility. The language is so open-ended that allows you to easily mimic any other language contruct (I still do miss the powerful concept of channels in Python though).

Also the simplicity of the Lua implementation as an interpreter can allow for easy understanding of the inner workings and makes it possible to even make changes right away. For example I was able to improve upon the garbage collection solution I laid out earlier so that I have an almost “realtime” GC, with steps constrained in duration, just by changing a few lines of code in the interpreter…

…I will save that for an other post though.