About a year ago, I wrote a blog post here about a weekly blog series on our company blog called “Game Face”. A blog post revisiting a blog post about blog posts? It’s blog posts all the way down. As a refresher, “Game Face” is “our weekly round-up of our internal and external open source work at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that.” How’d we do in our open source efforts over the last year? Read on…

THE BRASS TACKS

In 2011, we open sourced 22 projects. In 2012, we open sourced 33 projects. Obviously not all of these projects are going to “stick”, but one man’s trash is another man’s treasure right? The projects aren’t trash, BTW. A few projects came out of internal company-wide Hack-A-Thon sessions that we run every 2 months, but a fair number of them are taken from production applications. In looking at the 2012 project breakdown, by language, we have Ruby (22), Python (3), CoffeeScript (3), JavaScript (2), C++ (2), PHP (1). 7 of our engineers and 21 external people contributed to the various projects that we open sourced in 2012. Contributions range from clarifications in documentation to major new features that are developed, documented and tested.

It is very easy for us to track contributions to all of our open source projects. All of the projects are setup with a service hook to notify our company’s “Open Source” chat room when events happen like code is committed or a pull request is opened or an issue is filed. Every week I create a new “shell” for the “Game Face” blog post after the last one is published. I set it to be published automatically at 9 AM on the following Friday. As contributions come in to our open source projects, next week’s blog post is updated.

I’m not sure if we’ll open source more projects than we did in 2012. 33 is a fair number of projects to have open sourced. We obviously want quality over quantity. I’m always pushing to see if there’s anything we can open source. One of the ideas being tossed around as a way to expand our open source efforts would be to hold a weekly “Office Hours” session where a few engineers hop onto IRC or a Google “hangout” for an hour or so and field questions on our projects.

FIN

We are happy to contribute back in any way we can to open source software, whether it be our own projects or the projects we use on a daily basis. Again, if your company does anything with open source, I’d love to know about it!

It’s beneficial, sometimes, to buy in bulk. Or in this case, to rank in bulk. I’ll take a look at a recent improvement to our open source leaderboard library to show significant improvement when doing a bulk insert operation for ranking members in a leaderboard.

COSTCO AND YOU

Let’s face it, we’ve all been walking through Costco or Sam’s Club at one point and we’ve thought to ourselves, “I do need a palette of chocolate covered pretzels.” or “We could use a drum of grape jam at home.” Regardless of whether or not these are valid units of measurement, the time it takes to purchase an item in bulk is less than the time it takes to purchase the items individually when you know you’re going to need the bulk amount. So let’s explore this idea with leaderboards.

Let’s look at the performance of ranking 1 million members in a leaderboard.

1
2
3
4
5
6

insert_time = Benchmark.measure do
  1.upto(1000000) do |index|
    highscore_lb.rank_member("member_#{index}", index)
  end
end
=> 29.340000 15.050000 44.390000 ( 81.673507)

81 seconds isn’t bad, but can we do better? What if a catastrophic failure forces us to rebuild our leaderboards from scratch? Every second counts right? Let’s rank the same 1 million members in the leaderboard all at once.

1
2
3
4
5
6
7
8
9
10
11

member_data = []
 => []
1.upto(1000000) do |index|
  member_data << "member_#{index}"
  member_data << index
end
 => 1
insert_time = Benchmark.measure do
  highscore_lb.rank_members(member_data)
end
 =>  22.390000   6.380000  28.770000 ( 31.144027)

31 seconds! As it turns out, “buying in bulk” really paid off. It helps to understand your underlying data store, in this case Redis, to know when it can handle bulk operations to your advantage. If we were to compare the time it’d take to rank, say 10 million members in a leaderboard, we’d be looking at almost 14 minutes (ranking individually) vs. 5 minutes (ranking in bulk).

FIN

As shown in this post, bulk operations can significantly impact the performance of your systems if the underlying data store can optimize those bulk operations. So don’t worry about that desk of Cheez-Its you just purchased. You’re worth it!

I recently started a weekly blog series on our company blog called “Game Face”. It is “our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that.” This is important for our company and I think it should be for your company.

PUTTING ON THE GAME FACE

At the beginning of 2012, I did an assessment of all the projects we open sourced in 2011. Late much? 22 projects wasn’t bad and I want to beat that number in 2012, but it got me thinking that this kind of information is timely and needs more regular attention. So, I started kicking around ideas for a blog series name. After a day, I settled on “Game Face”. “Game” obviously referred to the work we do in developing gaming-related libraries and middleware. “Face” would refer to the code-wise public face of the company and the developer(s) working on the individual libraries.

It will help you become a better writer: If you’re a developer and you’re scared about writing a weekly blog series, you shouldn’t be. The focus of the blog is technical in nature, so you don’t need to spend a lot of time on exposition. If you’re good with your commit messages or keeping a CHANGELOG, the blog posts write themselves. For example, from an item in a recent blog post, “This releases addresses the first future idea from the README when the gem was released over a year ago to add a method allowing for bulk insert of data into a leaderboard.” Cut and paste for the most part my friends. The intrepid developer might even automate the creation of the initial blog post that can be wordsmithed by someone you deem more fluent in languages other than C++ :)

It will help you become a better developer: We get feedback on our code each day from our co-workers. By opening up your code to the world, you get peer feedback beyond your immediate peanut gallery. You also get comments, questions and sometimes code where developers are using your code in new and interesting ways that you hadn’t thought of yet. Case-and-point: I recently integrated a patch to our leaderboard library with all the code and tests to allow for leaderboards in “reverse” (lowest-to-highest) sorted order. I just hadn’t come across that use case in the games we’ve worked on where I needed a leaderboard appropriate for a racing game. But someone else did. And now our library is better off because of it. Beyond peer feedback, or any feedback, opening up your code and talking about it helps you to think about the ramifications of changes when there are developers other than you or your company using your code.

It will help you as a company: By highlighting your company’s open source work, it may motivate your developers to want to take a stab at open source in the first place or to clean up a library for external publishing. You might also attract developers who put a high value on knowing their contributions will be recognized, whether they are internal or external, and that their contributions may see the light of day outside of your company’s hallowed halls. The last “Game Face” blog post I wrote highlighted a new library one of our developers had released that allowed you to parse Beersmith2 (beer brewing software) files in Ruby. I imagine many video game companies rely on open source to some degree, and so by publicly promoting your contributions to open source, it can also help you in the eyes of the open source community.

It will help you have awkward conversations: I can’t tell you what to open source and what not to open source. You’ll have to talk as a team, as a company, and possibly with your lawyers to understand what you can and cannot open source. In 2011, there was only one instance where I went to our CEO to get a check on, “Can I open source this?” We walked around the block and our conversation went basically as follows:

Me: “I’d like to open source a leaderboard library. Given this is a core thing we do, is that OK?”

CEO: “So, someone else could do this before us using a similar method?”

Me: “Yes.”

CEO: “Ship it!”

FIN

Hopefully I’ve made some compelling arguments that your company should highlight your internal and external open source development. Our weekly blog post series highlighting our internal and external open source work comes out on Friday and now I usually get one or two messages throughout the week in our group chat room asking, “What’s going in to Game Face this week?” It feels pretty good when I can say back, “Your face.” If your company does anything with open source, I’d love to know about it!

I’m just going to cut to the chase here and present my “notes” when playing BioShock. You can find the original at its Flickr photo page.

I hate waiting for video games to load. Now I hate waiting for video games to load a little less. Why? I did some “science” to figure how just how long it takes for me to go from the point of clicking the button to select “Play” to where I’m actually playing the game. As it turns out, I just need to relax a little because I’m not actually spending a lot of time waiting for video games to load. I present the “Time To Play” numbers for 40 out of the 117 games I own on XBOX 360.

Time To Play (TTP)

The experimental “setup” was as follows:

1. Insert game disc into XBOX 360
2. Start the stopwatch on my phone when I click the “Play” square in the XBOX dashboard
3. Try to click through the loading screens and menus as quick as possible to get to a single-player (SP) or multi-player (MP) game
4. Stop the stopwatch on my phone at the point where the SP or MP game was loaded and I was playing

Notes:

• I do not have any games loaded onto my hard drive
• I was connected to XBOX Live the entire time
• If there was a title update for the game, I restarted from Step 1
• If I didn’t have a TTP number for SP or MP, it’s because I typically don’t play the game in that mode or that mode is not offered in the particular title
• TTP numbers are in minutes and seconds

These are the “Time To Play” numbers for 40 out of the 117 XBOX 360 games that I own.

If you would like to download the PDF of the spreadsheet, you can download it at XBOX 360 Time To Play.

Numb3rs

As it turns out, I’m only spending a little over a minute from the time I want to play a game to the time I am actually playing that game. Call of Duty: Modern Warfare 2 and Call of Duty: Modern Warfare 3 had the best TTP for both SP and MP games clocking in at just under a minute. I’m honestly not sure what was going on with Halo: Reach SP loading as it had the slowest TTP of 3:31.2.

I’m still going to be clicking through the menus to play games as quick as possible, save for that first time through where I don’t have a save game, but I’ll be doing so with a more tempered head.

The Real Travesty

Psychonauts was an “Unrecognized Disc”. I died a little inside. I think it might be due to the fact that Microsoft sent me a new XBOX 360 after my freezing issues with Halo: Reach and the latest drive doesn’t recognize the original XBOX discs.

Thanks for reading. You can find more hilarity over on my Twitter account, @CzarneckiD.

I recently got a new MacBook Pro and I wanted to do some benchmarking. Naturally I turned to the leaderboard library I maintain which already had a base set of benchmarks that I could compare the results to on my new computer. You may also recall I’ve written about leaderboards before here on #AltDevBlogADay. In this blog post I’ll discuss two “knobs” I tweaked to get more performance out of the leaderboard library, the underlying interpreter and an underlying client library. While there was definitely a speed improvement when switching the underlying interpreter to a more recent version, there was a more significant speed improvement when switching the underlying client library.

TWEAKING KNOBS

The benchmarks were run on a 2.5 GHz Intel Core i7 MacBook Pro with 8 GB of memory under OS X Lion 10.7.2 using Redis 2.4.5. They measured the total amount of time to rank 10 million people in a leaderboard using sequential scores, total amount of time to rank 10 million people in a leaderboard using random scores and the average time to retrieve an arbitrary page from the leaderboard. The basic benchmarking code can be found in the README on the leaderboard project page.

Switching Ruby Interpreters

Below are the benchmark results from switching between different Ruby interpreters, Ruby 1.8.7, Ruby 1.9.2 and Ruby 1.9.3. Values are in seconds.

Ruby 1.8.7

Time to rank 10 million people in a leaderboard (sequential scores): 794.157420158386

Time to rank 10 million people in a leaderboard (random scores): 849.301838159561

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.00165219999999999

Ruby 1.9.2

Time to rank 10 million people in a leaderboard (sequential scores): 656.387526

Time to rank 10 million people in a leaderboard (random scores): 748.955826

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.0011321999999999895

Ruby 1.9.3

Time to rank 10 million people in a leaderboard (sequential scores): 651.057383

Time to rank 10 million people in a leaderboard (random scores): 719.157958

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.001079199999999996

Summary

Ruby 1.8.7 to Ruby 1.9.3 (sequential scores): 18% improvement

Ruby 1.8.7 to Ruby 1.9.3 (random scores): 15% improvement

Ruby 1.8.7 to Ruby 1.9.3 (average page request): 34% improvement

Switching Redis Client Library

I decided not to benchmark Ruby 1.8.7 or Ruby 1.9.2 in tweaking the client driver “knob”.

Below are the benchmark results using Ruby 1.9.3 with the hiredis-rb client library. Values are in seconds.

Ruby 1.9.3 and hiredis

Time to rank 10 million people in a leaderboard (sequential scores): 472.544572

Time to rank 10 million people in a leaderboard (random scores): 549.911350

Average time to retrieve an arbitrary page from the leaderboard (50,000 requests): 0.0003803999999999928

Summary

Ruby 1.8.7 to Ruby 1.9.3/hiredis (sequential scores): 40% improvement

Ruby 1.8.7 to Ruby 1.9.3/hiredis (random scores): 35% improvement

Ruby 1.8.7 to Ruby 1.9.3/hiredis (average page request): 76% improvement

Ruby 1.9.3 to Ruby 1.9.3/hiredis (sequential scores): 27% improvement

Ruby 1.9.3 to Ruby 1.9.3/hiredis (random scores): 23% improvement

Ruby 1.9.3 to Ruby 1.9.3/hiredis (average page request): 64% improvement

WRAPUP

It seems every aspect of video game technology is tweaked to maximize performance, whether it be frames per second or requests per second. Fiddling with very basic “knobs” here, I was able to get decent improvements in writing to and reading from a leaderboard of non-trivial size. Further speed improvements could potentially be realized by using a socket-based connection versus a TCP-based connection as well as faster hardware optimized for a server environment versus an off-the-shelf laptop.

Thanks for indulging my leaderboard nerdery yet again. You can find more hilarity over on my Twitter account, @CzarneckiD.

In my last post about our internal studio Hack-A-Thon, I concluded with:

At our upcoming Hack-A-Thon, I’m going to be working on rewriting the internals of our open source leaderboard code. More specifically, I want to change the API to be more readable and self documenting as well as to take advantage of transactions to get consistent snapshots of leaderboard data. After that I’m going to re-run the performance metrics to see how transactions affects the leaderboard data retrieval. Next up will be updating the public documentation. Finally I’ll release a new library of the leaderboard code. And then I’m going to do all of that for the PHP, Java, and Scala ports. That’s the plan at least.

What was it they say about the best laid schemes of mice and men?

AWRY WE GO

I started working with the leaderboard code in Ruby at 4 PM on the day of the Hack-A-Thon. As stated above, I wanted to change the API to be more readable and self documenting as well as to take advantage of transactions to get consistent snapshots of leaderboard data. This was particularly important because leaderboards are constantly being updated and re-ranked in real-time and without transactions, the data returned to the client could have changed, resulting in stale/inaccurate data. After that I re-ran the performance metrics to see how transactions affected the leaderboard data retrieval. The good news is that transactions didn’t affect leaderboard data retrieval. At least in the planned usage scenario where you’d be pulling data from a leaderboard read slave. I do want to get concurrent read/write performance metrics as well. How about that for a follow-up blog post? I digress. Next up was updating the public documentation. Finally I released a new library with the updated leaderboard code. I actually waited a few weeks to do this, although there was no specific reason I didn’t cut a new library package at the end of the Hack-A-Thon. All of the initial leaderboard development work took me through 11 PM, with a couple of hours of non-working in-between to go to the gym, grab dinner at home, shower, and back to the studio. My trusty “companion cube” was with me as well. I had a game plan based on my revision history to the leaderboard code to tackle the rewrite in the next language, Scala.

The scala-leaderboard API, documentation and performance metric updates obviously went quicker. I finished those updates around 4:30 AM. 12 hours into the Hack-A-Thon, being up for nearly 24 hours at this point and I was still going strong. Let’s randomly pick another language, PHP.

That’s right folks, I decided to tackle the updates to the PHP leaderboard library. At 5 AM. We’re partying now, right? Speaking of dick killing parties (watch that video for one of the best ROFLMAO scenes from Bulletstorm). So yeah, the next time someone invites you to a PHP party at 5 AM, you know what kind of party it is. I digress. I basically got sucked into a vortex of hellfire and doom trying to get the PHP environment I somehow had set up on my home machine on my work machine. I worked at it for an hour or so. I greeted the sunrise. And then I took a 40 minute nap. Around 7:30 AM, people were starting to filter into the studio and a bunch of us decided to go for breakfast. I just needed a recharge. After breakfast I worked at the PHP environment for another couple of hours, flipped my desk over (not really), and rallied for the final push of leaderboard updates in the final language, Java.

I got through the API updates to the Java leaderboard library. I still have yet to finish off the documentation and performance metric updates.

FIN

All-in-all, the Hack-A-Thon was a success.

During the Hack-A-Thon, another colleague did his own port of the leaderboard library and so we have a good start on the erlang leaderboard library. Recently another colleague did a port of the leaderboard library to NodeJS and so we have the node leaderboard library.

In a week, we’re running an internal Hack-A-Thon at the studio. It’ll be the third one we’ve run. For 24 hours, everyone in the company participates and can work on whatever project they’d like to work on as long it’s somewhat (for a very relaxed definition of somewhat) related to the work that we do. It’s a great time.

HACKITY HACK HACK A ROO

Officially the Hack-A-Thon starts at 3 PM on a Thursday and finishes at 3 PM on Friday. Past Hack-A-Thon projects have included:

- Migrating the software on our continuous integration server

- iPhone application for interfacing to our platform services

- Unity video game where the studio was modeled and the game could be used as a testbed for sending game data to our platform services

- Video conferencing setup

- Internal project theme song, rap and artwork

- Open source testing framework

Before past Hack-A-Thons, I’ve gone home, hit the gym, grabbed dinner, showered and tried to nap. I’ve always been the first one to start before midnight and then it’s straight on ’till morning. Usually I can make it until around 7 AM when I have to take a 20 min nap, followed by breakfast, and then it’s back to coding until the wrap session.

Here’s some tips for making sure your Hack-A-Thon is successful:

1. Be inclusive – Everyone in the organization should participate. Encourage teams to be formed well in advance of the Hack-A-Thon. Let production work with development. Let development work with HR. However people want to form teams or work solo, let it happen.

2. Be open – No secret projects. Make sure everyone’s ideas are known ahead of the Hack-A-Thon. Even if someone wants to work on their project alone, it’s more exciting for everyone when they know someone might be rewriting a core piece of your game engine or simply getting video conferencing working flawlessly in your conference room. If you can involve the local development community, do that too. You never know the talent you might be able to attract by opening your doors for a day.

3. Be free – Be very lax in what people are allowed to work on. Projects should be somehow related to the work you’re doing, but they don’t need to be production-ready at the end of the Hack-A-Thon. Let people explore.

4. Be mindful – It’s the game industry. There are milestones. There are launches. There are conferences. Make sure you plan your Hack-A-Thon on a day that doesn’t preclude half your staff from participating because of a conflict.

5. Be focused – Shut down the tab to GMail or don’t open Outlook. Encourage teams to talk in person. Try and be as focused and productive as you can be in the allotted time. Unless absolutely necessary, e.g. our data center servers are actually on fire, focus on your Hack-A-Thon project and not your regular work.

6. Be scoped – Try and choose a Hack-A-Thon project you can actually scope to the allotted time. It’s more fun when you can demonstrate a near complete project at the end of your Hack-A-Thon.

7. Be closing – Provide a wrap-up session where people get 5 minutes or so at the end of the Hack-A-Thon to demo their project to everyone. And clap at the end of each demo. Even if people don’t complete 100% of what they wanted to get done, applaud the effort.

8. Be publicizing – Talk about the Hack-A-Thon on your company’s Twitter, Facebook or blog. Hint at some awesome upcoming projects that are spawned from the Hack-A-Thon or link to open source projects.

FIN

At our upcoming Hack-A-Thon, I’m going to be working on rewriting the internals of our open source leaderboard code. More specifically, I want to change the API to be more readable and self documenting as well as to take advantage of transactions to get consistent snapshots of leaderboard data. After that I’m going to re-run the performance metrics to see how transactions affects the leaderboard data retrieval. Next up will be updating the public documentation. Finally I’ll release a new library of the leaderboard code. And then I’m going to do all of that for the PHP, Java, and Scala ports. That’s the plan at least.

24 hours of coding.

\m/ \m/

You can find more hilarity over on my Twitter account,  @CzarneckiD.

I’ve also found that I prefer to ease into my day musically. I can’t just go for the gusto with Slayer – Raining Blood. It’s too much of a shock for my system. Most days I start off with one of the Rockabye Baby albums. In case you don’t know, they’re lullaby renditions of popular songs from artists ranging from AC/DC to The Cure to Tool. All of the albums are fantastic. I’ve found that it helps me to mentally prepare for the unexpected. For example, Tool – Sober, and Tool – Sober (snippet of the lullaby rendition). Expected in one case. Unexpected in the other case. Awesome sauce in both cases.

What makes you “tick” … musically?

You can find more hilarity over on my Twitter account,  @CzarneckiD.

In the last 2 weeks I’ve ported my leaderboard library, originally written in Ruby, to PHP and Java. It has been an extremely fulfilling endeavor for a number of reasons.

99 PROBLEMS

Let’s face it, we’ve all been on epic e-mail threads where someone poses a problem, e-mail armchair development ensues and people argue back and forth about what will and what not work, and at the end of the thread, the original author is left with more questions than answers, more what could be than what should be. That happened to me about two weeks ago. It all started with a Friday evening e-mail to engineering, “We have a vexing leaderboard issue I’m opening up for suggestions.” It was for a property we didn’t write, but one that was in dire need of help. The thread was 20 e-mails in before I responded at 9 AM the next day with (paraphrasing here): “I made this for you. Use it or don’t. At the very least, it can be tested right now as a potential solution.”

And so the php-leaderboard library was born. I had unleashed PHP unto this world. But, it was fully tested. And I also went through the steps to ensure that other developers could find it via the PearHub package system (php_leaderboard package).

If it had turned out, given the constraints, that php-leaderboard wasn’t going to be a viable solution, it really wasn’t going to phase me. I learned some PHP. I was like Jojo the idiot circus boy with a pretty new pet. Now the pet is my possible solution. “Oh, my pretty little pet, I love you. And then I stroke it, and I pet it, and I massage it. Hehe, I love it, I love my little naughty pet, you’re naughty! And then I take my naughty pet and I go …” Wait … what? The point is I found some inspiration to try something new and to present a reasonably complete solution to the problem at hand.

ONE MORE CUP OF COFFEE

You could say that after the PHP port, I developed shpillka in my ganektagazoink, but that’s besides the point. No, that is the point. I had the bug. The bug to learn new languages and re-learn old languages. So, before coming to Ruby, I developed in Java for nearly 15 years, with a 3 year stint of C# in there as well. And so this past weekend I ported the leaderboard library to Java and the java-leaderboard library was born. I also took the steps there to figure out how to properly release that as a library that other Java developers could integrate using Maven. Thankfully, after 3 or 4 days of what seems like more process than progress to me, I was able to get the java-leaderboard library out to the world. Search for it and you’ll find it.

MY ANACONDA DON’T WANT NONE UNLESS …

The PHP port was also inspiration for two colleagues to release the python-leaderboard package.

COMFORTABLY NUMB

It felt really good to work in a new language and to re-connect with an old language. And for whatever reason, I need that mental disruption to continue. And so it shall. My current in-progress ports of the leaderboard library are to NodeJS and Go. Porting the leaderboard library works for me because of a number of factors:

1. It is well-defined in terms of behavior/functionality.

2. It requires integration of a service and a service client library.

3. It requires some knowledge of control statements, types and higher-level data structures.

4. It has a well-defined set of tests.

I’ve had to become comfortable with being uncomfortable. New syntax, new packaging, new test framework, new release mechanism. Conformity be damned!

FIN

I would encourage anyone who feels they’re set in their ways or in need of a change to undertake a similar endeavor. I can imagine this applies to any number of disciplines in the video game industry. Find a well-defined, testable problem that you’ve solved again and again and solve it differently.

It’s that simple.

You can find more hilarity over on my Twitter account, @CzarneckiD.

This post is EXACTLY* what you think it’s about.

* NOT EXACTLY

Gotcha! But, you’re still reading, so what does that say about YOU? ;)

I write 2 types of posts here on AltDevBlogADay, one that is more developer-oriented and one that is not. Each type of post is gaming-related, however. This post falls into the latter type.

I couldn’t find the online article(s) from awhile back (maybe a couple of years or so?) where whatever site had famous or more well-known game developers post photos of where they game. If you know of the article(s) I’m talking about, please link me in the comments. One of our readers found it, Work and Play: A Peek Inside the Lives of Gaming’s Greatest.

In any event, I’m not famous, but I have a pretty baller setup as far as where I game and my gaming “infrastructure”. So, I’m going to show that to you now.

I do most of my gaming in the master bedroom. 42″ HDTV. PlayStation 3. Wii. PlayStation 2. Xbox 360. You’ll notice, or not, the wires run behind a very tidy wall conduit so that the wires aren’t exposed. The Xbox 360 power unit is actually zip-tied underneath the glass platform. As you can tell, I mostly play Xbox 360 games … mostly. I only play PlayStation 3 exclusives if I do buy and play PlayStation 3 games. The Wii never gets used except for when the nieces or nephews come over. The PlayStation 2 is still hooked up so that I can play Ico or Shadow of the Colossus.

The master bedroom remote is also pretty sweet. As you can see from the photo below, the Xbox 360 is hotkey’d in the remote menu. Click on it and it will switch the TV for you to Xbox 360 mode. Similar hotkeys exist for the other gaming systems.

The great room is where I can move a system to if I want a more immersive gaming experience. I own an extra set of video cables and power cords so that I can move a system down here without moving those items from the master bedroom. 45″ HDTV. Front speaker under-mounted on the TV with a custom bracket. Sub-woofer (hidden). Rear speakers (not pictured) mounted in the upper walls to the left and right of the oversized 2-person chair I was sitting in when taking the photo. This is where I played both Dead Space and Dead Space 2, only after dark, and only with the lights completely off. Frightened much? This is also where we had an epic Gears of War 2 Horde session with folks from work, when at 5 AM our CFO at the time said, “Guys, I need to go. My son is going to get up in an hour.”

The great room remote also has functions to switch the system into game mode.

Now that I’ve showed you mine, so to speak … it’s your turn ;)

FIN

Did I actually expose myself? No.

Did I actually expose where I game? Yes.

Leave a comment and link to a photo of where you game. If I get enough photos or interest, I’ll use my next non-developer AltDevBlogADay post to expose you, so to speak, on the Internet. Please keep it clean. This blog is rated E for Everyone!

You can find more hilarity over on my Twitter account, CzarneckiD.

You’ve got your continuous integration (CI) server setup so that whenever someone checks in an item (code, asset, configuration file, etc.) to your repository, a build process is started, tests are run, the build passes or fails, and then what? If your build passes or fails, who gets notified? Enter continuous notification.

CONTINUOUS NOTIFICATION

Continuous notification is not a new concept. It seems logical to me that you send out notifications on the status of the build in your continuous integration server. Here are ways you might want to go about notifying your team.

E-mail: This is the most passive form of notification and any continuous integration server worth a damn will support build notification via e-mail. You’ve probably already got a team e-mail list per project for the various projects being built by your continuous integration server. Make sure the team knows the build failed. You can optionally also send an e-mail to the individual who broke the build, but at that point it’s just insult to injury. Developers may not be checking e-mail on a regular basis, so consider adding additional continuous notifications.

Group chat: If your teams use a group chat like IRC, Campfire, HipChat, Yammer, or whatever, your continuous integration server might support notification for these services. This is a more direct form of continuous notification as opposed to e-mail. You will most likely interrupt developer conversations. Good. If you notify a separate build notification room, developers will never ever join that room. Ever. It also means there’s a formal record of the build status as these services typically support day-to-day transcripts.

Instant message: The build fails and the developer gets an instant message from the continuous integration server letting them know as such. This is one of the most direct forms of continuous notification short of the continuous integration server notifying Johnny 5 who subsequently comes over to your desk and says to you, “Los Locos kick your ass. Los Locos kick your face. Los Locos kick your balls INTO OUTER SPACE!”

FIN

Use your continuous integration server to notify your team on the status of your build(s). That is all.

Maybe this should have been the TL;DR?

You can find more hilarity over on my Twitter account, CzarneckiD.

Do all programmers have some kind of “tick” when they’re thinking at the keyboard? Mine is to tap the CAPS LOCK key on and off a few times. I use it as a way to still be connected in some way to the keyboard. I do it in writing blog posts as well. I’m doing it right now. But, you’d never know it.

What makes you “tick”?

You can find more hilarity over on my Twitter account, @CzarneckiD.

Around the studio, it’s a running joke that I’m the guy who beats video games. And it’s true. I love to play and beat video games. It might be a Sunday afternoon, a few days, or a month. Eventually I’ll start off in our AM standup with, “Oh, I just beat …” If there’s a game coming out, chances are I’ll be playing it. And by the time you’ve gotten around to playing it, I’ve beaten it. However in this post, I briefly analyze all of the video games where I have all of the achievements and try to assess why those games stood out as being “worthy” of that extra effort. These are a subset of the 100+ games that I own.

STICKS AND STONES MAY BREAK MY BONES, BUT …

I wouldn’t consider myself an “Achievement Hunter” or a “Cheevo Whore”, but those are two terms that may come to mind as you read this blog post. None of the games I played through to completion were games I played just to get the full gamerscore.

I take that back because first game I got all the achievements on was, “Avatar: The Last Airbender: The Burning Earth”. One might consider it a completionist’s “gateway drug”. Let’s face it, it takes less than 3 minutes to get all the achievements and 1000G. What do I have to say for myself for this transgression? I never inhaled.

This is the one Pac-Man game I’ve played from XBLA. Am I the only one who thought Ms. Pac-Man was better than the original Pac-Man in some ways? It had those intermission “cut scenes” that at least told a small story.

3 words. Doritos. Dinosaurs. Destruction. It’s a shame the game isn’t available from XBLA anymore.

I’m not surprised UNO translated this well into a video game. It’s such a fun casual game that you can play to wind down an evening.

One of my friends on XBOX Live turned me on to this game. I guess it was developed by 3 interns at Microsoft, but is a great example of a casual co-op game.

Another game sponsored by Doritos and another easy set of achievements. Almost too easy. It’s like Doritos wants to bump up your gamerscore each year. For free.

Zombies. Lots and lots of zombies. As far as a zombie game that you don’t necessarily have to think about, it’s fun. Although I think that the character resemblance to characters from Left 4 Dead isn’t any coincidence. And apparently zombies are attracted to teddy bears. Who knew?!

My only gripe with Stacking is that it wasn’t a full-fledged retail game. I loved and laughed every second I played this game. It was a treat trying to figure out what dolls you needed to discover, what combination of dolls you had to use to solve puzzles in different ways, and what “special things” you could do to other dolls. It is one of the best “short” games I’ve played in awhile. I really should pickup Costume Quest now.

I have a love/hate relationship with the Assassin’s Creed series. I love the story. I hate the game controls. For me, I think they really hit the sweet spot with that franchise with Assassin’s Creed: Brotherhood. I played the multiplayer in that game for about a week straight, 8+ hours a day, this past December. But, back to Assassin’s Creed 2. Most of the achievements were either story-related. So it was refreshing to try for these achievements that unlocked more pieces of the Assassin’s Creed story.

The Left 4 Dead series is the best video game featuring zombies I have played. Ever. Left 4 Dead was the first game I played on XBOX where I cared about the people I was playing with. It’s all about teamwork. I can remember the weekend where I finished the Blood Harvest and Death Toll campaigns on expert to finish the achievement where you survive all campaigns on expert. It was a rush. I still have one of the best legitimate times on the leaderboard for survival mode on the “The Church” map (one of the hardest survival maps IMO). I’d drop any other game to play Left 4 Dead if someone asked.

I actually left the studio early on the day Left 4 Dead 2 came out to play. I played it for nearly 18 hours straight and had 34 of the 50 achievements before I had to hop on a train for a company trip to New York City. It was an awesome trip. We had one meeting in the afternoon, but before that I organized a group of folks where we went to the top of the Empire State Building, the NY Public Library for a map exhibition, the Nintendo World Store, Katz’s Deli, and finally the Magnolia Bakery. All the time I thought, “I’ll be OK if the zombie apocalypse hits.”

I’ve probably played through the entire Borderlands campaign on Playthrough 1 and Playthrough 2 maybe 8 or 9 times? I’ve lost count all the times I’ve helped other folks out. All the DLCs, except for maybe Mad Moxxi’s Underdome Riot (which is just repetitive) are fantastic, especially The Secret Armory of General Knoxx. Ice cream soldiers anyone? Characters and character progression: awesome. Driving: awesome. Loot: awesome. Story and dialogue: awesome. Borderlands is simply fun.

Bulletstorm, for me, reinvigorated the FPS genre. It was so much fun going through the Echoes and re-playing parts of the campaigns to try and get the best score possible. All the ways in which I could use the environment and the unique skill shots make it a different game each time I play it. I played Gears of War 2 the other night on horde mode. It felt so repetitive and boring in comparison.

Dead Space may be the best survival horror game I’ve played. I only played Dead Space on my surround sound system in the dark. It freaked me the fuck out. Always. Once I knew Dead Space 2 was official, I felt I had to complete the game. #altmanbepraised

Dead Space 2 may have one of the hardest achievements in any game. “Hard to the Core”. You only get 3 saves and if you die, you go back to your last save. #protip: After the disc swap, if you die, the game will send you back to the beginning of Disc 2 unless you turn off your XBOX or end your game. I left my XBOX on for a day. Once I got to the final fight, I psyched myself up for a good 15 minutes. It was one of the best feelings beating this game. Looking forward to Dead Space 3 if that ever becomes a thing.

BioShock is the best video game I have ever played. Ever. My reasons there are deserving of a blog post in itself. Suffice it to say, I went a little cray cray towards the end. I present to you, BioShock Notes. I took meticulous (bordering on insanity) notes during BioShock. This is the final sheet and I got to the point where each save was described by a set of acronyms as to what transpired since the last save. Once BioShock 2 was officially announced, I was actually fearful of going back to finish off the last achievements for BioShock for fear of ruining my initial experience.

I relished at the thought of being able to go back to Rapture for BioShock 2. Although not as emotionally compelling as the original BioShock IMO, it was still a fantastic game. Minerva’s Den may be the geekiest DLC ever.

FIN

Video games are fun. I love to play and beat video games. You’ve gotten a glimpse into the video games I’ve wholly beaten. What video games have you beat? Why?

You can find more hilarity over on my Twitter account, @CzarneckiD.

I share my thoughts on data collection in video games. Examples of data collection might include performance data at the end of a Guitar Hero or Rock Band song or match data at the end of a Black Ops or Halo:Reach match or score and birds left at the end of an Angry Birds level. This is not an exhaustive list. YMMV.

YOU GOT A FAST CAR

But is it fast enough so we can fly away? There are times when your data collection is going to have to be as fast as possible. Launch day of your game and the week after launch, holidays like Thanksgiving and Christmas and Hanukkah, school breaks/vacations, evenings and weekends, etc. Make some worst-case scenario estimates on data traffic based on prior experience or Ouija board or however, but make sure that your data collection can scale at these peak times. If you need to spin up servers temporarily for these times, do it. This is also an opportunity to investigate micro frameworks that reduce application complexity and that may allow you to be as direct as possible in collecting game data into a database.

EGGS AND BASKETS

I haven’t looked myself, but I’m sure there’s some seedy, alternate meaning to the phrase, “Don’t put all your eggs in one basket” (Urban Dictionary: NOT SAFE). The same applies to data and databases. Don’t put all your data in one database. If you are going to be writing data from your game into a database, a good rule of thumb is a database per title+platform. If you use a single database for all your game data across platforms, when that one database “shits the bed” (Urban Dictionary: SAFE), and databases will eventually “shit the bed”, data collection will effectively be stopped for all titles+platforms. The single database also makes it a pain to do data archival when you want to move old entries into long-term storage. And a larger, single database means database backups are going to be slower.

U GOT THE LOOK(UP)

If you’re going to be looking up game data from your database, make sure you are using indexed fields. Any data retrieval resulting in a “full-table scan”, as the size of your data size grows, so does the time it takes for you to look up that data. This happened to me around Thanksgiving 2008 where collected game data, used for online tournaments, was taking longer than usual to post and affect the tournaments. This resulted in a complete rewrite of a data collector, that has since (knock on wood … hehehehe wood), performed flawlessly and that scales automatically during peak periods.

DON’T FEAR THE REAPER

Whether you’re working with a middleware provider to retrieve data or you’re doing it internally, presumably you’ll have some process to collect and do something with that data. I like to affectionately call these processes, “reapers”. Without knowing what data you have to work with, I can say this, make sure your reaper saves its state while running. Reapers, whether it be from solar flares or maxed out memory limits, will fail and they will need to be restarted. Without some bootstrap knowledge of where reaping left off, this may result in unnecessary processing of already processed data.

BATCH THAT THANG UP

Ensure that you use reasonable limits in the amount of game data that you pull at any one time from your database. Let’s say your reaper “shits the bed” on a Friday and you don’t notice until Monday and you need to restart the reaper. Furthermore, 1,000,000 (1 MILLION) new game data entries have been collected since your reaper stopped. Batching these unprocessed entries in groups of 1,000 (1 THOUSAND) will help to make sure you’re not trying to retrieve too much data in any one cycle.

USE A REASONABLE DATA FORMAT

It’s 2011. JSON is the shit and XML is dead, right? Whatever. Pick a reasonable, structured data format in which to package your game data.

XML:

<game>
<player>
<name>BronxZoosCobra</name>
<score>1000069</score>
</player>
</game>

JSON:

{
“game”:{
“player”:{
“name”:’BronxZoosCobra’,
“score”:1000069
}
}
}

SIZE MATTERS

I said it. You better make sure all your data is going to fit in your database fields. Don’t assume that an 8K database field in which to store match data, for example, is reasonable. Or maybe it is reasonable. But did you validate that a game data record with 16 players from a match with all game data statistics maxed out will fit into an 8K database field? If not, dummy up the data, try to stick that record in the database, and see if it fits. If not, increase the size of the database field. It’s that easy. No pills or creams required.

FIN

Data collection from video games requires some finesse to get right. Be quick to process, distribute your load, index your lookups, save state in processing, batch process, structure the data and size your storage appropriately.

Also, happy April Fool’s Day.

You can find more hilarity over on my Twitter account, CzarneckiD.

About once a week, I like to ask on the Twitter, “What are you playing tonight?” I’m generally interested to hear about the games that people are playing. I play video games … ok, a LOT of video games. I figure it’s a nice balance to the development work on games. So, here’s what I’ve played over the past couple of years, what I’ve played this year and what I will be playing.

So, what are YOU playing tonight?

THE NAME OF THE GAME

Unless otherwise indicated (*), I have completed the entire game or DLC. I also probably have most, if not all, of the achievements (or trophies) in the game.

2009

360: Silent Hill Homecoming (*), F.E.A.R, F.E.A.R. 2 (Project Origin), Halo Wars (*), Call of Duty: World at War, Guitar Hero: Metallica, Mirror’s Edge, RESIDENT EVIL 5, Guitar Hero: Hits, Batman: Arkham Asylum, Ghostbusters, Fallout 3, [PROTOTYPE], Section 8, Guitar Hero 5, Band Hero, Guitar Hero: Van Halen, Halo 3: ODST, ZOMBIE APOCALYPSE, Brütal Legend, Borderlands, Left 4 Dead 2, Modern Warfare 2, Assassin’s Creed 2, Borderlands – The Zombie Island of Dr. Ned DLC, Borderlands – Mad Moxxi’s Underdome Riot DLC

PS3: inFamous, Uncharted (Drake’s Fortune), Uncharted 2 (Among Thieves), Ratchet & Clank Future: Tools of Destruction

DS: Treasure World, Scribblenauts

2010

360: Darksiders, Mass Effect 2, BioShock 2, Dante’s Inferno, Borderlands – The Secret Armory of General Knoxx DLC, Battlefield: Bad Company 2, Alan Wake, Red Dead Redemption, Blur, BioShock 2 – Rapture Metro Pack DLC, Naughty Bear, LIMBO, BioShock 2 – Protector Trials Pack DLC, BioShock 2 – Minerva’s Den DLC, Plants vs. Zombies, Halo: Reach, Guitar Hero: Warriors of Rock, Borderlands – Claptrap’s New Robot Revolution DLC, Lucha Libre AAA: Héroes del Ring (*), Fallout: New Vegas, Super Meat Boy (*), Rock Band 3 (*), Dead Space Ignition, Call of Duty: Black Ops, ilomilo, Assassin’s Creed: Brotherhood

PS3: God of War 3

DS: Super Scribblenauts

2011

360: Fable 3, Dead Space 2, Bulletstorm

PS3: LittleBigPlanet 2

2011 Pre-Order List

Mortal Kombat, Portal 2, Brink, ICO and Shadow of the Colossus Collection

FIN

I didn’t intend for this to simply be a post listing out a bunch of video game titles, but … well … it is. I love playing video games.

So, what are YOU playing tonight? Tonight I’ll be playing more Bulletstorm Anarchy mode.

You can find more hilarity over on my Twitter account, @CzarneckiD.

PS

Contrary to what you might think in this post, I do have a life outside of video games. I like going out to eat where you can’t count the number of courses on your fingers and toes, actively write open source software (personal, work, other), recently got into beekeeping and love snowboarding.

You read my Leaderboards: A How-To Guide post right? I revealed a dirty little secret about this world … we still don’t have flying cars. But, we do have a good way of creating leaderboards in video games. At least I think so. I want to continue on the leaderboard love train and expand on a concept I call “Meta Leaderboards”. Traditionally, leaderboards rank players using one criteria, e.g. XP, kills, etc. What if you wanted to combine leaderboards to form a larger, aggregate leaderboard? I’m going to show you how to do that.

WHOOMP! (THERE IT IS)

Let’s say you’ve got a game and its multiplayer mode can be played across 5 maps. We’ll also simplify things and say that we’re only ranking players on each map on XP gained when finishing the map. If you want to generate a leaderboard of players ranked by XP who have played in any of the maps, you would need to perform a merge of each of the leaderboards for the 5 maps. If you want to generate a leaderboard of players ranked by XP who have played in each of the maps, you would need to perform an intersection of each of the leaderboards for the 5 maps.

This functionality is now present in the Ruby leaderboard gem.

Check it to wreck it, let’s begin!

Generate a leaderboard of players ranked by XP who have played in any of the maps:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

ruby-1.8.7-p302 > map_1_xp_lb = Leaderboard.new('map_1_xp')
=> #, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}>
ruby-1.8.7-p302 > map_2_xp_lb = Leaderboard.new('map_2_xp')
=> #, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}>
ruby-1.8.7-p302 > map_3_xp_lb = Leaderboard.new('map_3_xp')
=> #, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}>
ruby-1.8.7-p302 > map_4_xp_lb = Leaderboard.new('map_4_xp')
=> #, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}>
ruby-1.8.7-p302 > map_5_xp_lb = Leaderboard.new('map_5_xp')
=> #, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}>
ruby-1.8.7-p302 > map_1_xp_lb.add_member('member_1', 10)
=> true
ruby-1.8.7-p302 > map_2_xp_lb.add_member('member_2', 7)
=> true
ruby-1.8.7-p302 > map_3_xp_lb.add_member('member_3', 7)
=> true
ruby-1.8.7-p302 > map_4_xp_lb.add_member('member_4', 22)
=> true
ruby-1.8.7-p302 > map_5_xp_lb.add_member('member_5', 3)
=> true
ruby-1.8.7-p302 > map_1_xp_lb.merge_leaderboards('any_maps_xp_lb', ['map_2_xp', 'map_3_xp', 'map_4_xp', 'map_5_xp'])
=> 5
ruby-1.8.7-p302 > any_maps_xp_lb = Leaderboard.new('any_maps_xp_lb')
=> #, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}>
ruby-1.8.7-p302 > any_maps_xp_lb.total_members
=> 5
ruby-1.8.7-p302 > any_maps_xp_lb.leaders(1)
=> [{:rank=>1, :member=>"member_4", :score=>22.0}, {:rank=>2, :member=>"member_1", :score=>10.0}, {:rank=>3, :member=>"member_3", :score=>7.0}, {:rank=>4, :member=>"member_2", :score=>7.0}, {:rank=>5, :member=>"member_5", :score=>3.0}]

The default behavior for the merge leaderboards call is to sum the scores for members present in any of the leaderboards. You could also take a maximum or minimum score for members present in any of the leaderboards to generate a high score (or low score) leaderboard across the different maps.

Generate a leaderboard of players ranked by XP who have played in each of the maps:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

ruby-1.9.2-p0 > map_1_xp_lb = Leaderboard.new('map_1_xp')
=> #"localhost", :port=>6379}, @redis_connection=#>
ruby-1.9.2-p0 > map_2_xp_lb = Leaderboard.new('map_2_xp')
=> #"localhost", :port=>6379}, @redis_connection=#>
ruby-1.9.2-p0 > map_3_xp_lb = Leaderboard.new('map_3_xp')
=> #"localhost", :port=>6379}, @redis_connection=#>
ruby-1.9.2-p0 > map_4_xp_lb = Leaderboard.new('map_4_xp')
=> #"localhost", :port=>6379}, @redis_connection=#>
ruby-1.9.2-p0 > map_5_xp_lb = Leaderboard.new('map_5_xp')
=> #"localhost", :port=>6379}, @redis_connection=#>
ruby-1.9.2-p0 > map_1_xp_lb.add_member('member_1', 10)
=> true
ruby-1.9.2-p0 > map_1_xp_lb.add_member('member_2', 19)
=> true
ruby-1.9.2-p0 > map_2_xp_lb.add_member('member_2', 7)
=> true
ruby-1.9.2-p0 > map_3_xp_lb.add_member('member_3', 7)
=> true
ruby-1.9.2-p0 > map_3_xp_lb.add_member('member_4', 17)
=> true
ruby-1.9.2-p0 > map_4_xp_lb.add_member('member_4', 22)
=> true
ruby-1.9.2-p0 > map_4_xp_lb.add_member('member_5', 2)
=> true
ruby-1.9.2-p0 > map_5_xp_lb.add_member('member_5', 3)
=> true
ruby-1.9.2-p0 > map_1_xp_lb.intersect_leaderboards('all_maps_xp_lb', ['map_2_xp', 'map_3_xp', 'map_4_xp', 'map_5_xp'])
=> 0
ruby-1.9.2-p0 > all_maps_xp_lb = Leaderboard.new('all_maps_xp_lb')
=> #"localhost", :port=>6379}, @redis_connection=#>
ruby-1.9.2-p0 > all_maps_xp_lb.total_members
=> 0
ruby-1.9.2-p0 > all_maps_xp_lb.leaders(1)
=> []
ruby-1.9.2-p0 > map_2_xp_lb.add_member('member_1', 7)
=> true
ruby-1.9.2-p0 > map_3_xp_lb.add_member('member_1', 8)
=> true
ruby-1.9.2-p0 > map_4_xp_lb.add_member('member_1', 8)
=> true
ruby-1.9.2-p0 > map_5_xp_lb.add_member('member_1', 2)
=> true
ruby-1.9.2-p0 > map_1_xp_lb.intersect_leaderboards('all_maps_xp_lb', ['map_2_xp', 'map_3_xp', 'map_4_xp', 'map_5_xp'])
=> 1
ruby-1.9.2-p0 > all_maps_xp_lb = Leaderboard.new('all_maps_xp_lb')
=> #"localhost", :port=>6379}, @redis_connection=#>
ruby-1.9.2-p0 > all_maps_xp_lb.total_members
=> 1
ruby-1.9.2-p0 > all_maps_xp_lb.leaders(1)
=> [{:member=>"member_1", :score=>35.0, :rank=>1}]
ruby-1.9.2-p0 >

Notice in our first intersection here of the leaderboards, no one member has played in all of the different maps, so we have an empty leaderboard. After a member has played in all of the maps, we now have a populated leaderboard.

The default behavior for the intersect leaderboards call is to sum the scores for members present in any of the leaderboards. You could also take a maximum or minimum score for members present in any of the leaderboards to generate a high score (or low score) leaderboard across all of the different maps.

CAVEATS

Latin? Seriously?

When you create a merged or intersected leaderboard, this new leaderboard does not track changes made to any of the leaderboards that were combined. If you want to update the merged or intersected leaderboard, you have to perform the merge or intersection again. As such, generating meta leaderboards is something you probably want to do at some regular interval, not after every score update. Just make sure your users know when the leaderboard is going to be updated.

Unlike my previous post on leaderboards, I didn’t run any performance metrics on the merge or intersection operations. I know I need to do that at some point and I will. So … stay tuned?

FIN

Meta leaderboards are nothing more than an aggregate leaderboard. I’ve shown merge and intersection operations that allow you to create different types of meta leaderboards. Meta leaderboards can be generated on a regular, but not too frequent basis so as not to tax your servers too bad.

You can find more hilarity over on my Twitter account, @CzarneckiD.

Before joining Agora Games in May of 2008, I worked for a small company called General Electric for nearly 11 years in their Research and Development and Healthcare organizations. During that time, my interest in video games was re-kindled. I can remember the holiday shopping season when the slim PS2 was made available and OMFG I HAD TO HAVE ONE TO PLAY KATAMARI DAMACY! Thankfully I snagged one on Amazon.com and it was all downhill from there I guess. I got heavily into some now classics like God of War, God of War 2, Ico, Shadow of the Colossus, Okami and others. And then out of nowhere, I stumbled upon Dead Rising and OMFG I HAD TO BUY AN XBOX 360 TO KILL THE ZOMBIES. As it turns out, you could only reasonably play Dead Rising on an HDTV, so I attached that game onto a rocket ship and sent it flying into the sun (I still own Dead Rising, but have never completed it due to other issues that I will save for another blog post). I started playing the other XBOX 360 games that were all the rage at that time like the Halo series, Guitar Hero 2 and Guitar Hero 3, and others. However, I never really thought about joining the game industry even though there were 3 well-established video game companies here in upstate NY, Vicarious Visions, 1st Playable, and Agora Games (even though I hadn’t known of Agora Games … yet).

CRAIGSLIST: GOOD FOR FINDING JOBS (winking smiley face)

See what I did there? Get your mind out of the gutter! Anyhoo …

On the last project I was on at General Electric, I was working many long hours, many days of the week. A lot more than I ever wanted to. It wasn’t fun. At all. Life is too short to not do what you love. So, I started searching locally for a new job. I happened upon a job posting on Craigslist for a Software Engineer at Agora Games. I had no experience in the tools and frameworks outlined in the job posting, but I had been working in software professionally for nearly 15 years … and I loved video games … so that counts for something right? As it turns out, it does.

I submitted my resume on a Tuesday. I submitted a solution to a Pokemen traveling salesman problem (COLLECT THEM ALL) the following day. My first interview was the following Monday. Aside: In case you don’t know about Agora Games, one of the franchises we work with is Guitar Hero developing certain online and community functionality. So, here I am going into the interview thinking that not only am I going to be grilled technically, but I’ll probably have to demonstrate my skills playing “Through The Fire and Flames” on Expert. My first interview went well. I didn’t have to demonstrate my Guitar Hero prowess, but I did have to explain how I would implement leaderboards to my now colleague, Ola Mork (who’s laughing now Ola – see my other post on Leaderboards: A How-To Guide). I had a second lunch interview the following Monday with our CEO again and our new CTO. After lunch we went back to the studio, I had an offer, and I drove home and gave my 2 weeks notice to General Electric.

GO OUT WITH A LITTLE PIZAZZ

So, General Electric’s e-mail system has a Global Address List (GAL). At the time, General Electric owned NBC. I think they still have some stake in NBC, but whatever, not the point. The point is that almost all of NBC’s personalities like the entire cast of Saturday Night Live, the NBC Nightly News folks, Conan O’ Brien (at the time still with NBC), Jay Leno, and more, all had e-mail addresses in the GAL. So, in my parting e-mail to all my colleagues at GE, I added every single celebrity I could think of. What did I care? I was leaving the company. At worst, they never read that e-mail. At best, I get a “Congratulations!” reply. As a fan of South Park, I ended my e-mail with the following:

I will have to defer any “words of wisdom” to those of one Eric Cartman (yes … of South Park):

“Follow your dreams. You can reach your goals. I’m living proof. Beefcake. BEEFCAKE!”

BOOM!

TEQUILA ANYONE?

If you can, line up your start date at your new company with an awesome “holiday”. I started at Agora Games on May 5th, 2008 aka Cinco de Mayo. I was joining the Guitar Hero team and we were working feverishly on updates for Guitar Hero: Aerosmith. On my first day I got to play Guitar Hero: World Tour (possibly unannounced at the time). Think hard about how you spent your first afternoon at your new job?

FIN

Joining the video game industry was and continues to be one of the best decisions I have ever made. Ever.

I realize the long and short of this blog post is that I got into the video game industry by applying for a job in the video game industry on Craigslist, but maybe that’s the point? If you want a job in the video game industry, apply for a job in the video game industry. Whether you’re a programmer or an artist or a musician or a writer or a player, there are jobs for any discipline.

So, apply.

Do it.

Now.

Leaderboards in video games are an interesting beast. They are used to rank players against one another using some quantifiable criteria such as # of kills, XP, total time played, points in a level, etc. Leaderboards have been an intimate part of my life for almost 3 years now at Agora Games. As such, I’m always looking for ways to improve our leaderboard technology and answer questions like: How do you efficiently insert and rank players in real time? How do you efficiently update and re-rank players in real time? How do you retrieve information about a specific member of a leaderboard? How do you retrieve information about a player and players around them in a leaderboard?

It was the morning of January 1, 2011, I had literally ridden the short bus back from my New Year’s dinner some hours earlier, and I woke up with a desire to make the world more awesome. It’s 2011 and we still don’t have flying cars … so ,,|, future. Where to start? Leaderboards.

LEADERBOARDS USING REDIS

At the studio I had discussed with colleagues the possibility of using Redis, an advanced key-value storage engine, for leaderboards. In less than an hour, I had the set of Redis commands using their sorted set data type (a set of data that is sorted based on an associated “score”) to perform operations on leaderboards such as:

• Retrieving general information about a leaderboard such as total members or total pages
• Adding or removing members from a leaderboard
• Retrieving information about a member in the leaderboard such as their rank or score
• Updating score information for a member in the leaderboard
• Retrieving an arbitrary page of leaders from the leaderboard
• Retrieving the leaders around a given member in a leaderboard, also known as an “Around Me” leaderboard
• Retrieving information for an arbitrary set of members in a leaderboard, e.g. How do my friends compare against me?

The result of all this is now encapsulated in the leaderboard gem (a Ruby library) that I open sourced on GitHub. There are Redis libraries in a number of programming languages such as C, C# C++, Lua, etc. The leaderboard gem is merely a semantic wrapper around the underlying Redis commands. It is left as an exercise to the reader to port the leaderboard code to your favorite programming language.

RUBBER MEETS ROAD

Here’s how you’d actually use the library to generate and interact with a ‘highscores’ leaderboard as outlined above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

Create a new leaderboard or attach to an existing leaderboard named 'highscores':

   ruby-1.8.7-p302 > highscore_lb = Leaderboard.new('highscores')
   => #<Leaderboard:0x1018e4250 @page_size=25, @port=6379, @host="localhost", @redis_connection=#<Redis client v2.1.1 connected to redis://localhost:6379/0 (Redis v2.1.10)>, @leaderboard_name="highscores"> 
   
If you need to pass in options for Redis, you can do this with the redis_options parameter:

  redis_options = {:host => 'localhost', :port => 6379, :password => 'password', :db => 'some_redis_db'}
  highscore_lb = Leaderboard.new('highscores', redis_options[:host], redis_options[:port], Leaderboard::DEFAULT_PAGE_SIZE, redis_options))

You can set the page size to something other than the default page size (25):

  ruby-1.8.7-p302 > highscore_lb.page_size = 5
   => 5 
  ruby-1.8.7-p302 > highscore_lb
   => #<Leaderboard:0x1018e2130 @leaderboard_name="highscores", @page_size=5, @port=6379, @redis_connection=#<Redis client v2.1.1 connected to redis://localhost:6379/0 (Redis v2.1.10)>, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}>   
  
Add members to your leaderboard:

  ruby-1.8.7-p302 > 1.upto(10) do |index|
  ruby-1.8.7-p302 >     highscore_lb.add_member("member_#{index}", index)
  ruby-1.8.7-p302 ?>  end
   => 1 
  
Get some information about your leaderboard:

  ruby-1.8.7-p302 > highscore_lb.total_members
   => 10 
  ruby-1.8.7-p302 > highscore_lb.total_pages
   => 1 
  
Get some information about a specific member(s) in the leaderboard:

  ruby-1.8.7-p302 > highscore_lb.score_for('member_4')
   => 4.0 
  ruby-1.8.7-p302 > highscore_lb.rank_for('member_4')
   => 7 
  ruby-1.8.7-p302 > highscore_lb.rank_for('member_10')
   => 1 
  
Get page 1 in the leaderboard:

  ruby-1.8.7-p302 > highscore_lb.leaders(1)
	 => [{:member=>"member_10", :rank=>1, :score=>"10"}, {:member=>"member_9", :rank=>2, :score=>"9"}, {:member=>"member_8", :rank=>3, :score=>"8"}, {:member=>"member_7", :rank=>4, :score=>"7"}, {:member=>"member_6", :rank=>5, :score=>"6"}, {:member=>"member_5", :rank=>6, :score=>"5"}, {:member=>"member_4", :rank=>7, :score=>"4"}, {:member=>"member_3", :rank=>8, :score=>"3"}, {:member=>"member_2", :rank=>9, :score=>"2"}, {:member=>"member_1", :rank=>10, :score=>"1"}]

Add more members to your leaderboard:

  ruby-1.8.7-p302 > 50.upto(95) do |index|
  ruby-1.8.7-p302 >     highscore_lb.add_member("member_#{index}", index)
  ruby-1.8.7-p302 ?>  end
   => 50 
  ruby-1.8.7-p302 > highscore_lb.total_pages
   => 3 
  
Get an "Around Me" leaderboard for a member:

  ruby-1.8.7-p302 > highscore_lb.around_me('member_53')
	 => [{:member=>"member_65", :rank=>31, :score=>"65"}, {:member=>"member_64", :rank=>32, :score=>"64"}, {:member=>"member_63", :rank=>33, :score=>"63"}, {:member=>"member_62", :rank=>34, :score=>"62"}, {:member=>"member_61", :rank=>35, :score=>"61"}, {:member=>"member_60", :rank=>36, :score=>"60"}, {:member=>"member_59", :rank=>37, :score=>"59"}, {:member=>"member_58", :rank=>38, :score=>"58"}, {:member=>"member_57", :rank=>39, :score=>"57"}, {:member=>"member_56", :rank=>40, :score=>"56"}, {:member=>"member_55", :rank=>41, :score=>"55"}, {:member=>"member_54", :rank=>42, :score=>"54"}, {:member=>"member_53", :rank=>43, :score=>"53"}, {:member=>"member_52", :rank=>44, :score=>"52"}, {:member=>"member_51", :rank=>45, :score=>"51"}, {:member=>"member_50", :rank=>46, :score=>"50"}, {:member=>"member_10", :rank=>47, :score=>"10"}, {:member=>"member_9", :rank=>48, :score=>"9"}, {:member=>"member_8", :rank=>49, :score=>"8"}, {:member=>"member_7", :rank=>50, :score=>"7"}, {:member=>"member_6", :rank=>51, :score=>"6"}, {:member=>"member_5", :rank=>52, :score=>"5"}, {:member=>"member_4", :rank=>53, :score=>"4"}, {:member=>"member_3", :rank=>54, :score=>"3"}, {:member=>"member_2", :rank=>55, :score=>"2"}]	

Get rank and score for an arbitrary list of members (e.g. friends):

  ruby-1.8.7-p302 > highscore_lb.ranked_in_list(['member_1', 'member_62', 'member_67'], true)
   => [{:rank=>55, :member=>"member_1", :score=>1.0}, {:rank=>33, :member=>"member_62", :score=>62.0}, {:rank=>28, :member=>"member_67", :score=>67.0}]   

Let’s cover a few more scenarios in-depth as to what calls you might make.

I’ve based Scenario 1 and 2 from LittleBigPlanet. NOTE: I have absolutely no idea if this is what actually happens in that game.

Scenario 1: Player finishes a level in game, upload player’s score (add_member), display to the player the scores around them (set page_size and use around_me call for player).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

fossil:leaderboard dczarnecki$ irb
ruby-1.8.7-p302 > require 'rubygems'
 => true 
ruby-1.8.7-p302 > require 'leaderboard'
 => true 
ruby-1.8.7-p302 > highscore_lb = Leaderboard.new('highscores')
 => #<Leaderboard:0x1011b23b0 @leaderboard_name="highscores", @page_size=25, @port=6379, @redis_connection=#<Redis client v2.1.1 connected to redis://localhost:6379/0 (Redis v2.0.4)>, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}> 
ruby-1.8.7-p302 > 1.upto(10) do |index|
ruby-1.8.7-p302 >     highscore_lb.add_member("member_#{index}", index)
ruby-1.8.7-p302 ?>  end
 => 1 
ruby-1.8.7-p302 > highscore_lb.add_member('DavidCzarnecki', 6)
 => true 
ruby-1.8.7-p302 > highscore_lb.page_size = 5
 => 5 
ruby-1.8.7-p302 > highscore_lb.around_me('DavidCzarnecki')
 => [{:member=>"member_7", :score=>"7", :rank=>4}, {:member=>"member_6", :score=>"6", :rank=>5}, {:member=>"DavidCzarnecki", :score=>"6", :rank=>6}, {:member=>"member_5", :score=>"5", :rank=>7}, {:member=>"member_4", :score=>"4", :rank=>8}] 
ruby-1.8.7-p302 > 

Scenario 2: Player finishes same level in game, update player’s score (add_member), display to the player the scores around them (set page_size and use around_me call for player). Note, you can call add_member again, because under the hood in Redis, it will just update the player’s score (if it exists) and re-rank.

1
2
3
4
5
6
7
8

ruby-1.8.7-p302 > highscore_lb.add_member('DavidCzarnecki', 15)
 => false 
ruby-1.8.7-p302 > highscore_lb.page_size = 5
 => 5 
ruby-1.8.7-p302 > highscore_lb.around_me('DavidCzarnecki')
 => [{:member=>"DavidCzarnecki", :score=>"15", :rank=>1}, {:member=>"member_10", :score=>"10", :rank=>2}, {:member=>"member_9", :score=>"9", :rank=>3}, {:member=>"member_8", :score=>"8", :rank=>4}, {:member=>"member_7", :score=>"7", :rank=>5}] 
ruby-1.8.7-p302 > 

I’ve based Scenario 3 and 4 from Call of Duty: Black Ops playing a wager match. NOTE: I have absolutely no idea if this is what actually happens in that game. Also, in Scenario 3 and 4, you could certainly use the add_member call with the player’s total earnings. I simply want to illustrate the delta call (change_score_for).

Scenario 3: Players finish a match and you want to update their winnings (change_score_for) for that match.

1
2
3
4
5
6
7
8
9
10

ruby-1.8.7-p302 > total_money_lb = Leaderboard.new('total_money')
 => #<Leaderboard:0x101180680 @leaderboard_name="total_money", @page_size=25, @port=6379, @redis_connection=#<Redis client v2.1.1 connected to redis://localhost:6379/0 (Redis v2.0.4)>, @host="localhost", @redis_options={:host=>"localhost", :port=>6379}> 
ruby-1.8.7-p302 > total_money_lb.change_score_for('DavidCzarnecki', 15)
 => "15" 
ruby-1.8.7-p302 > total_money_lb.change_score_for('ChristianArca', 7)
 => "7" 
ruby-1.8.7-p302 > total_money_lb.leaders(1)
 => [{:member=>"DavidCzarnecki", :score=>"15", :rank=>1}, {:member=>"ChristianArca", :score=>"7", :rank=>2}] 
ruby-1.8.7-p302 > 

Scenario 4: Players finish another wager match and you want to update their winnings (change_score_for) for that match. NOTE: If you use negative values, the score is decremented.

1
2
3
4
5
6
7
8

ruby-1.8.7-p302 > total_money_lb.change_score_for('DavidCzarnecki', -8)
 => "7" 
ruby-1.8.7-p302 > total_money_lb.change_score_for('ChristianArca', 16)
 => "23" 
ruby-1.8.7-p302 > total_money_lb.leaders(1)
 => [{:member=>"ChristianArca", :score=>"23", :rank=>1}, {:member=>"DavidCzarnecki", :score=>"7", :rank=>2}] 
ruby-1.8.7-p302 > 

BUT … REDIS?

It’s fast because data lives in-memory. It’s durable because you can configure it to write to disk. It’s scaleable because you can setup master-slave replication to have multiple read-only slaves. I assume all of these are important to you as game developers.

All of this amounts to a Guarantee Fairy right? It makes a man feel good. You figure you put this blog post under your pillow at night, the Guarantee Fairy might come by and leave a quarter, am I right? What’s my point? The point is, how do you know the fairy isn’t a crazy glue sniffer? “Building model airplanes” says the little fairy; well, we’re not buying it. He sneaks into your house once, that’s all it takes. The next thing you know, there’s money missing off the dresser, and your daughter’s knocked up. I’ve seen it a hundred times.

Wait … what?

SCIENCE. IT WORKS B*TCHES

One of the things I love about Redis is that it’s backed by science … mathematics and computer science. In particular, all of the commands give their respective time complexity in Big O notation. If you don’t know what that is, consider yourself fired. Not really. But … still. It’s important because you want to know how long operations are going to take given a particular data set size. This might be useful information from a game developer’s perspective to figure out average or worst-case scenario time estimates for interacting with a leaderboard based on total expected players.

At the request of one of my colleagues, Ola Mork, I did some basic benchmarking. Insert 10 million sequential scores into a leaderboard (in practice you will probably not be doing this all at once) and determine the average time to request an arbirtrary page from the leaderboard. Insert 10 million random scores into a leaderboard (again, in practice you will probably not be doing this all at once) and determine the average time to request an arbitary page from the leaderboard.

The long and short of it is that it’s fast. Negligible difference in terms of leaderboard page retrieval.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

10 million sequential scores insert:

  ruby-1.8.7-p302 > insert_time = Benchmark.measure do
  ruby-1.8.7-p302 >       1.upto(10000000) do |index|
  ruby-1.8.7-p302 >           highscore_lb.add_member("member_#{index}", index)
  ruby-1.8.7-p302 ?>      end
  ruby-1.8.7-p302 ?>  end
   => #<Benchmark::Tms:0x101605660 @label="", @stime=173.61, @total=577.52, @real=911.718175172806, @utime=403.91, @cstime=0.0, @cutime=0.0> 
  
Average time to request an arbitrary page from the leaderboard:

  ruby-1.8.7-p302 > requests_to_make = 50000
   => 50000 
  ruby-1.8.7-p302 > lb_request_time = 0
   => 0 
  ruby-1.8.7-p302 > 1.upto(requests_to_make) do
  ruby-1.8.7-p302 >       lb_request_time += Benchmark.measure do
  ruby-1.8.7-p302 >           highscore_lb.leaders(rand(highscore_lb.total_pages))
  ruby-1.8.7-p302 ?>      end.total
  ruby-1.8.7-p302 ?>  end
   => 1 
  ruby-1.8.7-p302 > 
  ruby-1.8.7-p302 >   p lb_request_time / requests_to_make
  0.001808
   => nil 
  
10 million random scores insert:

  ruby-1.8.7-p302 > insert_time = Benchmark.measure do
  ruby-1.8.7-p302 >       1.upto(10000000) do |index|
  ruby-1.8.7-p302 >           highscore_lb.add_member("member_#{index}", rand(50000000))
  ruby-1.8.7-p302 ?>      end
  ruby-1.8.7-p302 ?>  end
   => #<Benchmark::Tms:0x10164ebf8 @label="", @stime=172.94, @total=577.91, @real=1356.57155895233, @utime=404.97, @cstime=0.0, @cutime=0.0> 
  
Average time to request an arbitrary page from the leaderboard:

  ruby-1.8.7-p302 > requests_to_make = 50000
   => 50000 
  ruby-1.8.7-p302 > lb_request_time = 0
   => 0 
  ruby-1.8.7-p302 > 1.upto(requests_to_make) do
  ruby-1.8.7-p302 >       lb_request_time += Benchmark.measure do
  ruby-1.8.7-p302 >           highscore_lb.leaders(rand(highscore_lb.total_pages))
  ruby-1.8.7-p302 ?>      end.total
  ruby-1.8.7-p302 ?>  end
   => 1 
  ruby-1.8.7-p302 > 
  ruby-1.8.7-p302 >   p lb_request_time / requests_to_make
  0.00179680000000001
   => nil 

FIN

Leaderboards may seem complicated, but they are not. Wrapping software like Redis with a set of semantic commands, we have a near complete set of use cases covered for how we might interact with a leaderboard. Finally, it’s fast.

Take a look at the leaderboard gem code and get involved. Comments or code. The choice is yours.

You can find more hilarity over on my Twitter account, CzarneckiD.