It’s this abundance of choice and the issues resulting from making the wrong choice that drives my involvement with SkillSet. Good courses need as many opportunities as possible to stand out from the crowd especially when prospective students may have to narrow down the Universities they’ll investigate let alone apply too.
 
 

What Is The Accreditation Process?

I’ve been a lead evaluator for the SkillSet accreditation process for quite a few years now. The process (I’ll keep it short) involves an application by the University to SkillSet, a paper based investigation into the University covering the skills taught, the attainment and employability of students (amongst other things) followed by an on-site visit by an evaluation team. This visit involves interviews with staff and students, examination of the work done and the facilities available with a recommendation to accredit based on their findings.

This evaluation team is always made up of active game developers working in the discipline the course focuses on.

Courses have and will continue to be rejected at various stages of the process if they are not up to scratch and the criteria used is often very strict with a high barrier for entry. Courses have to be producing quality graduates with the skills suitable for the industry before they can even start to think about applying for accreditation. Cross skill courses (those teaching programming, art and design in a single degree) have never been accredited and never will.

The process and documentation is publicly available so you can have a more detailed look here.
 
 

Student Choice

There are a lot of Universities in the UK and a large number of them provide a some kind of game related technical course. Unfortunately a lot of them don’t provide a high enough level of education to warrant the time and money students spend on them. Accreditation awards allows students to quickly narrow down the kinds of Universities they should be looking at and to spend the time they have investigating the best rather than trying to simply find the ones worthy of their time.

Word of mouth and past experience all goes into this but it still takes time that could be better spent especially when students will be applying to Universities at the same time as working towards their A-Levels, a period of time which may well be the most stressful time they’ve had in their academic career so far.

An accredited University at least gives them the knowledge that they’ll get the right kind of education allowing them to focus on finding one that best suits them rather than anything else.
 
 

Industry Involvement

A lot of developers want to get involved with the education of future game makers and University partnerships such as guest lectures and industry panels are one of the best ways to do that. Universities like industry involvement and some developers can end up being overwhelmed with requests especially if they are already working with other courses and word starts to spread. And because a developers time is so valuable, it helps to be able to target the Universities that we know are already providing the kinds of skills students will need in the future.

Building on a quality foundation allows much more scope for growth than having to start from the bottom and working upwards.

But you might think this is a path to ruin. If the industry only helps accredited courses surely none can become accredited because no-ones willing to work with them to get there!

But that’s not the case. Bigger companies such as Sony, MS, Blitz, Codemasters and others work with those up and coming courses, allowing them to get the point where they can apply for or work towards accreditation. Once that happens, it becomes a positive feedback loop, as the course gets better and gets accredited it leads to more industry involvement which leads to a better course…
 
 

It’s The Skills Stupid

The argument between game focused and traditional Computer Science courses is always the same and is usually spot on. Take a CS course over a game course because in a few years you may not want to work in the industry and the skills you learn on a CS could will put you in a stronger position should you want to do something else.

Without a doubt this is a good argument to make but one that I hope accreditation can resolve. In every evaluation I’ve taken part in the skills we look for and the modules on display show that the course could quite easily be rebranded as ‘Computer Science with a games slant’ rather than ‘Games Technology with a little bit of Computer Science thrown in’. As a result the skills taught will still put the student in a good position should they decided the industry is not for them and while a ‘Games Technology’ degree won’t look as good as a ‘Computer Science’ degree on a CV, accreditation should allow it to grow in stature depending on which University awarded the degree.

As someone who used to teach ICT at these level I have first hand experience of the kind of topics, software and skills taught to children in classes across the UK, and agree 100% that, as a subject, ICT should be kicked to the kerb and approached from a completely different angle.

And we’re certainly heading in the right direction.

The Next Gen. report by Livingstone and Hope details how we can provide the skills our children require to take our industries to the next level and beyond and even the current Education Secretary Michael Gove is making moves in the right direction. Change is coming and it’s change for the better.

But we must be careful not to rush these changes and to make sure the structure and content is right, that suitable facilities are available and we have the right people in place before we start this process otherwise we’ll be back to square one within a couple of years.
 
 

You Can’t Teach What You Don’t Know

The largest problem facing the introduction of a new, highly scientific and difficult subject is getting the right people in place to teach. And the most important word in that sentence is ‘right’.

Sure we can use existing ICT teachers but it’s clear that the skills required to teach ICT are not the same skills required to teach a Computing GCSE. Also, don’t get me started on the number of ICT teachers who don’t actually know how to use a computer…

We can draft in science and maths teachers to teach elements of the course others are unable to but those teachers already have working place directives in place limiting how many hours they can teach and those hours will already be more than accounted for.

If we really want to create a next generation of developers is passionate and knowledgeable educators pushing that passion onto the open minds in front of them. Sure, anyone can learn enough to teach a subject, but it wont be exciting, it won’t be boundary defining and it won’t make those pupils crave more. Sure, some teachers will really take to this, will develop that passion along with their students, but the majority won’t and as a result the subject could easily be seen as stale, boring and dull.

So the ideal solution is to hire for the shortfall but with approximately 3,900 state run secondary schools in the UK and with many schools having the need for more than one or two teachers per subject that’s a big recruitment drive.
 
 

Custom Content Is Risky But Rewarding

One of the most interesting aspects of this push towards a more computing focused subject is the idea of having a more flexible curriculum. Now this could mean a myriad of things though I’m taking it to mean schools being given the ability to cherry pick the elements they want to teach and to ignore elements they don’t.

This is extremely useful especially with a lack of specialist teachers and it also tackles one of the main criticisms of the National Curriculum, that it creates boilerplate content and restricts creativity and freedom in the classroom.

So by allowing schools to be flexible with what they teach we allow our teachers to experiment and push the boundaries of our children’s imaginations, if they have the knowledge to do that.

But it also opens up the possibility of a (and I hate myself for using this awful and overused phrase) postcode lottery where some schools are teaching more valuable skills than others. It complicates the act of awarding consistent and meaningful grades across the country and it could lead to stagnation as some schools resist pressure to improve as technology moves on.

Though you could make that argument for the education system as a whole due to the varying levels of skills between teachers of all subjects in all schools.
 
 

We’re A Fickle Bunch

With the issues highlighted above there is every chance that if we rush headlong into these changes we risk the first years being less than stellar as people find their feet, new teachers are recruited and less talented teachers are let go.

And this one worries me the most.

With every change of Government, or in many cases with every change in Education Secretary, schools are given newer mandates, newer targets and newer goals. And it’s always in response to a perceived failure by the last Government/Secretary.

If the introduction of a Computer focused subject is seen as a failure in any way then the next Education Secretary will trip over themselves to ‘fix’ the situation. Not by removing the subject but by constantly tweaking and ‘refocusing’, leading to a course that drives schools (and good teachers) away from the subject and turns it into something resembling what we already have.
 
 

ICT Isn’t All Bad

ICT isn’t just Word and Excel. There’s some really interesting content in there that shouldn’t just be discarded out of hand.

I’ve taught lessons in web design and planning using both graphical editors (Dreamweaver at the time) and text editors. We used some rudimentary programming tools in the early years (getting frogs safely across the road by managing traffic light systems) and video editing tools in later years. All of these were exciting, interesting and (usually) generated some pretty interesting results!

I’m confident that these elements will not be lost no matter what comes next, but we must be careful not to discard what we have and what actually works for want of a ‘better’ system.
 
 

Word Processing Skills Are Important

Something we need to acknowledge is that (this might nark some people) word processing and spreadsheet skills, and the ability to use MS products, are important skills that pupils need to have. Whether we like it or not the majority of the world uses MS Office and while this might change in the future having the ability to use these tools improves a child’s employability, their work rate at school and their computer literacy in general.

But the removal of these ‘skills’ from a specific subject is nothing but good news. By teaching these skills as part of other subjects will lead them to be seen as more natural tools that can be used in a wide range of situations rather than just in their ICT lesson.

But we need to make sure that time is available in all subjects to do this and acknowledge that it’ll take a whole school approach to achieve, something that will vary greatly on a school by school basis.

 
 
I’m certainly excited about how these changes will alter how our children see and use computers on a daily basis and how that can only improve the industry in general. But there are significant challenges that must be faced before we can move forward, confidently, and create a true next generation of developers.

If developers do not feel confident in the reasons behind their work, if they don’t fully understand how their work will fit into the project as a whole or indeed when it will be required, the team will produce lower quality work with last minute changes and requirements fostering an atmosphere of distrust and crunch.

But communication is one of the most difficult things to get right but so costly when it’s done wrong.

The following are methods I’ve used over the years to try and improve communication within the team. I’d be very interested to hear other ways people have tried too!

 
Team Wiki
Having an open Wiki that people can contribute information and notes is a good idea for documentation and persistent information. It is not a good tool for perishable short term information. Documentation on team processes (getting the latest build, creating submissions, setting up PC’s etc.) is usually the kind of information that finds a home on a wiki and while it’s useful it’s not something that affects the team on a daily basis.

And as it requires people to physically search for the information in the long term people don’t bother looking for new information on a regular basis.

As a result, the Wiki is useful but doesn’t actually improve the day to day communication on a team.

 
Blogs
We have a team blog that people update about 2 or 3 times a week, usually discussing their recent work, posting up screen shots or letting people know the state of the project. It’s a nice simple way to push information to the team, though it does require everyone to contribute to the blog to get good cross team communication going.

Discussions can take on a life of their own which is actually a good way to gauge buy in into a project but can’t be used to judge the success of the process.

But you’d be amazed how many people don’t have any kind of RSS feed reader set up…

 
Micro-blogging
Internal mico-blogging tools like Yammer or Status.net allow people to quickly thrown up what they’re working on, problems encountered or general team information. The best thing about micro-blogging is it’s push communication style with peoples updates being automatically feed to clients which people can update as much as they want (I usually recommended at least twice a day).

But so far I’ve had very little success with micro-blogging tools in a team environment.

Not because the idea was bad (when it worked it worked well) but I’ve yet to find a service where the official client is anywhere near usable and able to filter out the information people don’t want to read. Without a good way to filter and push information where you want it (like all the best third party Twitter tools do) it either becomes an information overload or a sea of noise, neither of which improve communication.

 
Wall Displays
Walls are valuable real estate especially in an Open Plan office and I’ve rarely seen them used to their full potential. But highly visible walls in the middle of a team area are one of the best ways to communicate information across a team.

As an example on my current project we have the entire timeline of the project (it’s only a short one) with dates/deliverables clearly indicated, a ‘where we are right now’ marker and a description – feature by feature – of what is required for a given milestone.

Next to that we have our sprint wall which is our most ‘live’ wall display. But position is key and in our case the sprint wall’s impact on the team is reduced due to it’s rather awkward position between a big TV, a constantly spinning fan and quite a lot of server machines. But I did say wall space is valuable real estate and it’s always hard to find a compromise between distance from team and accessibility.

 
Morning Meetings
Morning meetings are one of the best ways to push information across a team but I’ve found that you need to follow a few rules to make them really valuable.

• Keep the groups small. I’ve lost count of the number of 20+ people stand up meetings I’ve seen where the majority of the ‘participants’ are looking bored or simply waiting for their turn. If you’re groups are not small, the information is less targeted and much less relevant, meaning more information is lost than actually passed around the team.
• Keep them focused. There is nothing worse than 1 out of the 6 people speaking for 15 minutes about the most intimate implementation details, leaving everyone else itching to get back to their seats to carry on working.
• Don’t make them reporting sessions. If everyone is talking at a single person (usually their manager) take the manager out for a while and get people used to talking to each other as it makes it much more likely for people to take in what is being said.

 
Milestone Reviews
I love the concept of a milestone review. Everyone playing the game, lively discussions about what went right, what went wrong and what we can do better next time. But it’s easy for these to be less than stellar if not approached in the right way.

If these reviews are not focused, maybe even as structured as a schedule or points to cover, people may start to feel unsure as to what they can comment on or what exactly they should be doing. You’ve also got to make sure that people feel comfortable both giving and taking criticism and manage the situations when that goes pear shape (and sometimes it will).

I’ve found that when done right, when people contribute to discussions and when people can (importantly) see change as a result of these reviews, the quality of information coming out of them is invaluable. It also has the added bonus of making people feel like they are making a difference to the team and allowing their thoughts to be heard.

 
Sprint Planning
The days of managers sitting in a room building up a schedule and dishing it out to the developers is (almost) long gone. And there’s good reason for it.

Getting a group of developers (again with the morning meetings it needs to be small and focused) to discuss, schedule and plan the work ahead significantly improves the knowledge people have of what is happening across the team. Again, if people feel that have a say in how work will be implemented, how it will be assigned and when it’ll be done by is vital to spreading information about the project and the work being done.

 
So those are a couple of methods I’ve used to try and improve communication and information across the team. I’m sure there are plenty more (and I’ve tried a few which have been colossal failures) so what methods have you used and how well did they work out?

Title image by deathtogutenberg.  Used under license.

Refactor: Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.

I pretty much wish that word had never been invented. The above definition (taken from Martin Fowler’s Refactoring Home Page) seems to have lost nearly all meaning when used in day to day programming conversations.

To further describe the original definition of refactoring:

Its heart is a series of small behavior preserving transformations. Each transformation does little, but a sequence of transformations can produce a significant restructuring.

But when someone comes to me and says “I’ll just refactor that…” I can no longer assume to know what they’ll be doing. Based on how people now use this term it could be any one of a number of things including (in order of violence)

• Making small changes to make the system more understandable, simpler and easier to use without affecting it’s perceived behaviour
• Optimising elements of the system which should hopefully(?) not effect the external behaviour
• Making wide ranging changes to a system which will effect elements of its behaviour
• Deleting the whole thing and starting again without even looking at what we have now

It seems like there is a reluctance to use the words ‘re-write’, ‘modify’, ‘break’ or (in a some cases) ‘trash’ when discussing intentions towards an existing system. As a result what was a very descriptive and clear term has lost all meaning and resulted in something what can no longer be used to clearly describe a useful and often necessary development process.

Do I wish the word hadn’t been invented? Maybe I just wish it hasn’t taken on an image that ‘refactoring’ is somehow cooler or more elite than simply saying what you mean when describing what you’ll be doing.
 

I understand that Scrum has been applied mainly to software products and that the elimination of “specialties” means that the database programmer, UI programmer and QA engineer should all be able to perform each other’s roles equally. This is valid.

Now I’m concerning myself with only the technical side of an agile team but I’ve seen this raised in a number of different agile circles. In those cases there seems to be the impression that swapping a database, physics or audio developer with any other specialisation like UI, animation or graphics and an agile team should be able to roll up their sleeves and perform the different roles to the same level with the same level of outcome.

To me, this is emphasised in how the product backlog is often used, which is a priority and risk ordered document that doesn’t take into account the skill set of the team that’ll be working on the final product.

Processes such as pair programming, constant re-factoring and code reviews (to name but a few) seem to be seen as ways to not only communicate intent and project information but also skillset and ability across an entire discipline.

So What Do Specialists Bring?

But we have specialist developers for a reason. They are great at what they do, they understand the area in which they work and they know how to get the best results in the shortest amount of time. They have a passion for the area they are focusing in which usually means they’ll go a step further to research their area and keep up with developments which other developers may not have the time or the understanding to do.

By spreading your talent thin and assuming that people can fill each others shoes leads to the following issues

• You are not respecting the knowledge, skill, experience and passion that a specialist can bring to their work and as a result not respecting the developer themselves
• You’re reducing the impact these people can have on a team and it’s often the experienced specialists that inspire younger members of the team into an area they are interested in
• The ability of those specialists to learn more about their area and pass that onto others is drastically reduced.
• The ability for the team to push their development boundaries will be indirectly reduced as everyone on the team aims for the ‘generalist’ role to fit in

What About Pair Programming?

Now I’m a massive fan of the various agile techniques out there. Pair programming is an excellent mentoring, development and training tool but it won’t allow one developer to fit into the shoes of another. True, they will have a better understanding of the tools, pipeline and systems being developed which will allow them to fill in, but it won’t transfer the amount of background experience the specialist has.

The same goes for code reviews, constant refactoring and feature discussions. It spreads the knowledge which reduces the risk to the project should the specialist not be around when needed, but the core experience and drive that made the specialist who they are simply cannot be replaced by dropping in a new developer.

But Everyone Does A Bit Of Something Every Once In A While?

Of course, sometimes people do need to jump into another developers shoes (illness, staff turn-over, hit by a bus etc.) but this is not the same as expecting a people to be able to fulfil each others roles equally. We can take steps to decrease the impact this will have on the team using the processes mentioned above but it will not allow those specialists to be inter-changed as the project continues development.

We need specialists in any development field because it’s these people that can push their respective fields in directions we might not even be able to imagine. By treating them as interchangeable we might be gaining flexibility to schedule our staff, but we’re losing something far more important and vital to a development team and the products they are creating.

As I said to some one (in 140 character or less of course) when pointed out that people have done this, and even the author of the original post has done it (see the comments)

I’m sure he has done it, I’ve done similar, but it doesn’t mean we did both with the skill of an expert of either.

So I wanted to follow those by covering the different review methods we’ve used in the past and the ways in which each one worked or in a number of cases didn’t.

Just to note that in the previous posts people have commented on ways they’ve carried out reviews. I’m not covering those here as I want to cover methods I have explicit experience of, so please comment on any methods you’ve used at the end of the post.

Paper Based Reviews
One of the simplest methods we used was to take the code you want to review, print it out in whatever format you want (we found an A5 booklet worked best) and pass them around. Usually people would be given a couple of days to review the code before everyone involved would sit around a table and discuss what they had found.

The advantage to this is that it gets people talking and it focuses the discussion within a specific time frame. You’re sat around a table together and it’s a great way to get people participating and interacting with each other.

It also makes it easier to review larger code blocks, especially interacting elements. Often people would pull up the code in an IDE based on what they’d seen in the printed out code, and allowed them to make personal notes before the review.

One of the negative elements we found was this it seemed to force people into thinking they must contribute rather than simply being happy with what is in front of them. Often this resulted in comments which I suspect would not have really been an issue if it was carried out in another way.

People also had a tendency to review the code 30 minutes before the review meeting, which sort of rendered the longer lead in time pointless.

It’s also a drain on resources and time. It’s surprising how long it takes to get code from an IDE into a suitable (and readable) format, especially if all you have is a word processor. And wasting all that paper is something I’d like to avoid.

Smaller, per-change reviews were never reviewed in this way. Getting people in a room to review smaller changes is simply not time efficient.

Forums
One of our most short lived methods was posting code onto an internal forum when the review didn’t justify a full sit-down session. This worked very well in making the code easily visible – even to people outside the review group though that might not be suitable in every case. It also reduced the time it takes to set up and cuts down on waste.

But discussing specific parts of the code simply didn’t work. Referencing line 316 of PlayerCamera.cpp in a post makes it very hard for people to link to the work, and while forums are good for limited threaded discussions, trying to quote and re-quote just becomes hard to track, read and find out what needs to actually change.

Like I mentioned at the start, this was a short lived idea for pretty good reasons.

Automated Code Reviews
By automated I mean using a tool that has been created specifically for code reviews, one which can display code correctly, generate or show diffs and allow people to reference files and lines of code easily.

Of course, when using automated tools you have more scope as to when to do you reviews (do you review individual diffs, whole files or something in-between?) and I’ll spend the rest of this article going over the different methods we’ve used so far.

Blocking Code Reviews: Blocking code reviews are used when you want every piece of code reviewed before it is even committed to the repository. When a developer has made any changes to the code base they fire off a review and the developer cannot check in their work until everyone in the review group has flagged it as shippable. Any changes or requests need to be done before the code review can be submitted and checked into the repository.

This didn’t work. Developers need to get into a work-flow and having to constantly stop and wait for permission to continue or to review someone else’s code stops them in their tracks and is a serious drain on their productivity.

But it could be improved. The blocking edict only focused on work committed to the trunk. Working on an individual branch, where check-ins are free and quick, and then putting those up for review when merging to the trunk would mean developers could continue to work ‘in flow’ before having the merge process include a review step.

Also using a source control system which allows shelving or pushing of changes to a local storage before committing to the trunk would avoid the stop/start nature of this method.

Non-Blocking Code Reviews: The next obvious step. Every check-in needs to be reviewed so the review is generated, sent off to a selected group, but the developer is then free to commit the code with any changes or improvements being done in another commit.

This is beneficial on two counts. The original developer is able to continue working, either on the same area of code or another without it conflicting on any waiting commits, and the reviewer is able to continue their work, leaving any reviews the need to complete for when it’s suitable, maybe performing a group of them in one go.

While this worked well in regards to making sure that everything is reviewed, we found that you needed to monitor the size of the review groups and check that people are not being over-loaded. Filtering your reviews only to come to them at the end of the day with 10+ of them waiting for you can be quite challenging.

Feature Based Reviews: This is generally where we sit now when it comes to diff based code reviews and it puts the responsibility back into the hands of the developer.

If the developer is working on a new feature then they are free to check in their changes with a view to putting the whole feature up for review when it is finished. This greatly reduces the number of reviews and for new code allows the developer to put the whole code into context. It’s very similar to paper based reviews but in a more suitable environment.

But there is one caveat.

It only works if there has been a decent design stage to the feature being written. Having people bring up basic design flaws when the thing it practically finished can be a killer blow but this is showing up the work flow process more than anything else.

This can obviously be extended to large scale changes that someone might be working on. The can check in their small changes, which a review done on the final product, maybe generating the diff based on a collection of commits rather than just one.

Bug fixes are another matter. Once the main implementation is done and people are into a bug fixing stage then these should be reviewed as and when they are being committed (usually in the same manner as the non-blocking reviews). This is especially important when it’s coming up to a milestone and is the way continually developed projects will eventually spend most of their time.

Other Methods
Obviously there are other methods we could try.  Review buddies, review masters (one person assigned responsibility for checking all commits – sometimes this will usually be the lead or programming manager), e-mail based discussions, ‘important changes only’ reviews, and plenty of other variations on the same theme.

We haven’t tried these because the automated reviews have been working very well for us so far.

Review buddies would be the one I’d like to try (alongside, rather than replacing, the automated review process) but I would usually have this done between two developers working on similar parts of the code – though I would hope these developers would be talking to each other already!

Review Groups
One thing I’ve not covered is who does these reviews which can be just as important as what is being reviewed.

Initially we used review groups, which contained 2/3 developers and were mixed quite randomly. Every review would be assigned to a random group and fired off. While this worked well by spreading the work and the knowledge of the code base, it wasn’t producing the best results. Having one person involved in 2 out of 3 reviews for a particular feature, only for someone else to replace them on the last one would produce duplicated, or sometimes conflicting, results. And the new person isn’t as up-to-date with the feature as the original set of reviewers, which can limit the scope of the review and lead to it taking longer.

While this does spread the knowledge throughout the team, it isn’t necessary for everyone to know everything about the code base.

So now reviews are generally more focused, being sent directly to specific people who will have the best input on a particular feature. This does need monitoring to make sure people are not being over-loaded (or over-looked) but that’s a pretty simple management issues. People can also be asked whether they want to be involved in a certain review. Do they have a particular interest in that area, or does it use a new piece of technology that they would like experience with?

So that’s a brief over-view of the code review systems we’ve used over the past few years. We’ve settled on our automated code reviews as they produce the best results in the shortest time and give you much more scope in regards to how and what is reviewed. Choosing the correct piece of software to run this system is an important decision and I’ll cover why we use particular tools to do our reviews in the future.

Title image by fatboyke.  Used under license.

So what do I see as the biggest problems facing introducing and maintaining a code review process?

Aggressive reviewers & victim syndrome
Code reviews are there to improve the code base for the benefit of the team. They are not there to ‘go on the attack’, to rip some-one’s code apart or to make someone feel like they are not good enough.

If people start to fear putting up their code for review then the number of reviews will suddenly start to drop or even worse only the most trivial of code will be submitted.

The opposite of this is the review owner who is too sensitive to any kind of code critique. Any non-positive comment is seen as an attack on them rather than the code and criticism is met with defensive comments rather than seeing it as an opportunity to learn.

Having an over-aggressive reviewer is not that difficult to tackle. Taking them to one side, reviewing their comments and how they approach the reviews will often resolve the issue quite quickly. Most of the time the reviewer won’t even be aware of how their comments are coming across and are not intending to be aggressive at all.

The later is much harder to solve.

Quite often the developer is highly protective of the code and may be more defensive when reviewed by certain developers. It might be possible to start off by restricting who reviews their code or what is reviewed, starting smaller and slowly introducing more of their code to more people over time.

A Lack of Positivity
Remember that there is nothing wrong with the only comment on a review being “That looks great!”.

Lack of Outcomes
People will argue, and people will disagree, especially when it comes to code. “You should do it like this”, “I don’t want to change that because…”. Discussions are great and should always be encouraged, but at some point there needs to be a decision.

You have to know who has the final say – whether this is a manager who is on every review, or a senior responsible for the quality of the code. A simple comment along the lines of “That’s a great suggestion for the future” can put an end to most arguments with most people being happy with the outcome.

Sometimes a short face to face discussion will bring these discussions to a suitable close for everyone involved if the discussion starts to turn into a tit-for-tat point discussion.

Lack of Follow Through
The worry here is that nothing changes as a result of the code review. Obviously not every suggestion or comment needs to be acted on but if developers are seeing review after review being posted and time and again every comment being ignored or not even acknowledged, it can have a devastating effect on buy-in.

This kind of behaviour is both demoralising and infectious.

If developers see their comments being ignored time and time again they will start to do two things. Comment less and react less.

If their comments are being ignored then why should they continue to post more comments? And if people can ignore their comments why should they respond to comments on their reviews?

Sometimes it takes time to respond to a review comment. To avoid situations of reviews just ‘sitting there’ I like to make sure there is a closure plan for all reviews. That way people know that if a review is still open but not responded too, they are still processing the information.

But if reviews are constantly being closed and the suggestions ignored, then this needs to be resolved quickly. Maybe a manager or senior developer re-opening the review to ‘bump’ the comments to make sure they are referenced might solve the problem quickly enough.

Unfocused Reviews
Sometimes people don’t know what to review so every review dawdles around the code and nothing of note ever comes to the surface. In a lot of these cases review after review will seem to focus on coding standards and nothing else.

In situations like these I like to ask people to suggest where people focus their time on the review. This won’t be on every review and it’ll only be certain developers who do this (usually those who know where their weaknesses are).

By suggesting areas for people to focus on it will start to open up both those reviews and ones that don’t have specific suggestions as people start to get more ideas as to what to look for and what might be important.

Unrealistic Expectations
There are no silver bullets for a development team. No process will take you to a communication nirvana, there is no way of working that will stop code rot and nothing you can do to stop some bugs getting through to a possible submission build.

If you start to expect that a code review process will cause every bug to be spotted before the code is submitted or that suddenly every ones skill level will rise to that of the best developer on the team then you’re setting yourself up for disappointment.

But by accepting that the review process won’t solve all your problems but that it will have a generally positive and cumulative effect on the quality of the work being produced those expectations will start to lower but your team will be better off for it.

Any More?
There are not the only reasons why a code review process can falter but from my experience they are some of the main ones. What things have happened to your review processes in the past and, most important of all, what could have avoided it?

In the next post I’m going to describe some of the different ways we’ve reviewed code and the various benefits and pitfalls they had.

Apologies if I take some time to respond to comments to this post. I’ve scheduled this to be posted while I’m out of the country and I won’t have the time or opportunity to respond until I get back. But don’t let that stop you posting! :)

Title image by mroth.  Used with permission.

So why am I such a fan?

Sharing Experience and Knowledge

It might surprise you but this is one of the main reason I love code reviews. People learn best when they are surrounded by people they can learn from and allowing those people to cast an eye over the code you are writing and talking about how something can be improved are invaluable.

Having people suggest different ways to structure, optimise and write code might seem a little late when it’s already written, but it’s something people can take with them into their next task as the grow as developers.

And it works both ways too. Less experienced programmers might have their work discussed and improved but it also allows them to question and push code from more experienced programmers who might be a bit more stuck in their ways than they should be.

Improving Code Quality

This sort of follows on from the above point but as people are learning from each other they are becoming better programmers. They are learning to avoid common mistakes and as a result writing more stable and easier to read code.

Programmers will also start to spend a little bit more time checking over the code they write. No one wants to put up a review only for someone to find a rookie mistake in the first 5 seconds, so small errors that in the past would have been checked in only to cause havoc later get found before the code is even commited.

Finding Bugs

It might just be me, but nearly every time I’ve asked someone to put up something for the first time, at least one bug is always found. It might be a misplaced define, a + instead of an – or any number of things but there’s always one. And the time it took to post and complete that review will be much less than the time it takes for that bug to be found, a bug report written, the author to read the bug, find the code in question, do some tests and then finally fix it.

But I generally find that the number of bugs found during code reviews starts to drop off over time. This isn’t because people are getting lazy and not casting such a critical eye over the code but because the quality of the code is getting higher and higher.

As a result all those bugs that might have otherwise found their way into the code base aren’t even being added, let alone fixed early.

Sharing Domain Knowledge

The more people know about a particular system the better. Illness, team moves and resignations will always be part of the working environment and it means that the key team member you are relying on could end up not being available when they are needed the most. And if you need someone else to dive in there and fix a couple of issues that have been found by QA, how long do you think that will take?

If a system has been constantly reviewed as it was developed there will be a larger number of people who will feel comfortable getting stuck into that part of the code base.

Coding Standards

This is probably one of the main reasons people hate code reviews as they don’t want every conversation to turn into a discussion about where the bracket should go or what pre-fix that variable should have on it.

How much of a problem this is depends on how strict your coding standards are and how by the book your code needs to be.

So it certainly has it’s place in a code review.

Developers new to the team can be given documentation about coding standards, but it is one of those things that tend to easily slip, and there is no reason why this shouldn’t be picked up in a review. Of course you need to be careful that your reviews don’t start to get swallowed up in standards talk and the important elements like code quality start to suffer.

And There Will Be More

These are not the only benefits of code reviews but to me they are the stand outs that make code reviews a vital part of the development process. Let me know if you have any more!

In the next post I’ll look at some of the main reasons why code reviews can fail and why some teams will not get the benefits that they should be doing.

Title image by Sebastian Bergmann.  Used under license.

]]> http://www.altdevblogaday.com/2011/09/13/the-benefits-of-code-review/feed/ 0 http://www.altdevblogaday.com/2011/08/14/the-evils-of-unity-builds/ http://www.altdevblogaday.com/2011/08/14/the-evils-of-unity-builds/#comments Sun, 14 Aug 2011 11:05:25 +0000 Lee Winder http://altdevblogaday.com/?p=14142 Unity builds. I don’t like them.

Of all the tools at your disposal to make a build faster, this is the worst. And it’s not just the “hey let’s #include .cpp files” weirdness, but the way that it can make a well structure, modular code base become worse than spaghetti junction at rush hour.

Read more on The Evils of Unity Builds…

Unity builds. I don’t like them.

Of all the tools at your disposal to make a build faster, this is the worst. And it’s not just the “hey let’s #include .cpp files” weirdness, but the way that it can make a well structure, modular code base become worse than spaghetti junction at rush hour.

And the worse thing is that it’s not the fault of the programmer, especially when something as exceptional as Visual Assist can start helping you create non-modular code because of the way Unity Builds collect the code you write.

 

What Is a Unity Build?
Unity Builds have nothing to do with the excellent Unity Tool Set. I’ll just clear that up right off the bat.

These Unity Builds are a way of reducing build over-head (specifically opening and closing files and reducing link times by reducing the number of object files generated) and as such are used to drastically speed up build times. I say drastically because they are usually used after a code base has starting generating build times that drastically cut down on a programmer’s productivity. This might be 10 minutes, 30 minutes or even hours.

Because Unity Builds give you a quick gain, it’s seen as a pretty magical solution, and when a full build is reduced from 30 to 10 minutes you can see why. But the caveat in that statement? Full builds.

I won’t go through how to generate Unity Builds here as the blog post The Magic of Unity Builds does a great job so have a look there. It’s interesting that I’m linking to a blog post that is in the total opposite direction that I’m coming from, but unless you know both sides of a tale, you don’t know the tale.

So without any more delay, why don’t I like them?

 

Say Goodbye to Local Properties
Well written code is modular, and modular code relies on a few assumptions. One of those being that a single translation unit (a cpp file and any associated include files) is isolated from other translation units, so (unless you extern the contents) anything defined within a cpp file is not available outside, and as a consequence you can have local objects being called the same things without conflict.

Take for example the following simple code (easily expanded to a real world example I’m sure)

1 2 3 4 5

// In VectorTest.cpp namespace { const uint StandardTestCount = 10; }

1 2 3 4 5

// In ListTest.cpp namespace { const uint StandardTestCount = 3; }

In a normal compilation environment these variables are local to the file they are working in, and this is a good thing. Why should the vector tests file care what is defined within another test file?

But if you have a Unity Build with the following then you’re going to have issues…

1 2 3 4

#include "VectorTest.cpp" #include "ListTest.cpp" #include "ArrayTest.cpp" #include "QueueTest.cpp"

Specifically errors relating to the variable ::StandardTestCount already being defined. So we now have to be aware of what is defined throughout the entire project and resolve any issues, and inevitable this will end up with all variables being pre-appended with (in this example) VectorTest_ and ListTest_ etc.

Or worse still, because Intellisense (or whatever tool you might be using) informs you that StandardTestCount is defined, you start using that in your local .cpp file when it’s actually defined in a completely independent file somewhere else.

I don’t want to refer to the ‘Magic’ post to much, as it’s a good article, but there is a statement in there referring directly to this. Specifically the following

“Technically you shouldn’t have this problem if you’re writing “proper” code”

This is wrong. “Proper” code is well structured and well maintained, meaning it’s modular and only refers to what it needs to refer to. In a lot of cases you will have variables with the same name, functions with the same name and other elements that you naturally write as you go along. That’s why the anonymous namespace (or the C style ‘static’) is so useful, and so useless if you switch to Unity Builds.

 

Using Is (even more) Lethal
Never use the ‘using’  keyword in a header file. It’s a standard statement and one that stands up easily. The reasons behind this are pretty clear (declaring ‘using’ in a header file forces all code including the header file to use that statement – in effect wiping out a namespace).

But using it in a .cpp file isn’t a crime. In fact it’s remarkably useful, even within the whole files scope. As a developer, I should know what I’m using in a local file, what elements I’m bringing into the file and what would/could conflict. Some people might not agree that ‘using’ at the top of a cpp file is a good idea at all, and I see their point, but on a whole file scope, it can be massively useful and as it’s localised it’s not going to pollute anything other than the file it’s in.

But in Unity Builds, using (that is not scoped within a function or type etc.) is lethal. Declaring ‘using ‘ in a cpp file makes it available throughout the code base (or more confusingly only in those files that are included after the one declaring it). Suddenly, using is everywhere. And your namespaces are now useless.

 

Every Change is a Rebuild
This is one of my biggest problems with Unity Builds. In well structured projects (more on that below) changing the content of a cpp file (or even a local header file) shouldn’t require everything to be rebuilt. Every. Single. Time. Yes, the unity build rebuilds are faster than doing a full build without a unity file, but doing even a fast rebuild every time will build up. Quickly.

When-ever I change a .cpp file, all I need to build is that .cpp file. Granted, link times are a tad long because it still has to re-link the other object files, but it takes a second to compile that one file. On average (I took count today) when I changed a header file it compiled on average 5 .cpp files. And it (again on average) took about 5 seconds to build.

Very rarely should I be required to re-build the entire project, and most of the time I’m don’t. And that saves me a lot of time. Every single day.

 

Multiple Configurations
This is mainly a bugbear of mine rather than a direct issue with Unity Builds, but I see it in nearly every Unity Build project I use. The main project is the Unity Build project, but there is another project that is built and maintained that doesn’t use Unity Builds. Now there is a point here – by having an additional ‘normal’ project you are forcing the modularity that can collapse with Unity Builds to be checked (usually a Continuous Build server will be building this as well as the Unity Build every time).

But we have problems with this.

Firstly, the non-unity build is only ever being built on the CB server. So any problems are going to break the build, and it will break if people are not using it day-to-day. Secondly you now have multiple projects to maintain. Not too much of a problem if you have an automated project generation step (see below) but it is still another project to maintain.

People may occasionally have to use the non-unity configurations, especially if they are getting an unknown error on the CB server. So now they are left working on a configuration that is uncared for and probably builds so slowly and erratically that they are probably losing all the time they saved from those quick unity builds they have been doing all day.

 

But What About My Build Times Then?
Well structured software builds quickly (or as quickly as they can). But what is a well structured project when it comes to build times?

• Sensible file inclusion – Only include the files you need and try as best you can to limit them to .cpp files. This means the compiler only needs to bring in what’s necessary and when you change one of these header files only those things that directly rely on it will change. If you find yourself having to include files that constantly change into a large number of cpp files, then I’d wager that a refactoring of the large header file would be in order. You should be building only a small number of cpp files when you change the content of a header file.
• Forward Declarations – Where possible use forward declarations rather than including the header file in your class or function declaration. Annoyingly you cannot forward declare enums (on standard compliant compliers anyway) which sometimes throws this out of the window. But by not including the files until use, you’re cutting down on the amount of code being included and the number of files being opened.
• Pre-compiled Headers – Use Pre-compiled Headers (PCH). Using PCH’s is the one (built into the IDE) feature that will cause your build times to plummet, especially if you are being sensible with them (such as including files that don’t change – SDK’s and common, fixed headers for example). Using pre-compiled headers across multiple platforms can be a pain, but it is possible to unify them and get a massive boost. I’ll cover these a little bit more below.
• Library Files – Modular code usually leads towards easily extracting common code into a collection of libs. Reducing the amount of code in a project (and as a result how much you need to build) can speed up your productivity quickly.
• Parallel Building – If you’re IDE supports it (and it might do and you just don’t know) and you have a multi-core machine, turn on parallel building of files. Building individual files at the same time is obviously going to cut down on your compile times no matter how quick they are at the moment.
• Get a Better PC – It goes without saying (but I will anyway) doing this will speed everything up.

Pre-Compiled Headers
Pre-compiled headers are one of the best ways to get a massive boost to your compile times. So how fast are my build times compared to that of a Unity Build version when using PCH’s and taking into account the other build improvements suggestions?

As stated above the ‘average’ build time of a minimal build throughout the day was around 5 seconds. On a Unity Build it was a full build every time and was around 2 minutes on the slowest platform.

Changes to ‘core’ files, which are included by a higher than average number of files, resulted in builds of around 30 seconds on around 20 files. Again on a Unity Build this would have been around 2 minutes regardless.

Full rebuild (which I did twice today) was around 3 minutes. Granted a whole minute slower than a Unity Build but I did it twice rather than every single time.

Pre-compiled headers are not a silver bullet solution. Nothing is. And because of this here are issues that you do need to be aware of

• Compiler Support – Some compilers out there simply do not support PCH’s. On a daily basis I use between 4 or 5 different compilers and while I’ve never encountered one that doesn’t, they are out there. This can shoot my argument in the foot, but it is a rare problem and one most people won’t encounter.
• PCH Quirks – While I use a number of compilers that do support PCH’s, every single one of them has a slightly different way of building them and slightly different requirements before they can be used. This doesn’t affect the code that is written but does affect the content of your project, especially if you want to make them seem as consistent as possible.
• Over-Inclusion – Because your PCH usually includes common files and files that rarely change, it does mean that some files are being brought into the project in a compilation unit that wouldn’t otherwise be required

Unity builds are a solution for the wrong problem. Long compile times are not caused by not using Unity Builds, they are the result of having badly structured and badly referenced code. Fix that (and have better code to boot) and you’ll be able to use ‘proper’ build tools without resorting to a quick fix.

 

Making Unity Builds Better
I don’t want to just leave it at that, because no matter how much I argue, Unity Builds will always exist and people will always use them (after all it’s quicker to rescue a broken down build by making it a Unity Build than doing it the hard way). So what can people do to actually make Unity Builds a bit more desirable and less of a problematic fix?

• Automate Adding Files – A lot of teams have auto project generation code already (XML files, template projects etc.) so it’s important that you automate adding files to the project, otherwise people will forget to remove the file from the build step and they will forget to add it to the unity build file.
• Multiple Unity Files – Running a Unity Build doesn’t require you to have one unity file with every cpp file inside it. You can have multiple build files (usually per module or per component) which means at least some of the translation unit leaking is limited to each module rather than the whole program.
• Additional Project – No, this isn’t a contradiction from the above “Multiple Configurations” comment above. In this situation you will have a project that contains the cpp and header files so you can reference them but this project isn’t built. Instead, you have the ‘proper’ project simply contain the unity file(s). This isn’t something I’ve personally tried, but it does get around the issues of adding files if you don’t have an automated step.
• Use PCH Headers – Again? Well that’s because they are so powerful. I’ve worked on a couple of projects where the unity files have been split up into sub-unity files as suggested above. But they just starts to bring in slower build because of the multiple includes that are happening in every single Unity file group. Adding PCH’s to the mix will speed that up even more.

I originally wrote this article on my blog way back in 2009. Since then, my opinions really haven’t changed and I’ve seen some rather poor uses of Unity Builds that I’ll unfortunately never forget.

Title image by DiggPirate.  Used with permission.

]]>

But STL Allocators have a bad reputation and it’s for a good reason. They are complex, hard to understand and have some interesting behaviour that seems designed to confuse. As a result, I needed to look at how we were going to provide a custom allocation system that was both easy to use and simple to understand without restricting what people could do with them.

A Bit of History
A while back Bitsquid published a post entitled Custom Memory Allocation in C++. This excellent post covered how the BitSquid engine used an allocator model to perform memory allocation throughout their entire engine.

But the FTL requires a slightly different approach so I won’t be treading over already covered ground here.

FTL allocators are well hidden inside containers, the only control you have is specifying the allocator type which means it’s use, and how and when objects are allocated and created is completely fixed with only the allocator itself being able to effect the outcome. Because of this the allocator behaviour needs to be as customisable as possible without requiring any changes the container itself.

When the FTL was original started, it was quite small scale and only used by a couple of developers, so allocators were not a priority. The flexibility wasn’t needed so in-container malloc and free allowed us to concentrate on container creation but obviously this wasn’t going to last.

The follow just describes the various approaches we took, why we dismissed them and what we eventually ended up with.

The Initial Approach
Allocators should have a minimal over-head. What we didn’t want to do was increase the size of every container dramatically when we rolled out customisable allocators. As a result, we initially used an approach used by a couple of vendors and defined the allocator specification using only static members – removing the need for an internal allocator object.

1
2
3
4
5
6
7
8
9
10

// Completely made up code showing the use of a static based allocator
template <typename T, typename Alloc>
class vector
{
  void push_back(void)
  {
    void* new_memory = Alloc::allocate( sizeof(T) );
    T* new_object = new(new_memory) T;
  }
};

I knew this would limit the flexibility of the allocators, but it had minimal over-head (especially using default template parameters) and wouldn’t effect those containers already being used. And besides, programmers are a creative bunch and I wanted to see what people did with this before resigning it to the scrap heap.

But while programmers were able to work around the main limitation of not having allocator state per container, they were having to jump through hoops to get the behaviour they wanted. Which made it less likely that other programmers would feel confident enough writing their own allocators and their ability to really customise their behaviour was too limited.

So we ended up adding allocator state on a per container basis, making it much easier for developers to do what they wanted though it did add at least 4 bytes per container. But I felt that flexibility and simplicity were much more important.

1
2
3
4
5
6
7
8
9
10
11
12

// Completely made up code showing the use of an instanced allocator
template <typename T, typename Alloc>
class vector
{
  Alloc m_allocator;
 
  void push_back(void)
  {
    void* new_memory = m_allocator.allocate( sizeof(T) );
    T* new_object = new(new_memory) T;
  }
};

Allocator Specification
The allocator specification is complicated. While I’m sure there are good reasons for some of the decisions I wanted ours it to be much simpler. So removing the type information (since our allocators work on raw memory and nothing else), removing the rebind(!) and exception handling (which we don’t use) we ended up with the following.

1
2
3
4
5
6
7
8

typedef ftl::pair<void*, size_t> allocated_memory;
 
class Alloc
{
public:
  allocated_memory allocate(const size_t alloc_size);
  void deallocate(void* ptr);
};

And for the basic allocator, that’s it, it doesn’t require anything else to work, but it does have one interesting aspect.

1

typedef ftl::pair<void*, size_t>  allocated_memory;

As we can have all sorts of allocator types, what it allocates might not be exactly what you asked for. If it can’t allocate all the memory you requested, that’s fine, it simply returns allocated_memory(nullptr, 0) but it can also return more than was requested (for example, fixed sized block allocators will do this). This return type simply allows the allocator to return not only the allocated memory, but also how much was allocated, which allows calling objects to take advantage of this additional memory if possible.

Most of the time this isn’t queried and only what was asked for is given, but it offers an additional level of information which might allow better memory usage in some containers.

So a container will most likely end up with something like the following when adding and removing elements.

1
2
3
4
5
6
7
8

// Creating a new object in a container
allocated_memory new_memory = m_allocator.allocate( sizeof(T) );
if (new_memory.first)
  T* new_object = new(new_memory.first) T;
 
// Losing the object now we’re done with it
old_object->~T();
m_allocator.deallocate(old_object);

This is fine and gives us a very simple entry point for allocators. But by forcing the use of placement new and the destructor call in the container itself, we are limiting what an allocator can do. While the allocators are required to return raw memory, that doesn’t mean that it has to internally. Some allocators might pre-create the objects before returning them so creation is front loaded but the forced placement new could mean we’re over-riding an object that has already been created.

Construction Functions
As a result, we want developers to be able to not only over-ride the memory allocation, but also the object creation. 99% of allocators won’t need to do this so we don’t want to add additional requirements to the allocator so instead we can create non-member, non-friend functions, specialised on the allocator, which will do the creation for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

template <typename TConstruct, typename TAlloc>
TConstruct* construct(TAlloc& allocator);
 
template <typename TConstruct, typename TAlloc>
TConstruct* construct(TAlloc& allocator, const TConstruct& obj);
 
template <typename TConstruct, typename TAlloc>
TConstruct* construct(void* allocated_mem);
 
template <typename TConstruct, typename TAlloc>
TConstruct* construct(void* allocated_mem, const TConstruct& obj);
 
template <typename TConstruct, typename TAlloc>
void destroy(TConstruct* ptr);
 
template <typename TConstruct, typename TAlloc>
void destroy(TConstruct* ptr, TAlloc& allocator);

So our point of allocation/creation now becomes something much simpler and much more powerful.

1
2
3
4
5

// Creating a new object in a container
T* new_object = alloc::construct<T>(m_alloc);
 
// Lose our existing object and return it back to the allocator
alloc::destroy(old_object, m_alloc);

The default version of construct performs the allocation and placement new within the construct function, but should the allocator need something more (or less), simply over-loading the function on the allocator type allows complete control over both memory allocation and object creation. The same goes for the destroy function and the automatic call of the objects destructor.

No Base Allocator
One thing I didn’t add to the allocators was a base allocator interface. The allocators are created within the container, with their type defined at the point of the containers declaration. The addition of a base interface (and the associated virtual functions) would have increase the size over-head of the allocator which is something I wanted to avoid and something I thought didn’t add enough to warrant the overhead. I was less worried about the overhead of calling a virtual function as that would be insignificant compared to the overhead of everything else what was going on.

Conclusion
So by introducing an allocator model that is much simpler (only 2 required functions) with more extensible customisation should an allocator should need it, developers have complete control over all the memory allocation and importantly the object creation itself. Due to it’s initial simplicity, developers have no problem creating new allocators that improve both memory and container use in a project and can start to experiment with better and more efficient allocators quickly.

Title image by Squeezy.  Used with permission.

But if it’s not done with care, a CI process, while still providing some useful information, will stop being an important part of a development teams tool set.

The main problems usually stem from a single CI step taking to long. For example, it might take hours to build all the game assets or it might take 40 minutes to build a single configuration. You might have additional build steps (like copying files to a network drive) which can take quite a while if you’re dealing with gigabytes of data.

As soon as a CI step takes to long, you lose the main benefit – the fast turn around of information.

For example, when we first start a project, our simple CI process will consist of the following steps

• Detect modification
• Build code
• Build game assets
• Copy to network drive
• E-mail developers (who’s mailed depends on success or failure)

This is fine as a new project is tiny and the whole process doesn’t even take 5 minutes. And we need to build the whole thing constantly as we’re adding so much content the artists and designers need to be on the bleeding edge of what the programmers are creating.

But after a month (or probably less) this stops being suitable. A whole build might start taking 20 minutes then 30, an hour and then two hours, and if we leave it as is, the programmers are not getting the benefit of continuous turnaround and the designers spend ages waiting for a new build.

So what can we do about it?

The first thing is to look at what the CI process is doing, and exactly what we want to get out of it.

• Continuous Build – We need the process to constantly compile the source, all configs, all platforms. This is so we can detect any compile errors quickly without having to build everything manually.
• ‘Designer’ Builds – Creating an executable the designers, artists, animators etc. can get with the latest code changes. Ideally one they can request as required and one that is built as quickly as possible.
• Full builds – A complete build of the game including executable and all in-game assets along with anything else needed to run the game.
• QA Builds – QA could use the full build if needed, but this is an additional step which packages the build as it would be submitted allowing a better QA pass (DVD emulation, submission content etc.).

From my point of view, those are the four main things I want to get out of a CI process that having a single build step won’t give us. You might have other requirements and I’d certainly be interested in hearing what those are.

So what can we do to try and improve the initial process and still get what we want out of our CI machine?

Continuous Build
The first step is easy. We want a Continuous Build process with nothing to integrate, nothing to deploy and nothing to copy. This can be much quicker if we alter our repository modification checks to only monitor source code folders and not the entire repository.

For example if our repository contains scripts, configuration files or (shudder) game assets and executables we shouldn’t kick off a build if these changes as the CB results won’t be any different from last time.

We might also want to reduce the configs we’re interested in building (usually a debug and master build, the profiling builds might be skipped for speed reasons and due to them rarely being using). If we have a decent machine we might get the individual platforms (X360, PS3) to build in parallel as there will be no conflict between the temp files they generate – or even stick them on separate machines if we have the capacity.

The process only ever needs to notify on failure as no one is going to be using this build, it’s a sanity check pure and simple.

So already we have a much faster turn around time between check-in and ‘all clear’. In the past I’ve managed to reduce this from 60 minutes to 5.

‘Designer’ Builds
Initially it might be tempting to use the results of the ‘Continuous Build’ as the aim of this step is to provide the designers and artists with a new executable to take advatage of any new features (very) recently added.

This might be the right idea at the start, when the CB process is taking less than 5 minutes, but that doesn’t last, so we need a faster, more iterative, process to make sure our non-programmers are not hanging around waiting for the latest builds.

Most of the time, designers will use a ‘release’ build (‘release’ being a bit of a misnomer – it’s not releasable in any way, but it has just enough debug information to make it useful but still run at a ‘releaseable’ frame rate). So we only need to concentrate on a single configuration which means we can drastically cut down the length of time between when a modification is detected and a new usable build is generated.

As this is the fastest CI step we have and can often have the most people dependent on it, it’s the first one to run when a modification is detected.

In our case we don’t e-mail people when a new designer build is enabled. This is being built many times in an hour, and people will just end up sending the mails directly to the trash (I know I’ve done that on particularly spammy CI set ups). Simply allowing them to check the build status and update when it’s green works well enough.

Full builds
Developers generate a lot of content for games. Even small games can balloon in size depending on the scope and quality of the final product. As a result, we need a full integration build for a number of reasons

• It would take every member of the team far to long to rebuild all the assets themselves
• When a build gets out of sync with the assets, developers need a quick way to get everything back on track
• When testing the build it needs to have been built on an independent build machine

A full build could be brute force (just builds everything every time) or smart (it concurrently builds executables and assets on multiple machines). This really depends on how long a full build would take. Less than an hour and I personally stick with a brute force approach but any longer and a more intelligent build step would be needed.

Full builds always e-mail the entire team, since it’s rare they happen. This allows people to get latest as they become available (usually at the start of the day) without having to check the status of the build.

QA Builds
The QA build is a special build. It doesn’t rebuild any assets or executables and is automatically kicked off when the daily build has finished successfully. This step packages the build up as it would be presented as part of a final submission along with any submission assets that would be required.

But why not just use the full build as the QA build to save time?

Simply put when testing a build it’s vital that we test under the same situation that our final submission will be tested under. Making sure we run under DVD emulation and use the same assets the manufacturer will use is an important part of the process. Having our CI machine generate these builds for us makes sure we’re doing this from the very start.

Requesting Builds
In every case we give all members of the development team the ability to request any build. If the ‘Designer Build’ is to far out of sync, they might need a full build to get them back on their feet. An asset change might alter the executable but not trigger a ‘Designer Build’ so a designer might need to trigger one manually.

In our case using CCTray allows us to do this very easily, so a new build can be requested by anyone (including QA) at any point of the day without any input from a programmer, allowing them to concentrate on making the game rather than just enabling others.

Technically anyone in the company could request and get a new build (very useful for getting demo builds together without getting the team involved) but I’ve not seen that happen yet.

What I Haven’t Covered
One major thing I’ve not covered here: self testing builds. This can range from simple unit tests running after every build to a scripted run through of the game after every full integration. The scope of this will very much depend on the size of the game and the time you have available to add them. Since this is a big topic in it’s own right, I’ve left that for another time.

Conclusion
So by simply reviewing what we actually want to get out of the Continuous Integration process we’re able to streamline it down to be much faster and much more useful. Complete integrations still happen (and happen when needed) but the common information needed by the team is generated quickly and allows teams to get short turn-around from the CI machine throughout the day.

I’d be very interested to know what other uses people are getting from their CI processes and how they are still making sure the speed and quick turnaround is happening all the time.

Title image by highgroove.  Used with permission.

]]> http://www.altdevblogaday.com/2011/07/15/is-it-green-yet-improving-our-ci-process/feed/ 0