Typically on small programs linking is measured in mere seconds (often less) and is largely unnoticeable. However on larger projects with millions of lines of code, heavy template usage and optimizations such as inline functions and weak references (declspec selectany) your symbol count goes off the charts. Combo this with the probable use of libraries to compartmentalize your code and the likelihood that you’re compiling more than 1 target at a time (dll’s, tools, runtime). The linker is likely doing a significant amount of the work for you AFTER compiling your single file. As an example for me within our current code @ Bungie a single file change in the code I usually work on can cost as much as 10mins in linking (many targets). If i opt to build a single target (say the runtime) then this is reduced to ~2 mins. This is without any linker optimization such as Link Time Code Generation which is known to heavily increase Link time.

In my experience I’d say most of my changes (and most of the changes my colleagues make) before doing a compile involve a single file. More often than not its simpler than that: a single line. So the iteration is something like

• make change
• hit compile
• 1-2 seconds of compile
• 1-2 mins to link single runtime target.
• load game
• test
• repeat

Incremental Linking

Incremental linking sets up an ilk file. The first time you link with /INCREMENTAL enabled  it will generate a database of the symbols within the Target executable and the location that supplied them within object files of said executable. This file is used in subsequent links to cross reference a changed object file (the single file you changed) with final executable allowing the linker to “patch” the symbols that are different with the new version. The Patch is achieved using a code thunk so the resultant code can be slower. The recommendation is to do a full rebuild before doing any performance testing.

Incremental linking is on for link.exe by default. New Projects generated with MSVC will explicitly disable it for Release however. I would assume that most if not all of your simple programs are using it in debug. Sadly the most common “working” case is the one that needs it least; small programs have short link times. In my experience the more complex your project the less likely incremental linking is working for you. I have spoken with many engineers over the years about incremental linking and almost all believe it to be a flawed feature of MSVC. Sometimes it works, sometimes it doesn’t. When it doesn’t work it actually increases link times and therefore, for many engineers it isn’t worth wrangling with.

I experienced the same, sometimes it works and its fantastic. Sometimes it simply doesn’t… most of the time it tells you a reason.

If your project uses any linker options that alter the symbols the linker uses or explicitly lists the order in which those symbols are used then internally the linker will disable incremental linking (the MSDN docs explicitly mention this).

• /ORDER (forcing comdat order): http://msdn.microsoft.com/en-us/library/00kh39zz(v=vs.71).aspx
• /OPT:REF (dead stripping): http://msdn.microsoft.com/en-us/library/bxwfs976(v=vs.71).aspx
• /OPT:ICF (comdat folding): http://msdn.microsoft.com/en-us/library/bxwfs976(v=vs.71).aspx

What is a (.lib) library file.

What does that mean for Incremental Linking with Lib Files

If you’re simply linking against stable Libs, incremental linking works fine. However if the code you’re editing is within a Lib itself, it doesn’t work.  There seems to be no output from link.exe saying its not going to incrementally link it simply doesn’t do it. You can see this for yourself by using the /time+ (another undocumented option) option on the linker command line which describes the various passes the linker does. Experiment yourself with changing a file within a lib and a file within the target exe.

I believe this all stems from a simple fact: Libs are supposed to be stable from your builds perspective.

I have always used them for compartmentalization; a simple method to contain functionality within a single file (and therefore reference) rather than having each project have to contain a multitude of files it doesn’t really need to know about. I believe this is a very common usage pattern. Common enough that every team i’ve worked with used them in this way. I believe this is the main reason many engineers believe incremental linking is simply a broken feature.

But i want my cake

Given the information above the solution “seems” simple. Simple enough that VS now provides an option for it based on work by Joshua Jensen who discusses it at length here.

Within a standard VS2010 project browse to:

<MyProject>/Properties/Configuration Properties/Linker/General/Use Library Dependency Inputs

This option switches a “library” included from within your solution, to use the object files directly rather than the library file. This switches the library to my own previous understanding of lib files; all objects will be linked into the target explicitly as if they were Target source files. The lib file itself isn’t used at all.

One issue however, I mentioned earlier using a lib provides a feature i hadn’t previously understood “effectively object file level dead-stripping”. If your lib file contains symbols that have a dependency themselves, for instance D3d (very common) then using the above option will require you to explicitly list D3D as a dependency in your Target project setup regardless of whether the Target required the functionality itself. This could mean relatively few changes for you or it could mean you have to split up your code to remove dependencies that only exist for specific Target executables but not for all.

Deadstripping (OPT:REF) has to be off so this switch will in turn enlarge your Target executables.

Winning

Once Incremental Linking is active and working, you’ve removed the use of Lib files directly for the code you’re editing and stopped doing deadstripping for all local builds you’ll see a massive improvement in overall iteration times. For one of our tools projects the link time reduced from ~5mins to ~10seconds and now we’re aware of the pitfalls & reasons for linker performance (using /VERBOSE /time+) we’re reducing that even further. For our main runtime the link reduced from 2mins to 2-3 seconds, a single file change including deployment was reduced from 145seconds to ~10 seconds. This seems inconsequential when considered standalone however for engineers iterating on functionality within said runtime this is huge and equates to an estimated 30-45 mins per day per engineer.

*Only the penitent man shall cross

Once you have resolved all the issues that can stop incremental linking from working you are a short step from the Holy Grail. Something I personally have never had working before but will be actively fighting for in the near fugure. Consider the original process reformatted based on our above work:

• make change
• hit compile
• 1-2 seconds of compile
• 1-2 seconds to link runtime target.   <= incremental linking is great
• load game
• test

Notes & References

Managed C++ cannot link incrementally, if you’re combing managed & native consider splitting the native into a dll to achieve incremental linking. Obviously this is a lot more work and would likely deserve a cost benefit analysis beforehand.

MSDN Incremental Linking Help: http://msdn.microsoft.com/en-us/library/4khtbfyf(v=VS.100).aspx

Edit and Continue: http://msdn.microsoft.com/en-us/library/esaeyddf.aspx

Forum post re: incremental Linking: http://bytes.com/topic/net/answers/281196-incremental-linking-multiple-projects

Link Time Code Generation: http://msdn.microsoft.com/en-us/magazine/cc301698.aspx

Series

Part 1  Part 2  Part 3 Part 4

Once you get to this stage it is often very difficult to figure out what is going on. Depending on the age of your code base you can often be looking at an include tree 4-5 levels deep (sometimes far worse) and involving 1000′s of files. How do you retroactively figure out where to direct your attention? Hopefully I can help you there.

I should note that the basic method here was written by one of Bungies Senior Engineers Jon Cable who kindly let me take over the code and gave permission to discuss it here.

The intent of the algorithm is simple, to rank all header files based on a heuristic involving various aspects of their use. Add to this the ability to inspect the entire include tree of a specific file and you have a powerful tool indeed.

The initial version was brute force. Call the function from your source directory root and it:

1. hunted down all the cpp/h files and adds records for them.
2. It then recursively processed each cpp generating an include tree and counts per header file based the number of files they included (children) and the number of files that include them (parents).
3. Post processes the entire file list building counts for files included BY their children (Descendants) and number of headers that include THIS file (Ascendents).
4. Jon’s initial code used the heuristic
• (ancestors * (descendants + 1) )

This took a little while to run (4-5mins) but generated a top ten list of “bad” headers. Accompanying the top 10 were txt files allowing the user to search for a specific string in each txt DB and find the children, ascendants, parents and descendants thus enabling the optimization of that header. The information was good and it was used to optimize on several occasions during previous projects. It was however somewhat cumbersome to setup & use and often interpretation of the data was difficult so in the end very few engineers used it.

We are inherently visual beasts and I have a penchant for visualization of complex problems. Text is easy to read but often it is very difficult to grok a pattern or understand a tree without visualizing it. I also wanted this tool to be useful as a regular optimization pass to all engineers. I had to make it simple & fast.

Firstly I took the code and replaced the search for files with the dependency database the compiler already uses (we use a Jam Plus variant so this was trivial, I believe its possible using msbuild too tho), this also allowed for building of the include tree & counts directly from said source. The result of this was the information required was available in 3-4 seconds (11,000 file project)

The second pass was to output the “bad” headers information in a graph form. I used some old code to output both a graphviz document & a DGML document of the query i was doing. While doing this i also added the ability to simply request the header structure for an arbitrary file. A screenshot of a DGML example is shown here.

An Example Analysis from Our Engine

Obviously the above is too small to see the filenames on purpose :)

The colorization in the above graph is intended to offer some context such as an SDK include, a windows header, an internal library etc. I prefer the DGML form because it is an active document allowing searching and hotlinking to code as well as graph analysis and reduction of focus. I output the graphviz form also due to DGML being a feature of VS2010 Ultimate only (~$10k per copy) so few engineers have it. The Graphviz form is a simple image and thus not AS powerful but still really useful.

Once this was up and running i wrote a set of VS Macros to provide direct access. From any file in the project an engineer is now able to single click and see badness rank and/or the include structure of their files.

Future work on this will involve augmenting the heuristic to include characteristics of the code within headers such as template usage or inline expansion both of which contribute significantly to compile times.

Hopefully others recreate the tool (it took only a few hours) and use it to help their projects & compile times.

enjoy.

Series

Part 1  Part 2  Part 3 Part 4

At this point I began looking at some of our code and playing around with a few longer duration cpp files to try and get a “feel” for what was taking so long. The compiler isn’t very useful in this regard so at this point we’re reliant on empirical data. One particular file didn’t seem to do very much (main hook for unit testing “core” systems) but was taking 9 seconds to compile so i began decimating it, stubbed out all its functions and it still took ~8 seconds to compile. Because this was a core systems test file it was including a lot of headers for toolbox like functionality: templated containers, specialized math classes, job framework. Many of those systems do not contain much in terms of state full code, they simply provide a framework for the client to include and template functionality into their systems. There’s that word again, template, we’ll come back to that in a later post.

I see Dead People

Upon examining some of the includes for this single long build cpp file i realized that the development of the unit tests system was playing a part. When it was originally written the code to perform the actual system level testing resided in this cpp file. As the system developed and further modules added the file was split into a set of hooks into other cpp files and a framework added such that effectively the unit tests file literally only needed externs to those test blocks.

I pared them down, of the 50+ includes only around 10 were required. Compile time down to <1second…
So this innocuous file had 40+ includes it didn’t need, many of them heavy with inline functions, templates, all manner of other includes. The pre-processed file for it was huge (surprisingly so). Removing the headers was laborious and time intensive: remove header, compile all targets, if they all compile & link fine then we’re good and the header can be removed permanently. This process took around 40 minutes for this single file and we had (at the time) ~7000 source files active… this would take far too long.
But wait.. we have these machines, they do these things for us when we tell them to…

Do I know Kung-Fu?

At this point it was obvious that a plugin or macro could do this work for us. What i found however was that the help for VS Macros with VS2010 was somewhat lacking, searching the internet often gets more results in Word/Excel than any other application and very little seems to exist in terms of automation beyond trivial operations such as text insertion. The basics of what i wanted to do is simple

for each file in the project
    for each include in the file
        if (file compiles without include)
            remove include
    next
next 

I spent a weekend playing around with various options and hunting down a lot of the automation options available with VBA / DTE. It did turn out that there is a ton of info but it is very tough to search for it. In the end i found that the most helpful method was to RECORD macros yourself and dissect them into useful snippets for your work.

Some helpers for those who wish to do this themselves.

To build the startup project use

Function BuildStartupProject_f() 

    Dim sb As SolutionBuild = DTE.Solution.SolutionBuild
    Dim projName As String = sb.StartupProjects(0)
    Dim Config As String = sb.ActiveConfiguration.Name
    Dim Platform As String = sb.ActiveConfiguration.PlatformName 

    DTE.ExecuteCommand("View.Output")
    sb.BuildProject(Config + "|" + Platform, projName, True)
    BuildStartupProject_f = sb.LastBuildInfo.Equals(0) 

End Function

setup a find target

DTE.ExecuteCommand("Edit.Find")
Dim findResult As EnvDTE.vsFindResult
DTE.Find.FindWhat = "^:b*\#include"
DTE.Find.PatternSyntax = vsFindPatternSyntax.vsFindPatternSyntaxRegExpr
DTE.Find.MatchCase = False
DTE.Find.Target = vsFindTarget.vsFindTargetCurrentDocument
DTE.Find.MatchWholeWord = False
DTE.Find.Backwards = False
DTE.Find.MatchInHiddenText = False
DTE.Find.Action = vsFindAction.vsFindActionFind
CType(DTE.Find, EnvDTE80.Find2).WaitForFindToComplete = True
findResult = DTE.Find.Execute() 

create an undo context

DTE.UndoContext.Open("IncludeReplace")
' Do stuff here '
if you need to undo use DTE.UndoContext.SetAborted()

to iterate over files in a project use

http://msdn.microsoft.com/en-us/library/aa301042(v=vs.71).aspx

Apologies for not being able to supply a working set of macros but they fall under my employers contract so i can only provide snippets that are available online and guidelines. Feel free to ask any questions over email tho and i will answer what i can within the confines of said contract.

The greatest harm can result from the best intentions

The Brute force mechanism roughly described above does not come without its issues. I found that upon running it on some of our low level code I created a problem that can only be combated by looking through every change made. The main cause of this was cpp files where the entire system was compiled out (removed) based on a global define that was found IN the first include. This is a common pattern for inspection/debug systems and created 3-4 problems on my first run. To combat this i made the overall macro algorithm build all target executables. Without these inspection systems the final exe fails to link and our tests can continue. This does however send our iteration time down into the 20mins per file area. As an automated system however this didn’t really deter and avoids the false positive.

I know you! You’re dead! We killed you

Running this set of Macros on 1 project (our lowest level systems/hardware layer) garnered pretty good results. 30% of our headers were removed in 1 swoop using a purely brute force method. It took 9 hours in total and saved a respectable 10% off of the build of said project on PC/360.

This technique ultimately rests on the ability to cut off an entire tree of headers from the cpp, as such it is at the mercy of the programmers willingness and ability to setup headers in a form that reduces code dependencies whilst providing the functionality the client requires. Next i will be looking into include hierarchy and how we can analyze the cost of a tree with a mind towards culling off branches and reducing the implicit dependencies we create.

Series

Part 1  Part 2  Part 3 Part 4

It has always been surprising to me that many programmers don’t delve into the reasons why certain processes take so long but we’ll spend weeks/months optimizing the code itself to extract that last 1ms (guilty as charged). What became obvious however once i talked to the few that had tried, was that the tools available to aid this task are somewhat dire and often have issues with the scale of our projects. So I delved deeper and this is what i found.

We have 51 targets (libraries, exe’s, dlls), spread across 11,000+ files ranging from 5-10mins to 100,000+ lines. A full clean build of all our code across the targets we’re interested in can hit ~3hours on a single machine.

These aren’t the droids you’re looking for

The assumption from 9 out of 10 programmers i talked to is that file IO is the root cause of a slow build. Ergo disabling Virus Checking, getting a faster Hard Drive, switching to SSD or using a RAM drive must be the way we speed it up. This sounded fantastically sensible so i investigated speeding up these devices. I chose our mildest target which showed a build time around 7mins +- 5seconds. My laptop already has both an SSD & SATA installed as well as having 12Gb RAM, so i spent some time running tests… I won’t bore you with the details but needless to say i confirmed in each case that the circumstances were well setup and spent a full day running builds of various forms in various locations moving around temporary files, object files, source locations, splitting across drives for temp vs target, disabling virus checking… it was a very busy day for my poor laptop. Sadly the net effect to all these changes was… ZERO. Build time remained steady at m7+-10s for ALL but 1 test which was RAM Drive for source on a fresh reboot vs SSD for source fresh reboot; this is an obvious case as the ram drive would offset the windows cache. Even then the difference was nominal (7m30 for SSD, 6m59s for RD)

It was at this point I realized i’d made the cardinal mistake when optimizing; I must find the problem before attempting a solution.

Open your mind

At this years gamefest Bruce Dawson gave a great talk on a tool i’ve attempted (and failed) to use previously: XPerf. Thankfully his first 5mins was spent lamenting just HOW difficult the tool is to use when you are not a part of Microsoft and can’t simply email the dudes who would know the correct settings. His talk was “How Valve Makes games better using XPerf” and recounted what he did, how he did it and ultimately what he thought other studios should know about it and how we can improve upon it. The net gain was a much greater understanding of the tool AND BATCH FILES!!!. Magical batch files that setup all the internal components required by XPerf in order to provide decent information.

I used this sorcery to profile our code build, 10 mins later I had 13Gb of data and my mind is now open. File IO is the problem, but not in the assumed way. Yes there is a lot of reading and writing of data, but looking at the profile info it seems that most time is taken up with WAITING for a write while another write is progress. Most notably (~90% of cases) this was with the mspdbsrv process which seem to be attempting to service writes from many location to the same file. My theory is that the more concurrent builds (targets build concurrently at the project level for VS2010) the more the requests for write to the PDB back up and eventually the parent process has to stall which stalls the entire pipeline. In my test the Mspdbsrv process is occupying approximately 40% of the CPU time the build uses with ~64 CL’s in flight.

Now i’m obviously a novice when using XPerf so i could easily be reading the information incorrectly.

I’ll be back

Using this information one of our engineers spent some time looking into splitting up the PDB’s themselves. The setup he wrote essentially creates a pdb for every “N” files and his cursory tests shows a significant performance increase in build from doing this. It has not yet been officially timed. In later parts of this series i’ll detail the effects of this change (and others).

Where we’re going, we don’t need roads

There are obviously more brute force methods of optimizing the latency between a code change and a built target however i believe that starting as you mean to go on and actually learning the core issues is worthwhile to any process like this. The subjects i’ll be looking into are:

• hierarchical header optimization
• simple header removal
• pro-active dead stripping
• incredibuild XGE

please feel free to share any information you feel might help as we push forward on this project.

Series

Part 1  Part 2  Part 3 Part 4

This article will assume basic knowledge of floating point numbers

So we’ve all used them, likely for all and sundry within our games/tools so heres a question to ask yourself… what does the code below print out for the “test” variable?

{
    float small_value= 1.0f;
    float smaller_value= 1 / 100000000.0f;
    float test= small_value + smaller_value;
    printf("{%g,%g} => {0x%08x,0x%08x}\n", small_value, test, *(int*)&small_value, *(int*)&test);
}

Almost all programmers I spoke to assumed the value of test to be 1.00000001f and of course mathematically it should be, however it is not, the print out would show.

{1,1} => {0x3f800000, 0x3f800000}

Single precision floating point simply cannot represent the accuracy the result requires and as such the result is kept at 1.0f.

Now if you are mathematically minded this will likely poke at your OCD gene and “force” you into using doubles. This is a perfectly palatable solution in many situations where one does not require speed however double still suffers the same fate if you double the number of zeros in the denominator (1 / 10000000000000000.0f;).

For those of us who work in games math is important, but its far less important than simulation determinism and getting things done expediently whilst executing within performance budgets. This forces us to make decisions that might otherwise be seen as somewhat mathematically incorrect. The above is one such circumstance. In most games applications we cannot afford to switch math to use doubles and as such the inherent limitations of single precision floating point math are deemed acceptable, even becoming “normal”. I’ve been working with them so long that it is now natural to apply these limitations in all circumstances as par for the course.

As was mentioned in the link i posted above comparing of floating point numbers using == is only sensible in a purely deterministic (read: constant) form. If math is being used to generate the values to be compared then great care has to be taken; great care is rarely taken. The general rule in most studios is simple: don’t compare floats using == on pain of death or public embarrassment. The usual method is to apply some form of epsilon to the comparison

{
    float a= 1.0f;
    float b= 1.0005f;
    static const float epsilon= 0.0001f;

    float temp= fabs(b - a);

    if(temp > epsilon)
    {
        // not the same
    }
}

Within games this is one method used to enforce determinism; apply a decent epsilon and most math “behaves” (this doesn’t mean its accurate mind). One complication however is that epsilon is not always obvious nor can it always be constant, especially within helper classes such as Vector Math.

A good epsilon is usually dependent upon the data you’re representing. If you are dealing in world coordinates for instance where 1.0f = 1m then

0.01    = 1cm
0.001   = 1mm
0.0001  = 100µm
0.00001 = 10µm

Many of the games i’ve worked on use 1mm as their positional epsilon (0.001f) and 10µm for directional/rotational epsilon (0.00001f). For Positional Epsilon this provides an effective range of approximately +/- 10000.001 and for rotational epsilon +/- 1000.0001f.

The general rule of thumb i use is, 8 decimal places between your highest and lowest accuracy requirements. If you feel you need a larger range than this but still want accuracy then consider using a reference frame; 1 value to represent low accuracy high values (say kilometers) and another to represent high accuracy low values (meters down to µm)

Now one area that seems to catch a lot of people out (myself included on many occasions) is Infinity & NaN Rules… so here is a handy table borrowed from here, his looks prettier.

On the subject of error handling consider the standard method of normalizing a 3d vector.

vector3 normalize_vector(const vector3 & vec)
{
    float length= vector_length(vec);
    vector3 result=vec;

    if(length > 0.0f)
    {
        float reciprocal= 1.0f / length;

        result.x *= reciprocal;
        result.y *= reciprocal;
        result.z *= reciprocal;
    }

    return result;
}

We want to avoid introducing a problem by way of an INF=>NAN in the returned data or throwing an exception so a branch is inserted. This effectively removes the divide by zero problem however at great cost; on many platforms the branch is an instruction cache flush resulting in significant performance issues. The problem is there really isn’t another way to achieve the same avoidance mathematically.

There is however a method of avoiding it if you rely upon floating point math. We’ve established that a large value remains the same when a small value is added, we’ve also discussed that the effective range of floating point values for use in games is limited both for determinism and to avoid issues with the math itself. Combining the 2 provides a rather elegant method of avoiding the divide by zero issue under normalization and many other well known situations.

const float very_small_float= 1.0e-037f;
vector3 normalize_vector_2(const vector3 & vec)
{
    float length= very_small_float + vector_length(vec);

    float reciprocal= 1.0f / length;

    vector3 result;

    result.x = vec.x * reciprocal;
    result.y = vec.y * reciprocal;
    result.z = vec.z * reciprocal;

    return result;
}

Due to limiting the allowed range of vector3 components and understanding that addition of the very small value to any value larger than 1.0e-29 has zero effect we have effectively removed the possibility of receiving INF and thus NAN as the result. (The Denormal length is still possible however much less likely)

There is a plethora of information out there on floating point both practical and theoretical however the above is my attempt to represent those cases not covered or not well highlighted and hopefully make people think about their implementations more in terms of the specific requirements and less in terms of “floating point numbers handle everything”

Personally I’ve always been interested in what was going on under the hood so this suited me just fine. I always dug into the details and I almost always end up programming as close to the bone ON the hardware (or OS) as I possibly can both to eek out as much performance as possible AND to satisfy my own hunger for knowledge.

Combine the two and what you get today is someone who enjoys spending 5 days making that single function 20x faster, who revels in reducing the memory footprint of the primary data structure by 1 byte per element across the entire program whilst simultaneously writing a pre-caching system to avoid the special case issues…. who… well you get the picture… I’m an OCD level sport optimizing geek. To such a degree that I keep a notepad by my bed to take notes when I wake up from a “bug fix” dream or a “eureka optimization” doze…yes i’m that sick.

Over the last decade I’ve been involved in the hiring process at many studios and in more recent years I’ve noticed a pattern. Knowledge of what is generally considered “low-level” programming is waning. Many programmers know enough to get through a C# or C++ test, but don’t understand something as basic (and important) as the behavior of memory or god forbid a cache. They don’t seem to grasp that one must understand the native environment you’re working in before going ahead and writing a program to run within it. The intricacies of floating point vs fixed point math are completely lost on them as the term “fixed point” brings about a blank stare; floating point numbers are best right?. I once mentioned bit shifting to an experienced engineer of 10 years and was devastated by the complete lack of basic understanding.

It depresses me that so much of what I consider to be essential is simply not being taught anymore. I’m not talking about assembly language per se; even those of us who used to spend hours writing assembly now more often opt to use intrinsics built into compilers to avoid the stress and complication. What I’m talking about is simply the understanding of WHAT is happening when someone does i++ and not ++i, why one might opt to stripe a memory copy/set in certain circumstances.

So here goes… a list of things I believe all console programmers (and recommend to all programmers as good reading) should fully understand with links to educate where possible. (feel free to suggest more/better links)

• Floating Point Numbers
• They are very useful but often used in situations where they simply don’t suit the solution the programmer is attempting to write. The following links should provide some background and info on where they are not so useful, what the pitfalls are and sometimes even how to avoid them.
• http://www.cprogramming.com/tutorial.html#fptutorial
• http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/
• http://www.codeproject.com/KB/recipes/float_point.aspx
• http://drdobbs.com/184402741?pgno=4
• http://users.tkk.fi/jhi/infnan.html
• Fixed Point Numbers
• Fixed point math is mildly old school but it is VERY useful both to understand its makeup and to use. Sadly because it is considered old school many of the online sources are out of date.
• http://x86asm.net/articles/fixed-point-arithmetic-and-tricks/
• http://gameprogrammer.com/4-fixed.html
• Processor Cache Behavior / Memory
• http://www.akkadia.org/drepper/cpumemory.pdf
• http://en.wikipedia.org/wiki/CPU_cache
• http://igoro.com/archive/gallery-of-processor-cache-effects/
• Bit Shifting
• http://www.cprogramming.com/tutorial/bitwise_operators.html
• useful hacks (use carefully)
• http://graphics.stanford.edu/~seander/bithacks.html
• http://stackoverflow.com/questions/539836/emulating-variable-bit-shift-using-only-constant-shifts
• http://guru.multimedia.cx/avoiding-branchesifconditionals/
• Branch Prediction
• This may be lower level that people think they need to go… but they’d be wrong. Understanding how the hardware you’re programming for treats branches can affect performance to a HUGE degree… far more than most programmers may appreciate re: death by a thousand cuts
• http://cellperformance.beyond3d.com/articles/2006/04/background-on-branching.html
• http://igoro.com/archive/fast-and-slow-if-statements-branch-prediction-in-modern-processors/
• http://www.k8gu.com/ece.umn.edu/documents/classes/ece362-branch-prediction.pdf
• http://www.cs.ucr.edu/~gupta/teaching/203A-09/My6.pdf
• Sorting
• This isn’t really low-level but something i consider “basic” and its an area where many programmers are simply lacking in understanding. Do yourself a favor and play around with this link, read the links it sends you to for each algorithm and try to grasp when each might be used, the properties as described and the next time you need to sort something… consult it.
• http://www.sorting-algorithms.com/
• another good link with sub links: http://corte.si//posts/code/visualisingsorting/index.html
• a funny one (but still strangely useful), the bubble sort dance: http://www.youtube.com/watch?v=lyZQPjUT5B4

So there you go, I’ve likely missed some aspects that should make the list but if you can grasp the above then you’re more likely to get noticed at interviews and you’ll certainly be a better programmer for it.

 

Edit 8/8/2011 – It seems I didn’t do a very good job at explaining the core audience for the article, I apologize for that. The programmers I would like to see learning and absorbing these “low level” details are those who would see themselves working in console games. The HW is fixed for long periods of the time and resources become scarce usually within 1 game cycle; understanding how to better utilize resources becomes a key part of the job.

While i believe that most programmers would benefit from understanding more about the medium they work in it is certainly not required

Hopefully you enjoyed the article in the spirit it was intended.

Andy Firth

en·gi·neer
noun /ˌenjəˈni(ə)r/ 
A person who designs, builds, or maintains engines, machines, or public works

ar·chi·tect
noun /ˈärkiˌtekt/ 
((computer science) the structure and organization of a computer’s hardware or system software) “the architecture of a computer’s system software”

This basically translates to the same thing all engineers do:

• Analyze the problem
• Formulate a possible set of solutions
• Analyze the solutions
• Decide what to implement
• Implement
• Test/bug fix
• Profile/sanitize
• Repeat until satisfied

however with one important difference. My role involves the entire program, how its parts interact and how they are designed, what their dependencies are and how they affect the final output and the performance of that output. Combine this with data parallel infrastructure, task parallel objects, multiple platforms, myriad hardware limitations and a large programming team … under many circumstances one might imagine something akin to

and you often wouldn’t be too far from the truth on a daily basis. Long term however we (the team) have a plan and a set of goals. Over time we re-assess based on those we hit and those we don’t, new requirements and ultimately how the game is progressing and where it needs to go. The role I play in this is simply to ensure that under the stress and strain of day to day development, we’re still aiming in the right direction as a whole. That decisions take into account as much of the big picture as possible and weigh that against the immediate requirements of the current goal. There are several people in similar roles at Bungie and our interactions provide a simple but effective method of applying “Checks & Balances” to the progress we make; we each bring our own flavor to the table.

Day to day this involves tasks such as

• Advising/Teaching on how to handle concurrency
• Long term interface design
• Short term prototyping/bug fix hacks
• Discussing/Advising on future platforms
• Optimizing programmer iteration, debugging & general workflow
• Managing external teams
• Auditioning Middleware
• designing/writing/debugging infrastructure systems

Technically i’m a member of Bungies “Infrastructure” team. This means that if its something “unexciting”, “behind the scenes” or seemingly doesn’t affect the final game at all… we handle it. This involves systems like

• Memory Allocation
• File System
• Network Transport
• Crash Handling, Minidumps
• Debugger Plugins
• Multi-threading Infrastructure & Architecture
• Asset import/baking
• Math Library
• Schematization/Reflection
• Audio Engine
• Container Classes
• Profiler Infrastructure
• Compiler Configuration
• Low & High level Optimization
• Build systems
• Flux Capacitor Maintenance
• – you still reading?

as i said… the stuff most most programmers find tiresome and boring. Our team love this stuff and we’re good at it.

So there you have it, a much better idea of what i do @ Bungie and hopefully a guide to those in school who might want to progress towards a similar role (or avoid it).

So here’s the situation, you want to create objects “a” through “g” in various files who register themselves with a manager into a list on construction, a relatively common setup and relatively simple within a single compilation module; for VC++ define them from bottom to top manager first and it will work. However across compilation modules things are not so simple and more often than not the manager will construct itself at a very inopportune time; likely after some objects have registered themselves and before all objects have.

Without simply knowing that the manager will initialize first there really isn’t a “Good” solution tho there are several lazy initialization techniques.

So here’s where things get “fun” and by “fun” i mean, people look at the code and get confused looks on their face.

Put the following code into a cpp file on its own and add it to your project of choice.

#pragma init_seg( ".CRT$XCB" )

class c_blog_first_class_construction
{
public:
    c_blog_first_class_construction()
    {
        printf("first class construction\n");
    }

    ~c_blog_first_class_construction() {};
};

static c_blog_first_class_construction blog_first_class_construction;

the constructor for c_blog_first_class_construction will be called before any other constructor in your code (assuming you have nothing else like this in there).

NOTE: Expect C4075: initializers put in unrecognized initialization area, disable it if you feel the need.

the key to the code is

#pragma init_seg(...)

for more info on what this does see kb104248

The crux of it however is we are naming this compilation unit with a specific identifier, when the linker reads various .CRT groups, it combines them in one section and orders them alphabetically. This means that the user-defined global initializers (which the Visual C++ compiler puts in .CRT$XCU) will always come after CRT$XCA and before .CRT$XCZ.

So we are depending linker to do the right thing and insert our section into the CRT sorted sections (verified in VS2010 as of 7/6/2011 for x86,x64,x360). It works and its very useful.

Now if you want to get really fancy… make the constructor for c_blog_first_class_construction call a function that initializes your “Pre construction” requirements engine wide (usually memory management, assert systems).

For other platforms (GCC/SNC are the only others i use) there are other options such as “init_priority” which is a per instance attribute. These can be used to achieve the same result but on a more granular level.

It should be noted that this is not a method that one should use all over a codebase, personally i only ever use it on one compilation module and it is always VERY well commented.

enjoy.

Here I will discuss how to create a VS2010 plugin using C# (plugins also work for VS2008 however not using C#).

The Development Tools Environment framework was setup around a decade ago to allow Visual Studio extensibility. I would recommend that any programmers read through what is possible using DTE however i’m not going to invest much time into that here; I will simply use one aspect of it to achieve our goal: global symbol access.

VS2010 has a great Wizard to help us here, Select File->New->Project then on the left hand side go to Other Project Types->Extensibility->Visual Studio Addin, click next and then choose option (default) Create an Add-in using Visual C#, leave application host as default, then give your addin a recognizable name. Set your addin to load when host application starts. The rest is default.

This provides us with a very simple addin with a LOT of C# goodness and very little code. I should note that when i wrote this project it was my first foray into C# and thus my knowledge in this area is shady at best, i had lots of help from experts local to me in setting this up.

If you read through the code you will find some relatively basic functionality. Your new addin has interfaces to intercept callbacks for various VS2010 events, OnConnection is the only default interface i used during which i installed new event handlers.

So lets extend the default a little:

1. Add a member to the class of type “DebuggerEvents”
2. instantiate it to “_applicationObject.Events.DebuggerEvents”
3. setup handles for:
1. OnEnterBreakMode
2. OnEnterRunMode
3. OnEnterDesignMode
4. VS2010 should auto-complete and auto generate these for you but the basics should look like :

   _debugger_events.OnEnterBreakMode += new _dispDebuggerEvents_OnEnterBreakModeEventHandler(DebuggerEvents_OnEnterBreakMode);
   _debugger_events.OnEnterRunMode += new _dispDebuggerEvents_OnEnterRunModeEventHandler(_debugger_events_OnEnterRunMode);
   _debugger_events.OnEnterDesignMode += new _dispDebuggerEvents_OnEnterDesignModeEventHandler(_debugger_events_OnEnterDesignMode);

    

5. For now ignore all but OnEnterBreakMode, this will be called whenever we enter any type of breakmode (stepping, breakpoints, exceptions etc). At this point we can do something quite nifty… any expression you can type into a watch window, can be evaluated here for example

   Debugger debugger = _application_object.Debugger;
   Expression exp = debugger.GetExpression("&g_my_global_foo,d");
   
   if (exp.IsValidValue)
   {
       Int64 value = Convert.ToInt64(exp.Value.ToString(), 10);
       System.Diagnostics.Debug.WriteLine(exp.Value.ToString());
   
       // do other fancy things here
   }

6. At this point its really up to you to choose how you communicate this information to any external process. Personally my first simple solution was old school environment variable. This works perfectly well for a limited project, simply push your symbol address to the environment variable and let any other process (EEAddin for instance) read that environment variable and use it. Should you choose to publish the root address of your global systems for instance one might conceivably achieve full global memory addressing on your target.
7. The OnEnterRunMode hook should be used to “create” any resources you need on events “Attach” and “Launch”
8. The OnEnterDesignMode hook should be used to “destroy” those resources for events “Detach, EndProgram & Stop Debugging”

Debugging the debugger addin isn’t difficult but is not obvious. Given the setup (Client Project) using (Debugger Addin) which launches Watch Window DLLS (EEAddin) the debugging method i used is:

1. Open the Debugger Addin project and build/run. This should open up an instance of VS2010 with your Addin loaded (check under Tools->Addin Manager
2. Within this second instance of VS2010 open up your Client Project, Build and run.
3. At this point breakpoints within the Debugger Addin are possible and you should be able to trap the incoming events.

Notes:

• using environment variables will work for very simple projects however for production code i would recommend using process specific memory mapped files or another less hacky method
• VS2008 can be supported however this has to be done as a C++/ATL project and requires a LOT of boilerplate code. Jason Weiler discusses this here and played a large part in enlightening me to this process.
• As previous with EEAddin, this framework is not very forgiving and will require you to write defensively in all aspects. C# does handle a lot of this for you however the C++/ATL version will not. If you are writing an addin framework for a large group of programmers i would suggest significant boilerplate on all code.
• the DTE framework is VERY powerful, anyone using it should read through the docs thoroughly to appreciate everything it can do and share what you do if you can :D

consider the example

// blogtest.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <stdio.h>
#include <intrin.h>

enum e_union_test_case
{
    k_unset,
    k_type_string,
    k_type_int,
    k_type_float,
};

struct u_test_level1
{
    int m_type;
    union u_data
    {
        const char *m_string;
        int m_count;
        float m_value;
    }m_data;

    void set(const char *string)
    {
        m_type= k_type_string;
        m_data.m_string= string;
    }

    void set(int int_value)
    {
        m_type= k_type_int;
        m_data.m_count= int_value;
    }

    void set(float float_value)
    {
        m_type= k_type_float;
        m_data.m_value= float_value;
    }

    void print(void)
    {
        if (m_type == k_unset)
        {
            printf("unset\n");
        }

        if (m_type == k_type_string)
        {
            printf("%s \n",m_data.m_string);
        }
        if (m_type == k_type_int)
        {
            printf("%d \n",m_data.m_count);
        }
        if (m_type == k_type_float)
        {
            printf("%f \n",m_data.m_value);
        }
    }
};

struct s_test_level2
{
    static const int k_num_elements= 32;

    u_test_level1 m_elements[k_num_elements];
};

int _tmain(int argc, _TCHAR* argv[])
{
    s_test_level2 test;

    memset(&test,0,sizeof(test));

    test.m_elements[0].set("my foo");
    test.m_elements[1].set(7);
    test.m_elements[2].set(3.14159265358f);    // everyone loves Pi

    return 0;
}

similar to before setup a basic console project and drop in this code. Compile and run to the breakpoint and drop “test” into your watch window opening up the m_elements member… you should see something like.

not very readable at all… the print function shows what we “might” do to display these elements by using “m_type” to change the formatting options…

EEAddin is an option here, (Expression Evaluation Addin). This allows the autoexp.dat [AutoExpand] section to call into a dll for the preview display string. The dll can output any string it desires into the provided char buffer (obeying the limits of course).

you can find an example EEAddin within the VS2010 install, assuming you have the samples extracted (most installs will have a zip file) you will find the solution at

C:\Program Files (x86)\Microsoft Visual Studio 10.0\Samples\1033\VC2010Samples\C++\Debugging\EEaddin

NOTE: the example does not work out of the box, you must edit “ADDIN_API” and make it

#define ADDIN_API __declspec(dllexport)  __stdcall

then move the HRESULT return to the opposite side of the ADDIN_API on use. Optionally you could achieve the same results by altering the project compiler options.
now paste the code

enum e_union_test_case
{
    k_unset,
    k_type_string,
    k_type_int,
    k_type_float,
};

struct u_test_level1
{
    int m_type;
    union u_data
    {
        const char *m_string;
        int m_count;
        float m_value;
    }m_data;

    void print(char *result_buffer, int result_size)
    {
        if (m_type == k_unset)
        {
            sprintf_s(result_buffer, result_size,"unset\n");
        }
        if (m_type == k_type_string)
        {
            sprintf_s(result_buffer, result_size,"str:\n");
        }
        if (m_type == k_type_int)
        {
            sprintf_s(result_buffer, result_size,"int:%d \n",m_data.m_count);
        }
        if (m_type == k_type_float)
        {
            sprintf_s(result_buffer, result_size,"flt:%f \n",m_data.m_value);
        }

    }
};

HRESULT ADDIN_API AddIn_blogtest( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    DWORD nGot;

    u_test_level1 example;

    // read file time from debuggee memory space
    if (pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(u_test_level1 ), &example, &nGot) != S_OK)
        return E_FAIL;
    if (nGot != sizeof(u_test_level1))
        return E_FAIL;

    

    example.print(pResult, max);

    return S_OK;
}

Into the existing “timeaddin.cpp” file and add the export to both the header (timeaddin.h) and the def file (eeaddin.def).

Compile this and it will generate EEaddin.dll, copy this dll to your VS2010 IDE folder (C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE).

Add the line

u_test_level1=$ADDIN(eeaddin.dll,AddIn_blogtest)

to your autoexp.dat right below the previous example, below the [AutoExpand] tag.

debug your application… your new display should be

each element now displays using the m_type correctly. We have an issue however, the string is not immediately available. This is due to the EEAddin working in a different memory space than the executable it is accessing. You’ll note that the EEaddin DLL reads memory from the application manually using the supplied address AND by dint of the autoexp.dat line we know the size of the type we’re representing. We must therefore pull any pointed at string data over manually in order to display it.

the new code is

enum e_union_test_case
{
    k_unset,
    k_type_string,
    k_type_int,
    k_type_float,
};

struct u_test_level1
{
    int m_type;
    union u_data
    {
        const char *m_string;
        int m_count;
        float m_value;
    }m_data;

    void print(char *result_buffer, int result_size)
    {
        if (m_type == k_unset)
        {
            sprintf_s(result_buffer, result_size,"unset\n");
        }
        if (m_type == k_type_string)
        {
            sprintf_s(result_buffer, result_size,"str:%s\n",m_data.m_string);
        }
        if (m_type == k_type_int)
        {
            sprintf_s(result_buffer, result_size,"int:%d \n",m_data.m_count);
        }
        if (m_type == k_type_float)
        {
            sprintf_s(result_buffer, result_size,"flt:%f \n",m_data.m_value);
        }

    }
};

HRESULT ADDIN_API AddIn_blogtest( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    DWORD nGot;

    u_test_level1 example;

    // read file time from debuggee memory space
    if (pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(u_test_level1 ), &example, &nGot) != S_OK)
        return E_FAIL;
    if (nGot != sizeof(u_test_level1))
        return E_FAIL;

    static const int local_buffer_size= 1024;
    char local_buffer[local_buffer_size];

    if (example.m_type == k_type_string && example.m_data.m_string != 0)
    {
        // pull over the string to our local buffer
        pHelper->ReadDebuggeeMemoryEx(pHelper, (DWORDLONG)example.m_data.m_string, local_buffer_size, local_buffer, &nGot);

        example.m_data.m_string= local_buffer;
    }

    example.print(pResult, max);

    return S_OK;
}

this code uses the pointer within the original “example” and pulls over an arbitrary length data stream (1kb) relying upon the string being null terminated. It then points our local copy of example.m_data.m_string to that local buffer before calling our new print(…) function which will now output the string as before. Our new display is:

which is exactly what we need.

Notes:

• EEAddin is not forgiving
• if you do something heinous within an EEAddin entrypoint it WILL take out the devenv that loaded it. Within Addins you MUST program defensively at all points, assume you will be sent bad data, assume your strings will not be null terminated, assume that the pointers you need to follow are bad and take you into no mans land. Verify everything. You’ll note that i did not do this in my example but that is merely for the purposes of keeping the code samples short and sweet.
• Debugging the EEAddin
• load up the EEAddin project, Attach to the instance of devenv (debug it) that your Main project runs in, put breakpoints into your Addin entrypoint then hit debug on your application. Each time your Watch Window updates your EEAddin dll is loaded/called/unloaded so expect a lot of calls in the array case.
• Global Symbols are not accessible
• Accessing client memory requires the physical address of said client memory, if this address isn’t accessible from the element you’re debugging then you have no way to get to that address. An example of this limitation would be a handle into a global manager. The client code can call the manager to get the object, the debugger can only display the value of the handle, EEAddin can only access the internals OF the handle. It does not have knowledge of the manager and thus it cannot access the object itself.
• Not all Endianess was made equal
• if your target is not the same endianess as PC (little) such as xbox 360 then you must switch the endianess of data before reading it.
• Alignment & Pointer size
• you will have to manually handle both alignment differences (target <=> client) and pointer size differences. In my support thus far i have chosen 2 pathways. Some structures i have made cross platform entirely by supplying a construct that is 64bit and aware of its pointer size. For other types where this isn’t possible i have used an element of the next topic to disclose to the DLL the size of our pointer.

Setup a simple windows console project and use the following code:

struct s_test_level1
{
    int m_count;
    char* m_string;
    float m_value;

    void print(void)
    {
        if (m_string)
        {
            printf("%s %d %f\n",m_string, m_count, m_value);
        }
    }
};

static const int k_num_elements= 32;
struct s_test_level2
{
    s_test_level1 m_elements[k_num_elements];
};

int _tmain(int argc, _TCHAR* argv[])
{
    s_test_level2 test;

    memset(&test,0,sizeof(test));

    test.m_elements[0].m_string= "my foo";
    test.m_elements[0].m_count= strlen(test.m_elements[0].m_string);
    test.m_elements[0].m_value=3.14159265358f; // everyone loves Pi

    test.m_elements[0].print();

    return 0; // <<= breakpoint here
}

now run the program putting a breakpoint on indicated line. Open up a watch window and drop test into it opening up the m_elements array, you should see something like

This is a really simple but even now we see some details we don’t always need… lets reformat it.

open up the file

VS2010 PC: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Packages\Debugger\autoexp.dat
VS2008 PC: C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\Packages\Debugger\autoexp.dat
Xbox360: C:\Program Files (x86)\Microsoft Xbox 360 SDK\bin\win32\autoexp.dat for 360

search for “[AutoExpand]” and add the line

s_test_level1=<m_count,d>, <m_string,s>, <m_value,f>f

save that file and re run the test program…. you should now see

a much easier to read version of the same data.

All the original information is still available should you need it but this new format is much easier to quickly consume.

This form of watch window help is somewhat limited however, you can only interpret existing data, remove superfluous detail, do some VERY rudimentary math and generally make things cleaner and easier to consume; see the next post for MORE.

All of the help required to use autoexp.dat is within the file itself. Note that the “[Visualizer]” section is currently not available for xbox 360 targets but is VERY powerful for other targets.

enjoy and feel free to ask questions.