RPC

What exactly is a remote procedure call system? Simply put, it’s a function call from one process to another. The processes do not have to be running on the same machine or share the same architecture. To help understand, first consider a local procedure call in C/C++. The function arguments are pushed on to the stack, then the program branches to the address of the function being called. When the called function returns the result is stored on the stack for the calling function to access. This works because the caller, callee and data live in the same address space and agree on calling conventions. In a remote procedure call system, the caller, callee and data do not share an address space and thus can not share data by memory address. In order to make a remote procedure call the parameters must be marshalled for transmission over the network and packed into a message for the remote system. The return value(s) from an RPC are returned in a similar way. Still with me? Don’t worry, I will not spend too much time in the plumbing for this article.

The first step in building a RPC system is to pick the data format that messages will be exchanged in. Since we are talking about the web, naturally, I chose the Java Script Object Notation, better known as JSON, as the data format for my RPC system. JSON is a simple, human readable data format. It supports numbers, strings, booleans, arrays and key value maps. Here is an example:

1
2
3
4
5

{
  "type" : "command"
  "command" : "Memory.Stats"
  "id" : "50"
}

Another benefit of JSON is that web browsers natively support it. Your browser can convert almost any Javascript data structure into JSON with a single function call. You can also go the other way, from JSON to a native Javascript data structure just as easily.

Now that the data interchange format has been settled, we need to come up with the RPC message framing. That is, the mandatory portion of the RPC message. Each RPC is a JSON map with two required keys, the message type and the message serial number.

1
2
3
4

{
  “type” : “”
  “id” : “”
}

The type field is used to indicate the type of message. My system supports three types, “command”, “result” and “report”. I will explain these shortly. But first, the id field is used to connect related messages together. For example, this command has an id of “4”:

1

{ “type” : “command”, “id” : 4”, … }

The result of the command sent back from the engine also has an id of “4” allowing to easily connect calls with their return values.

Getting back to the type field:

• “command” : Commands initiate some sort of action or state change.
• “result” : Result of a command. Linked by the id field. Commands are not required to reply with a result.
• “report” : A regular, repeating message from a subscription. Clients can subscribe to a data stream and set an interval between reports.

The other, non-mandatory, fields in a message are type specific.

Examples

Some example commands that my system supports:

• “Echo” – Replies with a copy of the “message” field.
• “Subscribe” – Subscribes the caller the “reporter” field.
• “Unsubscribe” – Unsubscribes the caller from the “reporter” field.
• “Memory.Stats” – Replies with the state of all registered allocators.
• “Config.Set” – Sets “variable_name” to “variable_value”.
• “Config.Delete” – Removes “variable_name” from the configuration variable set.
• “Config.Get” – Returns the value of “variable_name”.
• “Config.Dump” – Returns all registered configuration variables.
• “Object.Create” – Creates a game object
• “Object.Set” – Sets “property_name” to “value”. “render_model” is an example property.
• “Message” – Sends a message to another connected client.

I have two reporters:

• “Memory” – Regularly reports the state of all registered allocators.
• “Config” – Sends configuration variable change notifications.

Here is a screenshot of a live graph of memory allocators:

Quake Style Console

What developer interface would be complete without a quake style console triggered by the ~ key? For this, I used JQuery Terminal. It is a great library which implements a command terminal. Adding custom commands to the terminal window is really simple. I have also made it possible for the user to construct messages by hand from within the terminal, making the terminal window very powerful. The following screenshot shows a user sending a custom Echo message by using the “dcc” terminal command. It also shows the response from the server.

C++ implementation notes

Connection Management

I started with the explicit goal of supporting multiple connected clients. My previous article discussed the importance of decoupling the code that waits for a connection over TCP from the code that manages a WebSocket connection. Because of that decoupling, supporting multiple connections was practically free.

Commanders

Support for commands is added by implementing the commander interface:

1
2
3
4
5
6
7
8
9

class doCommandCenterCommanderInterface {
public:
  doCommandCenterCommanderInterface() {};
  virtual ~doCommandCenterCommanderInterface() {};
 
  virtual const char* CommanderName() const = 0;
  virtual bool CanProcessCommand(const doCommandCenterCommandPacket* packet) const = 0;
  virtual void ProcessCommand(doCommandCenterConnection* connection, const doCommandCenterCommandPacket* packet) = 0;
};

A commander can process one or many commands (CanProcessCommand). Each commander is registered with the central command center, which does the routing of messages from connected clients to the appropriate commander. doCommandCenterCommandPacket just contains a parsed JSON object and doCommandCenterConnection has the WebSocket connection and various buffers in it.

Reporters

Support for reports is added by implementing the reporter interface:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class doCommandCenterReporterInterface {
public:
  doCommandCenterReporterInterface();
  virtual ~doCommandCenterReporterInterface() {};
 
  void Subscribe(doCommandCenterConnection* connection);
  void Unsubscribe(doCommandCenterConnection* connection);
  bool ShouldRefresh(palTimerTick tick);
  void ChangeRefreshDelay(doCommandCenterConnection* connection, palTimerTick delay);
 
  virtual const char* ReporterName() const = 0;
  // Update internal state
  virtual void Refresh() = 0;
  // Report state to all subscribed connections
  virtual void Report() = 0;
};

Each reporter is responsible for generating a single type of report. Similar to commanders, reporters are registered in the central command center.

Client commands to subscribe and unsubscribe are processed by a commander, like all other commands.

Javascript implementation notes

Connection

My web application has a communication class ‘Connection’. The interesting methods are:

FireAndForget(command) – Send a command. No callback function is registered.
Fire(command, callback) – Send a command. Register a callback to be called when the result arrives.
Subscribe(reporter_name, report_callback) – Subscribes the client to the reporter. Registers a callback to be called whenever a report comes in.
Unsubscribe(reporter_name) – Unsubscribes the client to the reporter. Unregister report callback.

Message Processing

When a message arrives the behaviour depends on the type of message. If it is a result and the caller registered a callback, the message is passed to the callback and the callback is removed from the callback table. If it is a report the report callback is called.

Modules

My web application consists of many modules. Each module covers a set of related tasks. For example, the Memory module can query for all open allocations and updates a live graph of memory used by each allocator. Each of the modules registers subscriptions and command callbacks with the connection to update itself when new messages arrive. Each module is given an HTML

At this point I expect many of you are saying “I’m not working on a web game this doesn’t seem relevant to me.” Well, neither am I. I embed a WebSocket server into my game engine and with a local web application use the WebSocket protocol as a medium to control, configure and monitor my game engine. Some concrete examples of what I’ve done so far:

• Monitor memory allocation statistics
• Monitor performance of subsystems
• Set and query configuration variables
• Live edit the world
• Loaded asset preview

Soon, I will attempt to stream the display of the game to the web browser and stream mouse,  as well as the keyboard data back to the game. In other words, remote desktop for the game engine. Another use case I would like to investigate is writing unit tests for the engine in javascript and drive it from the web browser. The possibilities are endless.

Another benefit of this approach is that your development UI is platform independent. This is nice when you are developing a title against many architectures. For example, consoles where the target does not usually have keyboard or mouse input are harder to interact with- by moving your UI to the web browser this problem is avoided.

Hang in there, this article is about the plumbing. My next article will be about the above applications.

WebSocket is a communication protocol that allows for bi-directional text and binary message passing. The client is a Javascript application running inside a web browser and, typically, the server is a web server. I say ‘typically’ because that is not how I use it.

WebSocket was developed because sending data between the web server and web application over HTTP was inefficient. The WebSocket protocol is very bandwidth efficient (message framing is at most 14 bytes) and the payloads are custom to the application. A WebSocket connection begins life as a regular HTTP connection. The connection is upgraded from HTTP to WebSocket. This upgrade is one way- you can’t revert back to an HTTP connection. You can read more about WebSocket at WikiPedia and the complete specification is available. The WebSocket protocol was only recently finalized in December 2011.

Okay, let’s dive into how WebSocket works. I will cover creating a connection, sending and receiving messages, and responding to pings.

1
2
3
4
5
6

       GET /servicename HTTP/1.1
       Host: server.example.com
       Upgrade: websocket
       Connection: Upgrade
       Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
       Origin: http://example.com

1
2
3
4

       HTTP/1.1 101 Switching Protocols
       Upgrade: websocket
       Connection: Upgrade
       Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

1
2
3
4
5
6
7

  SpecifcationGUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
  FullWebSocketKey = concatenate(Sec-WebSocket-Key, SpecifcationGUID);
    // dGhlIHNhbXBsZSBub25jZQ==258EAFA5-E914-47DA-95CA-C5AB0DC85B11
  KeyHash = SHA-1(FullWebSocketKey);
    // 0xb3 0x7a 0x4f 0x2c 0xc0 0x62 0x4f 0x16 0x90 0xf6 0x46 0x06 0xcf 0x38 0x59 0x45 0xb2 0xbe 0xc4 0xea
  Sec-Websocket-Accept = base64(KeyHash);
    // s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

Transmission

WebSocket is a message based protocol. Each message begins with a header defining the length of the message, the type (text, binary or control) and other meta-data. The payload immediately follows the header. All incoming messages will include a 32-bit mask which must be applied to the entire payload with a XOR operation. Each message will have a different mask. The masking is used to guard against simple snooping.

The header begins with a 16-bit mask (blue) and up to 12-bytes of optional header (orange).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

struct WebSocketMessageHeader {
  union {
    struct {
      unsigned int OP_CODE : 4;
      unsigned int RSV1 : 1;
      unsigned int RSV2 : 1;
      unsigned int RSV3 : 1;
      unsigned int FIN : 1;
      unsigned int PAYLOAD : 7;
      unsigned int MASK : 1;
    } bits;
    uint16_t short_header;
  };
 
  size_t GetMessageLength() const;
  size_t GetPayloadOffset() const;
  size_t GetPayloadLength() const;
  uint32_t GetMask() const;
  uint8_t GetOpCode() const;
  bool IsFinal() const;
  bool IsMasked() const;
  // …
};

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class WebSocketServer {
public:
  WebSocketServer();
  int AcceptConnection(TcpListener* listener);
  int CloseConnection();
  void Update();
 
  int SendTextMessage(const char* msg);
  int SendTextMessage(const char* msg, size_t msg_length);
 
  uint64_t PendingMessageCount() const;
  void ProcessMessages(OnMessageDelegate del, void* userdata);
  void ClearMessages();
};

The external linked list of type T looks like this:

template<typename T>
  struct ListNode {
  struct ListNode* next;
  struct ListNode* prev;
  T node_data;
};

template<typename T>
struct List {
  struct ListNode<T> root;
};

This is straight forward enough. The node contains a pointer to the next node, the previous node and memory for a single instance of T. The list itself contains the root node. I trust we have all implemented this linked list, at school or elsewhere.

I hope regular readers will notice this linked list couples memory allocation with algorithms, a pet peeve of mine. Whenever an instance of T is added to the list the following happens:

1. A new node is allocated
2. A copy of the instance being added to the list is made and stored in node_data
3. The list surgery is performed

Depending on the type of T the copy could be very expensive.

Another potential inefficiency is that root.next and root.prev can be NULL when the list is empty. Making the code to implement adding and removing nodes from the list unnecessarily branchy.

Also, depending on the implementation the node_data memory in root is wasted or we make the code even more complex. The complexity comes from handling special cases, for example, are we adding the first node to the list, in which case put the result in root.node_data otherwise, allocate a new node and store the copy in it. Ugh.

Okay, enough build up. Let me introduce the intrusive linked list:

struct IntrusiveListNode {
  IntrusiveListNode* next;
  IntrusiveListNode* prev;
};

struct IntrusiveList {
  IntrusiveListNode root;
}

Note that there is no mention of the type that the list stores. This probably seems strange at first. In particular, there is no memory to store the actual data item being stored in the list.

Intrusive linked lists flip the memory layout inside out. Instead of the list node providing memory for a T, T provides memory for a list node. The ‘intrusive’ part of the name comes from the fact that we store the list node inside the type T.

Another thing to note is that the list is circular. Upon creation this happens:

IntrusiveList () {
  // circular
  root.next = &root;
  root.prev = &root;
}

The circular nature of the list makes inserting and removing nodes simple and branch free.  Add and Remove look like this:

// Add node between next and previous
void Add(palIListNode* node, palIListNode* prev, palIListNode* next) {
  node->next = next;
  node->prev = prev;
  next->prev = node;
  prev->next = node;
}

// Remove node
void Remove(palIListNode* node) {
  node->next->prev = node->prev;
  node->prev->next = node->next;
  node->next = NULL;
  node->prev = NULL;
}
 
The API looks like this:

// Add to head of list
void AddHead(IntrusiveListNode * node) {
  Add(node, &root_, root_.next);
}

Again, note that this list does not know or care what type T the IntrusiveListNode is stored in. Or that the IntrusiveListNode is even stored in a type that C/C++ is aware of- it can work inside a blob easily.

At this point, at least one of you should be pointing out that Boost already has an intrusive linked list container type available.  Great.  Look everyone, it’s that guy. First of all, stop using Boost. That is only half serious. But, I don’t use it. It is just that the intrusive linked list that I am detailing is better than the Boost implementation because in my implementation, IntrusiveListNode, is just plain old data, allowing many of them inside a T. Unfortunately, the Boost implementation relies on T inheriting from the Boost intrusive linked list node type.

Time for an example:

struct GameObject {
  char name[32];
  int health;
  IntrusiveListNode render_list_node;
  IntrusiveListNode update_list_node;
};
 
GameObject monkey;
GameObject crate;
GameObject banana;
IntrusiveList render_list;

To add our game objects to the render list you might write this:

Render_list.AddHead(&crate.render_list_node);
Render_list.AddHead(&monkey.render_list_node);
Render_list.AddHead(&banana.render_list_node);

Let us take a moment to look at how an intrusive list looks in memory:

After looking at this diagram, you should note that as you walk render_list you don’t actually point at a GameObject, but an IntrusiveList item. So, how do you recover the address of the GameObject from the update_list_node pointer that is actually used in the list? It is quite simple, you just subtract the offset from the address. Yes, you can do that. You get the offset with offsetof:

size_t offsetof(type, member);

What is that you say? offsetof is only valid for plain old data? You cannot use it when you have diamond virtual inheritance? Uh oh, that guy is back. Yes, you are correct. But, luckily no sane programmer would ever use such a thing.

This might sound tedious to work with. Having to walk the list using IntrusiveListNode pointers and apply the offset when you want the data is definitely not as nice as the standard linked list implementation. To solve this, you create a list node iteration helper:

template <typename T, size_t offset>
struct IntrusiveListForeach {
  …
}

Aside from the expected, Next() and Prev() functions for moving along the list, the GetListEntry() function applies the negative offset and returns a pointer to data associated with the list. IntrusiveListForeach also has a Remove method that removes the node safely:

void Remove() {
  palIListNode* safe_next = current->next;
  list_->Remove(current);
  current = safe_next;
}

Compared to the external linked list, the intrusive linked list decouples memory allocation from the container itself. Meaning that to add something to the list, you do not necessarily need to perform an allocation. A side effect of this decoupling is that the linked list container never copies the data by calling the copy constructor. This will avoid many unnecessary copies. You could even add some static data to a list without creating a copy at run time. Another benefit of the intrusive list is that the list node and the surrounding data are guaranteed to be share a cache line, you can even place it strategically! The circular implementation allows the core list manipulation routines to be implemented without branches and very little code (Note: circular list implementations are not restricted to intrusive linked lists). Unlike the Boost intrusive linked list, this implementation does not use inheritance.

A downside of this approach, as discussed in my previous article is that the debugger is not very helpful at displaying this structure out of the box. If you’re going to spend the time implementing your own container structures, it is very little extra work to teach your debugger about them.

I have both an intrusive and external in my container library. I have not done this, but,  it should be possible to implement an external implementation on top of the intrusive.

I should note that I rarely use linked lists anymore. Dynamically sized arrays are almost always more efficient because they are packed tightly.  In cases where you need to add/remove items from arbitrary locations and keep the order (stability), a linked list is still a good bet.

What do your linked list look like? Does it decouple memory allocations from the container work itself? Can it work inside a blob? I am always interested in hearing about your ideas.  Yes, even if you are that guy.

Before presenting my blob array, I will present my foil, the typical dynamic array:

template<typename T>
struct dynamicArray {
  T* buffer;
  size_t capacity;
  size_t size;
};

Pretty straight forward, right? You have a contiguous chunk of Ts in buffer, of which [0, size) are used. The buffer has room for capacity elements. When elements are added and size becomes equal to capacity, the capacity is increased and the old buffer is copied into the new larger buffer’ before being deallocated. Text book.

The keen reader won’t be satisfied with this array. First of all, it has memory allocation and algorithms coupled together. What if you want to use a static buffer to store the contents of the array? Sure, the capacity is fixed when using a static buffer but the algorithms for working with the array haven’t changed and thus shouldn’t be tied to the memory management scheme. Also, what if you know that you only ever need 8-bits to store the capacity and size information instead of 64-bits (the size of size_t on a 64-bit OS)? What if you want to serialize this in and out of a file? This dynamic array is starting to look tired.

There’s gotta be a better array.

template<typename T, typename INDEX_TYPE = int32_t>
struct blobArray {
  uintptr_t memory;
};

That’s it. Where is the capacity, size and contents? As usual with blob structures, you will have to look inside the blob to gain insight:

Assuming INDEX_TYPE is int32_t, capacity is offset 0 bytes into the blob, size is offset 4 bytes into the blob and contents is offset 8 bytes into the blob.

blobArray has an Initialization method which takes a memory address and a size. The type of contents is specified as a template parameter, so Initialize just has to zero out size and set capacity using the following formula:

capacity = (blob_size - sizeof(INDEX_TYPE) * 2) / sizeof(T);

Note: I’ve simplified the above formula by removing alignment considerations.

Compared to my foil, my blobArray is looking good. Most importantly, it has decoupled memory allocation from the algorithms used to manage a dynamic sized array. Put another way, it works beautifully with static buffers. The type of the index is parametrized, allowing for the most compact storage. Because the entire state is encoded into one contiguous piece of memory, serializing is free, you can write the entire array to a file and read it back atomically.

Keen readers should take this opportunity to point out that a blobArray couldn’t possibly expand it’s capacity.

Don’t worry, I’m with you. Enter dynamicBlobArray:

template<typename T, typename INDEX_TYPE = int32_t>
struct dynamicBlobArray {
  blobArray<T, INDEX_TYPE> blob_array;
  allocatorInterface* allocator;
};

This class couples the algorithms offered by blobArray with a memory allocator. Allowing the capacity to increase when needed. dynamicBlobArray exposes the same interface that blobArray exposes by forwarding function calls, example:

T* dynamicBlobArray::begin() {
  return blob_array.begin();
};

Both of these template classes will be implemented in header files, allowing the compiler to inline the function forwarding. Of course, dynamicBlobArray will special case methods that add elements to the array, example:

void dynamicBlobArray::push_back(const T& e) {
  if (IsFull()) {
    ExpandCapacity();
  }
  _blob_array.push_back(e);
}

Hopefully, no one thought I was going to make push_back a virtual method, use inheritance and override it in dynamicBlobArray. Ick.

Both blobArray and dynamicblobArray expose a super-set of std::vector methods. Making them transparent replacements for an existing code base.

There is one down side to using this blob structure: the debugger cannot automatically display state information for these arrays. To get around this, you need to extend the debugger so that it can visualize the internal state hidden in the blob. You can extend Visual Studio by adding custom visualizers to autoexp.dat.

This container has worked well for me. I get both static and dynamic capacity arrays with just one implementation. Also, easy serialization and compact storage. How have you implemented yours?

Some of you may be asking ‘So what is a blob?’. A blob is just a chunk of memory. It does not care the type of data being stored in it, whether or not it’s homogeneous, etc. I differentiate between blobs by the type of access patterns it offers. This is my first blob, the basic blob:

struct BasicBlob {
    uintptr_t buffer;
    size_t buffer_size;
};

That is all the data it stores. It’s either 8 or 16 bytes depending on the pointer size. Aside from initialization, my basic blob only has two methods:

size_t GetBufferSize();
 
template <typename T>
T* GetPtr<T>(size_t byte_offset);

The purpose of the first method is obvious. The second, GetPtr<T>, does the pointer offset computation and returns a pointer of the requested type.

The most important feature of the basic blob is that it does not own the memory. It only stores an address and a size. This moves the memory allocation policy outside of the blob, where it belongs. As a side effect, the basic blob is a POD-type. Meaning that, instances can be passed by value, re-initialized with different addresses and sizes without side effect. I use this blob everywhere I need to pass a pointer and a size.

The second blob I use is the growing memory blob. Unlike the basic blob and the remaining two, this blob does own the memory. It is initialized with an allocator, has a (dynamic) capacity and a (dynamic) size. This blob could and should be used in your dynamic array and dynamic string classes, which are just dynamically sized contiguous blobs of memory anyway. Also, my file system abstraction relies on this blob when returning the contents of a file on disk.

Like the basic blob, the growing blob offers the GetPtr access method. It adds an Append operation:

template <typename T>
int Append<T>(const T* items, int item_count);

There are a couple of variations on this definition, but you get the picture. It appends item_count * Ts to the buffer. If the buffer is full, the capacity is increased and then the append takes place. I primarily use this blob when I am building larger blobs to be written to disk and later read back in as a read-only resource (a basic blob).

My third, and most complex blob, is a ring buffer blob. This blob does no memory allocation and does not own the memory, but keeps track of a read and write pointer inside the buffer. It has the same Append method as the growing blob. It adds an analogous Consume routine.

A ring buffer, while simple, is not as simple as a linear chunk of memory. Both appending and consuming from a ring buffer must handle the case when the pointer must wrap from the end of the buffer to the beginning. There are also many choices on how to implement a circular buffer

This blob is starting to feel kind of heavy, isn’t it? What if you want to read some data without removing it from the blob or if you just want to append in small chunks and then atomically adjust the write pointer? A good abstraction will always get out of the way and let you, the one with the brain, the programmer, dig inside it’s guts. My ring blob offers that in spades. You can access the read and write pointer through a template function like the above GetPtr<T>. But, what about the discontinuous point at the end of the buffer? You can query for the consecutive amount of bytes each pointer has before it will wrap around back to the beginning, allowing you to split your direct reads and writes around the end of the buffer. You can Peek into the buffer and later Skip those bytes. You can write into the buffer and then MoveWriterPointer over what you wrote.

I use the ring blob primarily as a message passing pipe. One part of my system writes messages with the following header:

struct MsgHeader {
    uint8_t op_code;
    uint16_t payload_length;
};

While another part of the system consumes those messages and reacts to each op code and payload. This avoids the coupling together of two different modules in my code, allowing a module to be completely replaced without impacting the rest of the code base so long as it still responds to the same messages.

The ring blob is great, but it is heavy and I found I wanted something simpler. Also, the ring blob did not work well when dealing network streams and native socket APIs. So, the final blob I use, I call the ‘Append & Chop’ buffer. This blob keeps track of an append pointer and offers the above Append method. The Chop method could also be thought of as a shift operation. When you’re done with the beginning of the buffer you Chop it off and shift all the memory down. Because this blob’s memory is continuous within the region it works better with network stream APIs and is generally simpler.

I want to say something about implementing blobs that also applies to programming in general. Separate your memory allocation policy from your algorithm code. A higher level module should be used to glue the two together in the most optimal way. It always troubles me to see modules that allocate their own memory, restricting programmer freedom to pass in a pointer from inside a blob or on the stack. This is another form of coupling, and coupling sucks.

Another implementation note is that although these blobs share subsets of each others APIs, I don’t use inheritance or virtual methods. This avoids all of that unnecessary shit overhead. But, because the syntax of calling the methods is shared across blob types, I can still swap out the type of blob being used and have minimal impact on existing code.

I also have functions that read from one blob and write into another.

I love my blobs. They’ve become a core part of my technology and the freedom of not being tied to C/C++’s type system and structure layout algorithms is very liberating. What do your blob abstractions look like? By including small descriptive headers (see MsgHeader) you can describe the data inside the blob for the module using it. This gives us a very simple type system. Have you experimented with a more sophisticated blob type system than an op (or type) code followed by a length field?