Sharing Structs Between JavaScript and Native Code

When writing native modules in C/C++, it is often useful to structure data using something called structs. A struct is a low-level data schema you can use in C/C++. They basically act as an encoding/decoding scheme, similar to protocol-buffers and many others data encoding schemes, to read/write the properties specified in the struct declaration.

#include <stdio.h>
#include <stdint.h>

// declare a struct with two properties
struct test {
  uint32_t an_unsigned_num;
  int32_t a_signed_num;
};

int main () {
  // instantiate a new struct
  struct test an_instance;

  // write some properties
  an_instance.a_signed_num = -42;
  an_instance.an_unsigned_num = 42;

  // read them out again
  printf(“a_signed_num=%i, an_unsigned_num=%u\n”,
    an_instance.a_signed_num,
    an_instance.an_unsigned_num
  );

  return 0;
}

Try compiling the above example using gcc by doing:

 gcc program.c -o program 
 ./program 

Behind the scene structs are just syntactic sugar for calculating the byte offset of each property specified in the struct declaration. Reading/writing a property value is sugar for reading/writing the value into a buffer at that exact offset. In fact, the struct itself is just a buffer that has enough space to hold all the properties. Since we know the type of each of the properties (fx int32_t) and since each type has a static size, these offsets aren't too complicated to calculate.

Looking at the struct from our above example let's try calculating the offsets and total byte size needed to store all values.

struct test {
  uint32_t an_unsigned_num;
  int32_t a_signed_num;
};

The first property, an_unsigned_num has offset 0 and the size of its type, uint32_t, is 4 bytes. This means the offset of the second property is 4 and since the size of int32_t is also 4, the total struct size is 8.

We can verify this using another C program

#include <stddef.h>
#include <stdio.h>
#include <stdint.h>

struct test {
  uint32_t an_unsigned_num;
  int32_t a_signed_num;
};

int main () {
  // We can use offsetof and sizeof to verify our above calculation
  printf(“offset of un_signed_num=%zd\n”,
    offsetof(struct test, an_unsigned_num));
  printf(“offset of a_signed_num=%zd\n”,
    offsetof(struct test, a_signed_num));
  printf(“total size of struct=%zd\n”,
    sizeof(struct test));
  return 0;
}

Compiling and running this program should produce similar output to our calculation above.

Type Alignment

If all of the types do not have the same size, it's a little bit more complicated than that. If we were to use a bigger number type such as uint64_t, which is 8 bytes, something interesting happens to our offsets.

#include <stddef.h>
#include <stdio.h>
#include <stdint.h>

struct test {
  uint32_t an_unsigned_num;
  // Use a int64_t instead of int32_t which is 8 bytes instead of 4
  int64_t a_signed_num;
};

int main () {
  // We can use offsetof and sizeof to verify our above calculation
  printf(“offset of un_signed_num=%zd\n”,
    offsetof(struct test, an_unsigned_num));
  printf(“offset of a_signed_num=%zd\n”,
    offsetof(struct test, a_signed_num));
  printf(“total size of struct=%zd\n”,
    sizeof(struct test));
  return 0;
}

The output of program now changes to

offset of un_signed_num=0
offset of a_signed_num=8
total size of struct=16

Notice how the offset of the property we changed to int64_t changed to 8 instead of 4 even though we didn't change the size of the previous property? That is because of something called type alignment. When using raw memory in a program your computer uses something called memory pages behind the scene. These pages have a fixed size and to avoid having to store half a number in one page, and the other half in another, your computer prefers storing numbers that have a byte size of 8 at offsets that are divisible by 8. Similarly, it prefers to store numbers of byte size 4 at offsets that are divisible by 4. This number is called the type alignment. The total struct size is always divisible by the type alignment of the largest number type in the struct as well. That's why the total size of the struct in the above example is 16 since 16 is divisible by 8 and large enough to contain all the types.

Using Structs from JavaScript

So how do structs relate to JavaScript? If we're writing a native module we can use them to store properties in Node.js buffers. This is very useful as it allows us to allocate memory in Node.js but still use it in C/C++ programs, without the need for mallocs and other C/C++ ways of organizing memory. If we do it in JavaScript we can take advantage of the garbage collector running there as well, which makes memory leaks much easier to deal with.

Let's look at an example of this, using n-api and the napi-macros module

#include <node_api.h>
// we are using the napi-macros npm module make the native add-on easy to write
// do a npm install napi-macros to install this module
#include <napi-macros.h>

struct test {
  uint32_t an_unsigned_num;
  int32_t a_signed_num;
};

// This defines a function we can call from JavaScript
NAPI_METHOD(do_stuff) {
  // This method has 1 argument
  NAPI_ARGV(1)
  // This takes the first argument and interprets it as struct test
  NAPI_ARGV_BUFFER_CAST(struct test *, s, 0)


  // Let’s mutate the struct
  s->an_unsigned_num++;
  s->a_signed_num--;


  // And return one of the properties to js
  NAPI_RETURN_INT32(s->a_signed_num)
}

NAPI_INIT() {
  // Export the function from the binding
  NAPI_EXPORT_FUNCTION(do_stuff)
  // Also export the size of the struct so we can allocate it in JavaScript
  NAPI_EXPORT_SIZEOF_STRUCT(test)
}

Try saving the above file as example.c. Then, to compile, we need to have a gyp file as well:

{
  "targets": [{
    "target_name": "example",
    "include_dirs": [
      "<!(node -e \"require('napi-macros')\")"
    ],
    "sources": [
      "example.c"
    ]
  }]
}

Save the above file as binding.gyp.

To compile the add-on, install napi-macros and run node-gyp:

 npm install napi-macros 
 node-gyp rebuild 

We can now load this add-on from JavaScript and allocate the struct there, pass it to our add-on and mutate it in C.

// Load the add-on
const addon = require(‘./build/Release/example.node’)

// Allocate the struct. We exported the size from the add-on
// Buffer.alloc zeros the buffer, which is a good idea for struct defaults
const buf = Buffer.alloc(addon.sizeof_test)

// Call the do_stuff function and pass our struct buffer to C
let signedNumber = addon.do_stuff(buf)

console.log(‘struct.a_signed_number is’, signedNumber)
// Call do_stuff again. It should return -2 now since we mutated the struct in C twice
signedNumber = addon.do_stuff(buf)

console.log(‘struct.a_signed_number is’, signedNumber)

We can even log out the buffer using console.log and see that it updates every time we call addon.do_stuff.

// Will print the buffer that is representing the raw memory of the struct
console.log(buf)
addon.do_stuff(buf)
console.log(buf)

npm Install Shared-Structs

So we can allocate buffers in JavaScript and pass them to a C add-on and mutate them there. What if we could read/write the values from JavaScript as well? There is npm module for that. It's called shared-structs and it implements the offset/size calculations described above in the post.

It's pretty straightforward to use. Simply pass the struct declaration to the JavaScript constructor and it'll generate a JavaScript object exposing the same properties as the struct.

const sharedStructs = require(‘shared-structs’)

// parse the struct and generate a js object constructor
const structs = sharedStructs(`
  struct test {
    uint32_t an_unsigned_num;
    int32_t a_signed_num;
  };
`)

// allocate a new struct
const test = structs.test()

// use it as a normal js object
test.an_unsigned_num = -1
console.log(test.an_unsigned_num)

The shared-structs module uses TypedArrays behind the scene to encode/decode the values. The way TypedArrays, such as a Uint32Array, encode values is the same way native code does it, which is useful when trying to implement structs in JavaScript.

To print out the underlying buffer that the struct is storing its data in access test.rawBuffer

console.log(test.rawBuffer)

In fact, we can pass this buffer to C, to manipulate it there. Let's update our previous example to use structs in JavaScript.

const addon = require(‘./build/Release/example.node’)
const sharedStructs = require(‘shared-structs’)

// parse the struct and generate a js object constructor
const structs = sharedStructs(`
  struct test {
    uint32_t an_unsigned_num;
    int32_t a_signed_num;
  };
`)

// allocate a new struct
const test = structs.test()

// Call the addon, but pass test.rawBuffer as the struct
addon.do_stuff(test.rawBuffer)

// Now we can read out the values in JavaScript as well!
console.log(test.an_unsigned_num)
console.log(test.a_signed_num)

We can also write the values in JavaScript before passing it to C.

// Write a value to the struct and call the add-on
test.a_signed_num = -100
addon.do_stuff(test.rawBuffer)

// Will print -101 since the addon decrements this property
console.log(test.a_signed_num)

Values that are impossible to represent in JavaScript, such as pointers, are simply skipped in the JavaScript struct, but still stored in the buffer.

An interesting side-effect of parsing the structs in JavaScript is that it can reduce the number of calls you need to do to your native add-on. Every call to a native add-on is much more expensive than calling a normal JavaScript function since it has to do a "boundary cross" between the JavaScript VM and the native code, which requires certain checks. Reading the struct in JavaScript can be done inside the JavaScript VM and can, therefore, be much faster if you have to read/write properties a lot. This kind of memory mapping is actually used by Node.js core itself to speed up interactions between C++ and JavaScript.

The shared-structs module even works with nested structs, arrays, defines, constant expressions, and more!

I hope you find it as useful as I have in making it easier to write native add-ons in Node.js.