Serialization in web apps/JavaScript

Mon, 28 Apr 14 17:48:41 +0000

Serialization is a problem that pops up in a persistent software. In web apps, it announces itself as a problem at requirements just beyond CRUD. You suddenly have all these data models that need to be represented in local memory at runtime and also need to be saved persistently in some form.

JSON

Serialization is actually a laborious problem that touches pretty much your entire codebase.

You might think "but JavaScript objects are basically JSON, right?" Well, yes, they are. "And JSON is just a serialization format, right?" Well, yes... but it's easily possible to overestimate how much it does for you (which is fine, and is not a valid criticism of JSON).

The limitations are encountered very early on: JSON won't help you deserialize a date, and although it will help you serialize var steve = new Person('Steve') into {'name': 'Steve'}, it won't help deserialize that into something that fulfils person.getName() => 'Steve'.

Types

The way I've handled this in the past is to have a central serialization system which handles wrapping things up into JSON and unwrapping it into your application's data structures, by augmenting JSON representations with type data, and delegating down to data-structure specific implementations of serialize and deserialize.

I'm not talking about the JSON.stringify() call here, which is trivial, I'm talking about representing my model in a way that is JSON-compatible and also includes all information necessary to deserialize automatically.

For example, JSON has no date type, so we have to represent a date as a JSON primitive and do some magic either side of the serialization to convert it to a primitive and to turn it back into a JavaScript date. A serializer for date might just convert the date into a time string, like:

function serializeDate(date) {
    return date.toString();
}

This will be invoked as part of a larger routine, which uses the above function to populate the 'value' field, and itself populates the $type field:

{
    "$type": "date", 
    "value": "Mon Apr 28 2014 01:00:00 GMT+0100 (BST)"
}

A deserializer for this might look like:

function deserializeDate(value, cb) {
    cb(null, new Date(value));
}

which again exists as a part of a larger routine which analyses the $type field and finds the deserializeDate to pass the value field into.

Deserializers are best made asynchronous by default because sooner or later you'll want to represent references to objects that are stored remotely, which require a HTTP request to retrieve, and it's hard to go from synchronous to asynchronous.

deserialize()

To tie everything together, you have a top level deserialize(data, cb) function. It's useful if this function is fairly intelligent and handles anything you throw at it. For convenience, I've added the ability to recursively deserialize arbitrary JS maps, because later on it's useful to just throw a block of data at this function and get a block back, instead of having to queue up a lot of calls. I have not implemented the same case for Array in an attempt to keep the code brief, but you should consider doing so.

In the code below I've used Async and Underscore.

var typeMap = {
    'date' : deserializeDate
};

function deserialize(data, cb) {

    // JSON primitives are handled easily
    var primitives = {
        'string' : true,
        'boolean': true,
        'number': true
    }
    if (primitives[typeof data] || data == null) {
        // This is a primitive - we can just return it.
        cb(null, data);
        return;
    }

    var typeField = data['$type'];
    var deserializer = typeField && typeMap[typeField];
    if (typeof deserializer === 'function') { 
        // This is an object conforming to our ($type, value) structure.
        deserializer(data['value'], cb);
        return;
    }

    // Handle arbitrary JS objects by recursively deserializing its contents.
    else if (data.constructor.name === ({}).constructor.name) {
        var ret = {};
        async.each(_.keys(data), function(key, cb) {
            deserialize(data[key], function(err, value) {
                if (err)  { log('Some error deserializing ', data[key]); }
                else { ret[key] = value; }
                cb();
            });
        }, function(err) {
            cb(null, ret);
        });
        return;
    }
}

I've omitted the serialization code, but it's very much the same idea. You have a top level serializer that generates JSON-compatible objects by delegating to data-type specific serializers.

Deserializing your own data structures

That was pretty easy. The harder parts come when you consider your own objects.
Let's say you have a Person class. It's useful to have a way to merge serialized data into an existing Person (because this allows us to re-purpose our serialization code for handling live-update events), and it's also useful to have a way to create a new Person. Luckily, the second is just a trivial special case of the first.

Person.fromSerialized = function(obj, cb) {
    var p = new Person();
    p.mergeFromSerialized(obj, cb);
}

// Add this to the typeMap, so it becomes visible to deserialize()
typeMap['Person'] = Person.fromSerialized

Person.prototype.mergeFromSerialized = function(obj, cb) {

    // The correct way is to use deserialize(). This looks recursive, but 
    // the subtlety is that obj is not a typed object - it's just a plain 
    // JS map of properties. Because of this, the deserializer won't 
    // try to delegate and will just deserialize each member

    function take(object, key, defaultValue) {
        if (object.hasOwnProperty(key)) { 
            return object[key]; 
        }
        else { 
            return defaultValue;
        }
    }

    deserialize(obj, _.bind(function(data) {
        this.name = take(data, 'name', this.name);
        this.dob = take(data, 'dob', this.dob);
        cb(null, this);
    }, this));
}

take() is a helper function which returns the given key from an object unless that key doesn't exist, in which case it returns a default. It avoids a lot of if (data.hasOwnProperty()) {} blocks and makes the code a bit more legible.

What about references?

We still very quickly encounter yet another case: that objects need shared references. If a Person object has a friends array, the JSON doesn't want to embed the friends in that array, it wants to just store a reference. JSON has no reference support so you need to encode your own.

In this case you need a way to refer to the object itself, not its contents.

The way you handle this is to ID persistent objects and serialize them as special '$ref' objects, and expose a method on your server to return a specific object with the given ID.

{
    "$type": "Person",
    "value": {
         "name": "Steve",
         "friends": [
             { 
                 "$ref" : {
                     "$collection": "People",
                     "id": 3
                 }
             }
         ]
     }
}

This presents an interesting point: that you need two different serialized representations for objects that may exist in collections. One returns a normal keyed object representing the object's state, the other returns a $ref object. I've found the latter case to be the generally useful one, and the former to be a special case which the caller should only invoke purposefully. Meaning: serialize(myPerson) => {$ref: ... }, and myPerson.serialize() => {name: ..., dob: ..., friends: ... }.

The deserializer for a $ref looks something like:

var collections = {};
function deserializeRef(data, cb) {
    
    var collectionName = data['$collection'],
        id = data['id'];
    if (!collections[collectionName]) {
        collections[collectionName] = {};
    }
    var collection = collections[collectionName];
    if (collection[id]) { cb(null, collection[id]); }
    else {
        yourServerApi.getCollectionElement(collectionName, id, 
                                           function(err, response) {
            if (err) { cb(err); return }
            deserialize(response, function(err, response) {
                if (err) { cb(err); return }
                collection[id] = response;
                cb(err, collection[id]);
            });
        });
    }
}

Minification concerns

The mergeFromSerialized code is still a bit irritating that we have to manually write out a line for each property.

It's tempting to rewrite it to something like this:

Person.prototype.mergeFromSerialized = function(obj, cb) {
    var myFields = ['name', 'dob']; 
    deserialize(obj, _.bind(function(data) {
        _.each(myFields, function(field) {
           this[field] = take(data, field, this[field]);
           cb(null, this);
        }, this);
    }, this));
}

Unfortunately there's a glaring problem with this code: You've just trashed static analysis. If you are using a compiler which aggressively renames class members, your deserialization will fail, because your code will write to this['dob'], which your compiler has renamed to this.a.

Using such an aggressive minification might not be of great importance for a smaller project, but if your compiled JS file is measuring several megabytes, it's useful to be able to be able to trim the source.

I am not really sure what the answer to this is, other than auto-generating the long form serialization source code.

Angular vs jQuery

Thu, 03 Apr 14 20:57:08 +0000

On HN today there is an inflammatory article called 'The Reason Angular JS will fail', which tl;drs to "it's too complicated compared to jQuery".

It's not a well argued opinion, but it is one I'm inclined to agree with.

The comments all jump on the fact it is badly argued and state that comparing Angular's complexity to jQuery's simplicity is apples to oranges because jQuery isn't suitable for building large apps, whereas Angular is.

Yeah, well, no. I don't agree there.

I'm curious how many of those people have actually written a large JS app, instead of just imagined what it might involve. I've been working on what is currently a 70kloc (and growing) JS app for the last 18 months. I have used JS frameworks before, but we don't use any on this project. I think, in general, frameworks have properties that make certain kinds of projects easier to put together, and none of them is closely related to the size of the project.

We briefly looked at Angular but it seemed like it was causing us problems instead of solving them so it didn't survive long (being overly complex didn't help its case).

The statement that Angular is for big apps and jQuery is for small apps seems like a false dichotomy to me.

We use jQuery for manipulating the DOM because without two way data binding, you'd be crazy not to. Data binding is a big attraction of Angular (and others).

I like data binding from a theoretical perspective, but in practice, generating and syncing DOM really is not really a hard or notable problem. It's just a relatively minor implementation issue. I don't think data binding really provides an advantage in terms of time or effort or maintainability - it seems like it should, but you revise your opinion somewhat after spending many hours debugging data binding templates. Processes relying on other people's magic are hard to debug.

Data binding is largely a distraction from things that really do take time and energy, which tend not to be problems touched on by frameworks anyway. Things like "how do I make all these subtly inconsistent but nevertheless intuitive requirements sort of co-exist". Software is a lot about bridging the gap between reality and someone else's vision of how reality should be. That stuff is hard. That's the risk to a big project. Keeping your GUI and data models in sync is not.

But saying that our app is built with jQuery instead of Angular is sort of missing the point. We just use jQuery at the view level and it works very well. But that's all it does. We don't use the DOM to store state, which, I think, is the assertion usually levelled at 'jQuery apps'. It's just a tool for generating DOM a bit easier.

What I am saying is that I am sceptical of Angular being useful for anything more than a CRUD app on steroids, but I am not sceptical at all of jQuery being useful in virtually any HTML/JS project. This is because jQuery (roughly) adopts the philosophy of "do one thing and do it well", which means it's a very good fit for all the use cases it's aiming at. The same isn't true of Angular.

asgaard