Stratifying asynchronous storage

December 01, 2010 by Tom Germeau

I came across this awesome article explaining why asynchronous programming in JavaScript can be painful, called "Taming asynchronous JS with CoffeeScript" by Tim Cuthbertson. Here's a quote from the article, but I suggest you read the whole thing.

Control flow is a big part of how applications are structured. Conditional statements (if/else) and looping constructs (while/for) are the building blocks of programming. The problem is, callbacks make proper control flow difficult, because asynchronous events happen outside of the regular program flow.

In the article, Tim makes his point with examples that use asynchronous storage, in particular the Lawnchair browser storage library. One of Tim's examples, retrieving a set of documents by their ID, looks trivial at first sight. In a synchronous world, it would look like this:

Naive sequential version
var documents = [];

while (document_ids.length) {
  var document_id = document_ids.shift();
  documents.push(database.get(document_id));
}

console.log(documents);

It would be nice if we could write code like this, but the problem is that database.get() is an asynchronous method: It returns immediately (without any useful return value) and only later yields the actual document data by invoking a callback. To deal with this, we need something a little more elaborate:

Without StratifiedJS
function get_all_documents(callback) {
  var documents = [];
  function get_next_document() {
    if (document_ids.length == 0) {
      return callback(documents);
    }
    var document_id = document_ids.shift();
    database.get(document_id, function(doc) {
      documents.push(doc);
      get_next_document();
    }
  }
  get_next_document();
}

get_all_documents(function(documents) {
  console.log(documents);
});

As we can see, even for such a seemingly simple task as sequencing a bunch of asynchronous database accesses, we ended up with a relatively long piece of recursive logic. Tim notes:

While fans of recursion may prefer this, there’s no doubt that it is far less concise – and all because we had to use callbacks!

Tim goes on extending the example to the point where it gets pretty unmanageable in conventional asynchronous callback style. He then proposes a solution based on a his CoffeeScript 'defer' construct. What I want to show in this article is what the solution looks like in StratifiedJS.

It's surprisingly simple actually:

StratifiedJS
var documents = [];

while (document_ids.length) {
  var document_id = document_ids.shift();
  documents.push(database.get(document_id));
}

console.log(documents);

That's right, in StratifiedJS the code looks exactly like the naive sequential version I presented earlier! This is what StratifiedJS is all about: Allowing you to code asynchronous logic in a conventional, callback-less, blocking style.

So how does this work? The 'magic' lies under the hood in our 'stratified' implementation of database.get(). It looks something like this:

StratifiedJS
database.get = function(key) {
  waitfor(var rv) {
    this.async_get(key, resume);
  }
  return rv;
}

Here, waitfor()/resume is one of the special constructs that StratifiedJS adds to the JavaScript language. You can find out more in the StratifiedJS docs, but in a nutshell, the above code executes this.async_get(), passing in a special callback function called 'resume', and blocks execution of the current stratum until async_get() calls back resume().

While the stratum that called database.get() blocks and waits for database.async_get() to perform its work, the browser UI will stay responsive. At the same time, other parallel strata not waiting for database.get() will happily continue running.

Stratified storage APIs in practice

As we've seen, conventional asynchronous APIs can quickly get cumbersome and land us in 'asynchronous spaghetti land'. Fortunately, armed with the waitfor()/resume construct, we can easily stratify any asynchronous API.

Oni Apollo, the browser runtime for StratifiedJS, ships with a collection of such 'wrapper' APIs. In addition, there are also a bunch of stratified modules for clientside and serverside database storage:

Lawnchair

The stratified Lawnchair module wraps Brian LeRoux's clientside storage library Lawnchair.
There's an example using the stratified Lawnchair library on the Apollo examples page.

Web SQL Database

Lawnchair is great for simple key-value object storage, but sometimes - even on the clientside - you want the full might of a relational database. For Chrome and Safari browsers there is a stratified wrapper of Webkit's WebDatabase. Let's compare how to copy a row in StratifiedJS and in conventional JavaScript:

StratifiedJS
var db = require("webdatabase").openDatabase("CandyDB", ...);
try {
  var kids = db.executeSql("SELECT * FROM kids").rows;
  db.executeSql("INSERT INTO kids (name) VALUES (:name);", [kids[0]]);
  alert("done");
} catch(e) {
  alert("something went wrong");
}
Without StratifiedJS
var db = openDatabase("CandyDB", ...);
db.transaction(function(tx) {
  tx.executeSql("SELECT * FROM kids", function(results) {
    tx.executeSql("INSERT INTO kids (name) VALUES (:name);", 
      [results.rows[0]], 
      function() {
        alert("done");
      },
      function(e) {
        alert("something went wrong");
      }
    );
  });
});
There's an example using the stratified WebDatabase on the Apollo examples page.

CouchDB

Finally, as an example of a stratified API for accessing server-side storage, there is a CouchDB+StratifiedJS project on github. As the name says, it provides a stratified frontend to CouchDB, the popular NoSQL database.

Since accessing a CouchDB database involves roundtrips over the network, there is an particular potential for timeouts and other network-related failures. The following snippet shows how to use StratifiedJS's 'alt composition' construct waitfor/or to execute database operations while simultaneously waiting for a timeout and giving the user a chance to cancel the pending operations.

StratifiedJS
var db = require("couchdb").couchdb("_users");
waitfor {
  var user = db.get("trishia");
  user.firstname = "Trillian";
  user.save();
}
or {
  // timeout after 10 seconds
  hold(10000);
}
or {
  // let the user cancel the operation
  dom.waitforEvent(cancelbutton, "click")
}

Here, the waitfor/or construct simultanously waits for all of its clauses. As soon as any of them finishes, the other still blocking ones are cancelled. You can find out more about waitfor/or and other forms of strata composition available in StratifiedJS in the SJS documentation.

What about the server-side?

We know all of this would make life on the asynchronous server-side a lot easier. You can expect to be stratifying your nodes pretty soon! Stay tuned.