Oni Apollo 0.12: Cross-domain module loading directly from GitHub

March 28, 2011 by Alexander Fritze

We're pleased to announce the release of version 0.12 of the client-side StratifiedJS runtime Oni Apollo.

The main user-visible change in this release is an update to Apollo's require system. Apollo's require.hubs variable can now be used to customize the way modules are being loaded for a given url scheme. Previously this variable was used only to resolve the place where a module was loaded from, by specifying a replacement string for a given prefix, e.g.:

require.hubs.push(["mycode:", "http://www.mydomain.com/"]);
...
require("mycode:foo"); // loads http://www.mydomain.com/foo.sjs

In Apollo 0.12 you can now customize the loading process itself by associating a loader function with the prefix:

function mycode_loader(path) {
  var code = ... custom loader algorithm
  return { src: code };
}

require.hubs.push(["mycode:", mycode_loader]);
...
require("mycode:foo"); // calls mycode_loader() to get the module's
                       // source, then compiles it.

Builtin GitHub Support

We're leveraging this new require functionality to provide builtin support for GitHub. require.hubs is now pre-configured to load modules with a 'github:' prefix directly from GitHub (much like the 'apollo:' prefix is configured to load from the canonical standard module library location at code.onilabs.com). The syntax looks like this:

require("github:USER/REPO/BRANCH_OR_TAG/MODULE_PATH");

E.g. to load the module https://github.com/afri/testmodules/blob/master/foo.sjs and call its hello function, you could write:

var foo = require("github:afri/testmodules/master/foo");
foo.hello();

The GitHub module loading process works cross-browser and without any intermediate proxies. The browser talks directly to the GitHub API using JSONP-style requests.

The loading functionality also works transitively. I.e. if you load a module from GitHub that in turn references another module through a relative url (e.g. require('anothermodule')), it will load fine through this mechanism.

This new GitHub functionality comes in pretty handy for those situations where you just want to quickly try out a 3rd party module, or if you want to share modules during development without constantly having to upload to a webserver. In fact it works so well, it can be used in (small-scale) production as well. E.g. the xlate Chrome extension uses it to pull in a sjs4chromeapps support script:

var tabutil = require('github:afri/sjs4chromeapps/master/tab-util');

How does it work?

Behind the scenes, the GitHub loading process works something like this:

First the 'tree SHA' is determined by simultaneously querying GitHub for both tags and branches in the given repository (using the Repositories API):

waitfor {
  // search for 'tag' in repo's tags; hold forever if not found: 
  (tree_sha = jsonp([github_api, 'repos/show/', 
                     user, repo, '/tags']).tags[tag]) || hold();
}
or {
  // search for 'tag' in repo's branches; hold forever if not found:
  (tree_sha = jsonp([github_api, 'repos/show/', 
                     user, repo, '/branches']).branches[tag]) || hold();
}
or {
  // timeout after 5 seconds
  hold(5000); 
  throw new Error("Github timeout or tag/branch not found");
}

Using the tree SHA, the file data at the given path is then retrieved using GitHub's Object API:

waitfor {
  var src = jsonp([github_api, 'blob/show/', 
                  user, repo, tree_sha, path]).blob.data;
}
or {
  hold(5000);
  throw new Error("Github timeout");
}

A fine example of stratified programming, we hope you will agree :-)

Preparing for Node.js and Chrome OS support

In this release we also refactored the Apollo internals in preparation for the upcoming Node.js and Chrome OS support. We have been working on Apollo proof-of-principles for these environments for a while (see sjs-nodejs and xlate, sjs4chromeapps) and identified the major sticking points. In Apollo 0.12 we've begun systematically addressing them (see the github commit history for details).

The goal is to have Apollo working seamlessly with a unified standard module library not only cross-browser, but also on the server and in browser apps. We plan to achieve this within the next couple of Apollo releases.

We get asked about Node.js support a lot these days. We want to get this right though, so please bear with us a little longer!