Recently my new hobby project, elisp.js, reached version 0.1.0! It is a basic Elisp environment that runs of top of Javascript and runs code through transpiling it into Javascript dynamically and evaluating the intermediate JS code.

I consider the core of the implementation reasonably self-sufficient by now: lambdas and macros work reasonably well (with proper dynamic scoping and Lisp-2 namespace separation, unlike Samsonjs’s elisp.js from 2009!), code may be evaled, the most used types are implemented (symbols, lists, strings, numbers, bools, vectors).

On the other hand, the implementation is still very bare, lacks many essential types (e.g. hashtables) and even language-level functionality (e.g. &optional and &rest arguments in functions), still wants for proper error handling and debugging and it completely lacks any chrome around the core primitives (even defmacro and defun are very recent additions).

Also, my implementation is not a transpiler in the strict sense of “code in, code out”: Lisp is still too dynamic and fluid in runtime to map into some static Javascript. Macros make a full transpiler necessary even in runtime. I’d rather call it a “Just-In-Time Transpiler”.

If you are still keen to try it and are fine with using its source code as a reference, you can find a basic online demo here: https://dmytrish.net/elisp.

Why?

My first and the most moon-shot reason has been the dream of seeing org-mode in the browser. I still hope that I might be able to write just enough functionality to support significant chunk of org-mode on top of, say, Ymacs.

Also, I’ve been curious to learn more about Javascript/v8/node.js and v8’s performance. This goal has definitely paid off: I gained some insight into speeding up Javascript code and inner workings of v8, even though I am still very far from an expert in this.

Development notes

Test-Driven Development is cool–when you have a clear specification/reference implementation like the Elisp reference manual and Emacs (just look at this for example!). The right amount of end-to-end tests makes writing Javascript almost not scary, Mocha is pretty fast and provides with nice guardrails to structure lots of tests. Hopefully, extensive testing will make the project at least a bit more maintainable and evolvable.

Dynamic scoping. With lexical scoping (i.e. almost any other kind of Lisp) translation to Javascript would be a lot easier (at least leaving out macros), but Elisp is the only surviving dinosaur of dynamic scoping, so I had to map dynamic bindings into Javascript carefully. This hinders v8 from many optimizations (obscuring the intent) and produces ugly JS code like this:

elisp> (defun sqr (x) (* x x))
(lambda (x) (* x x))

elisp> (jscode (symbol-function 'sqr))
"(() => {
  let f1 = global.env.fun('*');
  let v2 = global.env.var_('x');
  return f1.get().fcall([v2.get(), v2.get()], global.env);
})"

Pain of macros. Before I started working on macros in Elisp, the source code organization was straightforward and dependencies were unidirectional: I had types.js describing Lisp types, parser.js was turning an input string to a Lisp object, translate.js was taking a Lisp object and producing ephemeral JS code from it; elisp.js nicely packed everything into a high-level wrapper.

Once I’d added macros, the architecture became a mess: now even types.js requires elisp.eval_lisp(), creating circular references and lazy-loading in parser.js and translate.js. Maybe, I will be able to abstract the runtime parts from these files later, but macros really made the interpreter source code tightly coupled and intertwined. Just look at this pearl (yes, every function call might require a recompilation in Elisp):

LispFun.prototype.fcall = function(args, env) {
  try {
    return elisp.fcall.call(this, args, env);
  } catch (e) {
    if (e.message !== 'Macro accessed as a function')
      throw e;
    /* trigger recompilation */
    this.func = undefined;
    return elisp.fcall.call(this, args, env);
  }
};

Repl tab completion is easy and surprisingly useful. My heuristic for tab completion is very crude and dumb at the moment: just looking back until the start of the current symbol and if there’s a (, complete the symbol as a function, otherwise complete it as a variable. Still, it works pretty reliably and saves a lot of typing. Also, this gave me a little insight why on earth anybody would prefer Lisp-2 (an ugly, ugly idea compared to Lisp-1!): now I understand that Lisp-2 might be slightly easier for tooling.

Profiling and benchmarking

v8 is fast!

Still, my first naive version that could run silly-loops made even v8 stumble compared to the slow Emacs interpreter. I had to learn about v8 profiling and it took several significant optimizations for it to become competitive with Emacs (this shows execution time in seconds):

$ npm run bench

> elisp.js@0.1.0 bench /home/mitra/code/elisp.js
> ./bench/bench

funcall
     now: 0,285
   emacs: 0,255
silly-loop
     now: 0,834
   emacs: 1,845

There is no free lunch, a slow implementation is still a slow implementation even on a fast engine. Profiling can guide to faster code though, so it’s better to say “v8 can be fast”.

Still, I find the results pretty satisfying for my limited resources.

What exactly is done in 0.1.0

Citing the CHANGELOG,

  • language types: integer, symbol, nil, cons, string, vector; LispFun and LispMacro as invisible “upgrades” to lists when appropriate;
  • language special forms: let, lambda, quote, setq, if, progn, while;
  • language subrs (builtin functions): subrp, functionp, macrop, listp, numberp, booleanp, read, eval, macroexpand-1, jscode, jseval, +, -, *, <, car, cdr, cons, list, fset, symbol-function, error, print, float-time.
  • functions: positional arguments only, dynamic scoping;
  • UI: repl.js with defmacro, defun, defvar; tab-completion of functions and variables;
  • UI: a browser demo using webpack;
  • benchmarked and optimized, in the ballpark of Emacs interpreter or faster.

Things to do before 0.2.0

The main goals for 0.2.0 at the moment are:

  • Make it parse and interpret non-trivial amount of source code from lisp/emacs-lisp/ of Emacs;
  • Add meaningful robust error handling with stacktraces, pluggable debuggers and references to source code;
  • Extend the standard library of subrs on the go as needed;
  • Give more thought to Emacs-specific types like buffers, cursors, charmaps, etc;
  • Starting at least some form of documentation.