introduce tamejs-style asynchronous constructs to avoid callback pyramids

[michaelficarra]

• update: Looks like tamejs has changed their syntax a little since July: twait is now called await and mkevent is now called defer. Luckily, they don't appear to have changed any of the semantics, though I haven't looked that hard. Take that into account when checking out tamejs.
• (originally from a comment in Add syntax for handling nested callbacks #1704)

A few months ago, I was checking out tamejs, liked the ideas, and started thinking about how it could be incorporated into coffeescript. For the uninitiated: tamejs basically just takes the JS you write and compiles it to use continuation-passing style. So the output's not so pretty. Anyway, I took a few of the examples from the website, pasted them into a gist, and rewrote them in what I called "imaginary-coffeescript-with-defer". I'll include one gist inline and just link to the other.

Imaginary CoffeeScript

{resolve} = require "dns"

do_one = (host, cb) ->
  (err, ip) <- resolve host, "A", *
  console.log if err then "ERROR! #{err}" else "#{host} -> #{ip}"
  cb?()

do_all = (hosts) ->
  defer
    for host, i in hosts
      do_one host, null
  return

do_all process.argv[2..]

Original tamejs Example

var dns = require("dns");

function do_one (ev, host) {
  var err, ip;
  twait { dns.resolve (host, "A", mkevent (err, ip));}
  if (err) { console.log ("ERROR! " + err); }
  else { console.log (host + " -> " + ip); }
  ev();
}

function do_all (lst) {
  twait {
    for (var i = 0; i < lst.length; i++) {
      do_one (mkevent (), lst[i]);
    }
  }
}

do_all (process.argv.slice (2));

There's a pretty simple mapping from the added coffeescript constructs to the tamejs additions.

• defer block is just one big twait
• (arg0, arg1, ..., argN) <- expression:
  • compile to a twait unless inside a defer
  • save args for compilation of any bare * (or whatever syntax we pick) inside expression
• bare * (or whatever syntax we pick) compiles to mkevent using args from containing <-

Now I'm not sure how appropriate it would be to add to CS because of the possibly irreparably ugly compilation. But I think it's worth a discussion even considering the numerous, extremely lengthy tickets on defer-style constructs. Hell, I think people would sacrifice the readable output for a powerful feature like that. And they would only need to do so when using that feature.

I think it makes a really good use of both <- and defer. That syntax just really seems to fit their proposed functionality.

Pinging list of tamejs contributors: @maxtaco, @malgorithms, @m8apps, @frew

[coolaj86]

I'd like to suggest that coffescript stay backwards compatible with vanilla JavaScript by using one (or some) of the flow-control micro-libraries that also work in the browser.

For example

{resolve} = require "dns"

do_one = (host, cb) ->
  (err, ip) <- resolve host, "A", *
  console.log if err then "ERROR! #{err}" else "#{host} -> #{ip}"
  cb() if typeof cb is 'function'

do_all = (hosts) ->
  defer
    for host, i in hosts
      do_one host, null
  return

do_all process.argv[2..]

Could become

var dns = require("dns")
  // includes join and futures from FuturesJS
  , Join = require("join")
  ;

function do_one (host) {
  var err
    , ip
    , join = Join()
    , resultJoin = Join()
    , resultCallback = resultJoin.add()
    ;

  dns.resolve(host, "A", join.add()); 
  join.when(function () {
    var args = arguments[0]
      , err = args[0]
      , ip = args[1]
      ;

    if (err) { console.log ("ERROR! " + err); }
    else { console.log (host + " -> " + ip); }
    resultCallback();
  });

  return resultJoin;
}

function do_all(lst) {
  var resultJoin = Join()
    , i 
    ;

  for (i = 0; i < lst.length; i += 1) { 
    // there would need to be some dditional syntax to describe
    // whether these should complete in parallel or in sequence
    do_one(lst[i]).when(join.add());
  } 

  return resultJoin;
}

do_all(process.argv.slice(2)).when(function () {
  console.log("All completed in parallel");
  // all have completed
});

The big issues are

• should be compatible with browser enabled libraries
• There should be 3 deferable types
  • a single callback (FuturesJS / Promise does this)
  • multiple callbacks in parallel (FuturesJS / Join does this)
  • multiple callbacks in sequence (FuturesJS / Sequence does this)
• what to call each type - defer vs promise vs when vs then vs chain vs foo, etc

I could make a library more focused and lightweight than FuturesJS just for this purpose with a syntax such as

• new Defer(this) - creates a single callback with the context of the current scope
• new Defer(null, true) - creates a parallel multi-callback (as in the example above) with a null context
• new Defer({}, true, true) - creates a sequence multi-callback with a new object as the context

I'm not yet a heavy CoffeeScript user, but CoffeeScript is a leader in the community and so my interest in the direction of this is that whatever conventions CoffeeScript adopts, for better or for worse, will likely be backported to vanilla JavaScript and gain long-term traction.

[frew]

(Note: I guess I'm technically a TameJS contributor but I've only submitted one bug report. I've also played around with StreamlineJS a bit, and with a couple of flow control libraries. I haven't found a silver bullet yet.)

I think the key here is to clearly enumerate the problem that needs to be solved and then work from there to a syntax to solve it. Otherwise, you're doomed by people arguing that another is a system that's better for solving their particular view of the problem.

As I see it, problems that flow control libraries/flow control precompilation steps solve include:

1. Visual yuckiness of many-nested callbacks (i.e. 15 nested inline functions).
2. Difficulty composing synchronous and asynchronous functions (i.e. a sync function must be aware if any of the functions it calls use any async functions).
3. Forced return to C-style error handling (i.e. async callbacks don't interoperate with throw/catch/finally).
4. Annoyance of having to roll your own barriers if you want multiple async operations to execute in parallel.

Empirically, different library/language designers have attributed very different weights to the 4 problems. To the best of my understanding, as a result, historically, the Coffeescript community has claimed that problem (1) is solved by Coffeescript's better functions syntax, that (4) is solved by flow control libraries, and that (2) and (3) aren't significant enough to add a continuation transformation step.

[maxtaco]

I've been trying to argue the merits of the Tame way of programming for about 5 years now, and I've long since given up on trying to convince anyone to use it who otherwise is dead set against it. However, I just wanted to add three small points to this discussion.

• When comparing tamejs to something like StreamlineJS, an important design point to consider is whether or not the system makes a distinction between "calling the callback passed in as an argument" and returning. In tamejs, these two operations can happen separately; in StreamlineJS, they are conjoined. The advantage to tamejs's approach is that the separation gives added expressiveness. A tamed function can fire its callback and then do other stuff. This is, for instance, how tamejs can implement windowing of network calls. The downside, and therefore the advantage of the StreamlineJS approach, is that most of the times you don't need that expressiveness, and when you don't, it's more code and just one more thing you can forget. With the tamejs model, every time you return, you also need to call your callback manually. This might lead to bugs if you forgot. For instance:

void function foo (callback) 
{
    if (/*something*/) {
         callback();
         return;
     } else if (/* something else */) {
         // BUG! You forgot to callback() and your program will hang.
         return;
     }
     // other stuff
     callback ();
}

• Hand-written async-style code and exceptions don't ever play well together. If you don't have threads, you will constantly see the top (read: interesting) parts of your call stacks lopped off. I can come up with examples, but I'm sure you guys have all experienced this in your own code.
• A structured system like tamejs actually makes debugging stack traces easier than hand-rolled callback code. See the new debugging features in tamejs for some examples.

[paulmillr]

/subscribing to this

[benekastah]

@michaelficarra To clarify, are you proposing we compile coffeescript into tamejs, which would then be compiled by tamejs into vanilla javascript, or are you proposing we make our own tamejs-flavored language features that compile directly into js?

[michaelficarra]

@benekastah: I'm proposing we compile to JS-with-await-and-defer and optionally have the command-line tool automatically pass the output through tamejs when necessary. We could always pull the actual CPS compilation into coffeescript if it got popular/common enough, but I foresee two separate compilation steps if this gets accepted at all.

[mcoolin]

Flow control is becoming the norm on both the client side and server. It sure would be nice if coffeescript provided a simple way to implement async coding.

Looked over tamejs. Looks awesome. Been wrestling with a few others but was not happy with the results.

Sadly it appears to only run server side.

[smathy]

Hasn't this all been done before in #350 ?

[maxtaco]

Work in progress here:

https://github.com/maxtaco/coffee-script

I'll hope to have a pull request sometime soon to pick apart. Just met with @jashkenas who had some great suggestions.

[michaelficarra]

Awesome. Looking forward to it. How did @jashkenas feel about the ugliness of the CPS compilation? I believe that was the main argument holding this feature back.

[jashkenas]

Turns out @maxtaco works right around the corner ... we talked about the ugliness of the CPS transformation, but that wasn't the only thing holding back previous versions of defer ...

• It needs to compile into the minimal transformation required to handle the particular defer/await.
• It needs to handle defer within a loop, and break/return/continue.
• Code that does not use defer should not be affected.
• It needs to have syntax for both the sequential and parallel versions of multiple defers.
• The implementation can't be so extreme as to balloon the compiler's size, or to make it hard to work on.

... @maxtaco has already solved most of these. The defer/await split allows you to get either sequential or parallel behavior; he has a pretty clean addition to the AST (although it can still use a bit more cleaning up); special values for deferring in loops are only emitted when you actually defer in a loop; and so on.

I think that there's more ground to be covered to boil down the generated code to the minimum amount that we can lexically determine is necessary for the particular "defer" use -- but hopefully y'all (and maybe @gfxmonk as well) can help with that.

[markbates]

/subscribing to this thread

[weepy]

/s

[maxtaco]

Update: I moved over from master to the "tame" branch in my repo in preparation for the pull request. I added a fair amount of documentation to TAME.md. Before I submit, though, I thought we'd get some more experience using it in practice. We've already found a few a bugs, and maybe a few more will show up. More to come.

[smathy]

Is that a call to action? You want us to start testing this?