Fix implicit loop closure, support `do (x) ->`

[TrevorBurnham]

This is an attempt to tie together a few recent issues and come to a consensus on the current feature of closing-over variables in a loop.

The purpose of the feature

Originally proposed in issue 423, the feature is nicely illustrated by this test case:

for i in [1, 2]
  setTimeout (-> console.log i), 0

If converted straightforwardly to JavaScript, this will generate the output

2
2

Why? Because the closure doesn't get invoked until after the loop has finished executing, by which point i is 2. The closures always references the same i.

But because of this feature, you get the output

1
2

Now the closure references the loop's indices (and other variables with "loop scope"—see issue 948) as they are when the closure is defined. This is implemented using a function called from each loop step:

var i, _fn, _i, _len, _ref;
_ref = [1, 2];
_fn = function(i) {
  setTimeout((function() {
    return console.log(i);
  }), 0);
};
for (_i = 0, _len = _ref.length; _i < _len; _i++) {
  i = _ref[_i];
  _fn(i);
}

Inconsistencies resulting from this feature

Unfortunately, there are several problems with this implementation, most notably issue 954: The transformation doesn't work if there's a break, continue or return anywhere in the loop. So the code

for i in [1, 2]
  setTimeout (-> console.log i), 0
  return if false

gives you

2
2

This is highly unintuitive: Why would the addition of a control statement affect the code's output?

Another inconsistency, issue 950, is that loop indices are immutable when this transformation occurs (that is, when there's a closure and no control statements in the loop), but are mutable otherwise.

Yet another inconsistency, pointed out by satyr at issue 728, is that arguments won't work within the for loop if the transformation occurs: It will reference the arguments of _fn rather than the arguments of whatever function the for loop exists in.

A more sophisticated implementation could correct these problems, but it would not be simple or elegant, and it's certainly not going to happen in the next few days.

Philosophical problems with this feature

Even if it weren't for the inconsistencies pointed out above, would we want this feature? It's a pretty dramatic transformation on the CoffeeScript compiler's part, and makes an unintuitive leap from JavaScript's semantics.

While it may be the case that you usually want a closure in a for loop to reference surrounding variables as they were at the time the closure was declined, this isn't the way that closures behave anywhere else in the language.

One of my favorite things about JavaScript is that only functions have scope. It's a nice, easy, consistent rule, one that lets you easily determine scope in CoffeeScript by looking for -> and => (and class). There have been several attempts at giving loops special scope in CoffeeScript, but all have proven problematic. I think loops should be kept simple.

Alternatives

Let's look at that original example again. How do we say "let's preserve the value of i when the closure is invoked"? The simplest approach is to write our own explicit closure,

for i in [1, 2]
  ((i) ->
    setTimeout (-> console.log i), 0
  )(i)

Of course, the syntax here is terrible. There was a better syntax that was proposed at issue 788 and had fairly wide support, not to mention a working implementation courtesy of satyr. If a proposed variation on this syntax were added to master, then you could write

for i in [1, 2]
  do (i) ->
    setTimeout (-> console.log i), 0

where do (i) -> ... is shorthand for do (i = i) -> ..., which in turn is shorthand for the current ((i) -> ...)(i).

This, I believe, is the perfect solution: A simple, succinct way of explicitly stating which variables you want to capture. This is a much better fit with CoffeeScript's philosophy of letting you write syntactically elegant code with JavaScript's semantics.

Conclusion

The implicit loop closure that's been in CoffeeScript since issue 423 should be removed due to the unpleasant surprises listed above. Even a perfect implementation would not, in my view, be worth the increased distance between the CoffeeScript and the underlying JavaScript. And the do keyword, rejected as insufficiently practical, should be reconsidered as an elegant way of capturing variables.

[geraldalewis]

Nice writeup, Trevor. I was having trouble teasing apart the issue before this.

I'm in favor of the alternative approach you outlined. I'm comfortable with my control structures not closing over my variables, as I'm sure many other JS devs are, and I'd be more confused by the existing behavior. There being a syntactic way to semi-simulate block level scope seems like the best of both worlds.

I'm curious, though: why overload the do keyword? It's a pretty, two-letter word, but maybe doesn't communicate its intent clearly.

[TrevorBurnham]

@geraldalewis I don't want to get too deep into the issue 788 discussion, but in my opinion, do (which is reserved but unused by CoffeeScript) is a pretty readable way of saying "run the following function, passing in these arguments." Maybe run or call would better communicate the intent than do, but that would mean reserving new keywords.

The full form is do (x = value) ->, but do (x) -> is shorthand for do (x = x) ->; that's how do (i) -> above, both declares a function argument named i and passes in i to the function.

[geraldalewis]

@TrevorBurnham - ahh -- I'd missed that do was unused in CoffeeScript; I don't use do regularly. Makes total sense now. Thanks for the explanation :)

[satyr]

passing in these arguments

This isn't a part of the proposal (nor the current implementation in Coco).

[TrevorBurnham]

@satyr Any particular reason? I'd suggested the shorthand at issue 788.

[satyr]

Any particular reason?

The purpose of do in that proposal is to write less parentheses--nothing more.

[TrevorBurnham]

@satyr Thanks, I've clarified the use of do in the original post and raised a Coco issue. (Update: And now that Jeremy has added your implementation of do to master, I've copied that issue over to CoffeeScript as issue 960.)

[odf]

I don't have any philosophical problems whatsoever with introducing block scope. In my eyes, it's less of a deviation from Javascript then the compulsive list comprehensions. But since obviously it's really hard to get this right, I agree that it would be better to drop the feature in favour of simplicity.

I think I like the idea of the do statement, which reminds me of the let statement in Lisp, ML and probably a bunch of other languages. It clearly communicates the idea that a new scope is created with some bindings and then a block of code is executed within that scope.

I'm wondering about the best syntax, though. Should the arrow be mandatory? What about the argument list. For example, one could imagine
for i in [1..3] do
setTimeout (-> console.log i), 0
to be shorthand for
for i in [1..3]
do (i = i) ->
setTimeout (-> console.log i), 0
which in turn would expand to
for i in [1..3](%28i%29 -> setTimeout %28-> console.log i%29, 0)(i)
Is this too confusing, going too far? Maybe the arrow should be mandatory in order to indicate that a function is created. But the special form for i in [1..3] do ->, in which do would expand to do(i = i) as a more explicit way of getting the current behaviour back might be useful.

Another question I'm asking myself is whether to use let as the keyword instead of do, which would immediately be familiar to Lispers. I don't see why do has to be tagged indefinitely as a keyword if it's not used, so the argument about not wanting to introduce another keyword does not quite convince me.

[TrevorBurnham]

@odf Good points. I think that the proposed for...do, while appealingly succinct, suffers because it would make break, continue and return fail to act as you'd expect them to in a loop setting (not to mention arguments). By making the variable capture construct an explicit function, you can continue to interact with the loop in ordinary ways outside of it.

As to let, I'm not a Lisper myself, but do f as a synonym for f() makes more sense to me than let f... besides, do has to be tagged as a keyword because it's not a valid variable name in JavaScript.

[odf]

@TrevorBurnham Yes, I think I agree that the for...do form would be too confusing. But what about for...do ->? I'm not sure myself, honestly, but it does have some appeal.

I didn't realise that about do in JavaScript, but I still don't feel the fact that it's already tagged makes for a strong argument. I think it's a good word for expressing what it would do in this proposal, so I'm all for it. On the other hand, it might confuse people coming from languages where do is just syntactic glue introducing a loop body.

[TrevorBurnham]

Yes, I suppose for ... do -> would be fine, with do -> being shorthand for do (i) or do (key, val) or whatever.

It's true that do is almost always paired with until, but I don't see that as a big problem. I guess I wouldn't really care if let were used instead.

[jashkenas]

Thanks for detailing this issue, Trevor. On master, the current situation is this:

• The implicit-closure magic is gone, and loops are just loops, regardless of if they're generating functions or not.

• There's a new syntax for "scoped functions", that act like the closure-wrapped functions, or a forEach loop. You write it like for variables in object -> For example, one of the test cases:

  obj = {}
  
  for method in ['one', 'two', 'three'] ->
    obj[method] = ->
      "I'm " + method
  
  ok obj.one()   is "I'm one"
  ok obj.two()   is "I'm two"
  ok obj.three() is "I'm three"
  

[TrevorBurnham]

I'm fine with the scoped for syntax, as an extra sugar for a common use case (and I'm glad it uses an explicit ->), but I really, really want the enhanced do syntax. Soooo badly. Because sometimes, you want to explicitly enclose some variables but not other.

The do syntax I suggested at issue 960 is so much more readable than the mess of parentheses currently required to define and call a closure in-place. It's simple, versatile, and free of pitfalls.

[odf]

I like the for...-> very much. Explicit is better than implicit, as they say in Python land.

Like TrevorBurnham, I'm still very much in favour of the proposed do statement, tough. It's a clean syntactical encapsulation of a common use-case that is otherwise tedious to express. Having the do would make code that needs to close over certain values but not others DRYer and more readable.

[jashkenas]

I've talked about this a bit with Trevor out-of-band, but it's currently my opinion that loops "that need to close over certain values but not others" are an anti-pattern. Those loops can always be written in a clearer, more structured way, and it's pretty nasty to have to check a declaration to determine why some variables are being shared across iterations, and some variables aren't. If we can keep you from having to bother dealing with that whole can of worms, in CoffeeScript, I think we'd be better off in the end.