es5.d.ts lib does not match the ECMAScript 5.1 Spec

[ericdrobinson]

Types in es5.d.ts Should Match the ECMAScript 5.1 Standard

As a TypeScript user, I expect that an environment configured with "lib": ["es5"] should only resolve types specified in the ECMAScript 5.1 Language Specifications. Any further type declarations necessary should be optionally imported or specified by the TypeScript user. Indeed, this appears to be the suggested usage, as pointed out by @mhegazy in #1995 (emphasis mine):

you can always augment your program with additional types that you know would exist on your target system. for instance, if you know you are always running on an engine that supports promises, just add declarations for Promise in your program. The idea is that the lib file reflects the baseline of engines supporting the target of the compilation (ES5 v.s. ES6)

However, including only the "es5" lib will also bring in the following non-ES5 types (and their support types):

• Decorators - Introduced when es5.d.ts was core.d.ts (2015/03/17).
• PromiseLike - This was pulled into core.d.ts from es6.d.ts on 2015/05/05.
• ArrayBuffer and TypedArrays - This was merged into core.d.ts from es6.d.ts and extension.d.ts on 2015/08/15.
• Promise<T> - This was pulled into es5.d.ts from es2015.promise.d.ts on 2016/02/13.
• Several others not listed...

Including these types breaks expectations in properly initialized environments and has the potential to lead to confusion, if not bugs.

Why Are These Types In es5.d.ts to Begin With?

In short: to support the DOM API declarations.

From what I've been able to glean from the history, the fact that the above type declarations exist in es5.d.ts is effectively a kludge to more simply support the constantly-evolving DOM API libraries. From the comments on #14053 (which brought Promise<T> into es5.d.ts):

Move the declaration of the Promise interface to es5.d.ts (and subsequently lib.d.ts). This allows for DOM APIs that return Promise to be typed accurately.

Indeed, supporting DOM APIs appears to explain much of the reason for including the previously listed types in es5.d.ts. Again, in #16077 we have the following comment:

The sole purpose of including Promise and Typed Arrays to prevent errors from lib.dom.d.ts where APIs e.g. Fetch and Web Audio use them.

[It should be noted that in the above case, the discussion revolves around whether or not to move the types further back in time (to ES3).]

Moving "Support" Types Out of es5.d.ts

So the types are defined in the ES libs to support the DOM APIs (which suffer from granularity issues). Here are some options for how they might be removed from the es5.d.ts file:

1. Create a Middleman Library

It appears that ES5 is used as the default ECMAScript library for TypeScript - at some point es5.d.ts was even called core.d.ts. But a core.d.ts could still exist: it could import es5.d.ts and then some extra "support" types (e.g. those outlined above). That file could then be specified in the lib.d.ts library source map (as well as others).

2. Specify Necessary Features Directly in the Library Source Map

Specific features that DOM APIs require might be included from their respective "future" libraries. An example for promises might look like:

    // ...
    { target: "lib.dom.d.ts", sources: ["header.d.ts", "dom.generated.d.ts"].concat("es2015.promise.d.ts") },
    // ...
    { target: "lib.d.ts", sources: ["header.d.ts", "es5.d.ts"].concat(hostsLibrarySources, "es2015.promise.d.ts") },
    // ...

provided Promise type interfaces were moved back into es2015.promise.d.ts (unless I'm missing something important).

3. Declare Necessary Interfaces Inline

Declare required, but possibly new/future types as empty interfaces inline in the libraries that require it. In at least one Node Type Declaration library I've encountered, they did just this:

interface MapConstructor { }
interface WeakMapConstructor { }
interface SetConstructor { }
interface WeakSetConstructor { }

What's more, this even seems to be a suggested approach (emphasis mine):

you can always augment your program with additional types that you know would exist on your target system. for instance, if you know you are always running on an engine that supports promises, just add declarations for Promise in your program. The idea is that the lib file reflects the baseline of engines supporting the target of the compilation (ES5 v.s. ES6)

Wrap-up: Being Explicit About Type Definition

Instituting a change like this would not only clean up the ECMAScript type declaration file(s), it would explicitly document why those types were included with certain libraries. Presently, a user trying to figure out why UInt8Array is a suggested type when they've only imported type declarations for ECMAScript 5 ("lib": ["es5"]) will find that it's confusingly included in the es5.d.ts file, sans any documentation for its mysterious inclusion.

Further, it would decouple DOM support from ECMAScript support, enabling users to more explicitly configure their environments without surprising side-effects.

[saschanaz]

I think the empty interfaces or the inclusion of Promise in lib.dom.d.ts are good, but they do not remove your problem with incorrect suggested type, right? Because by default DOM types are included and then you will still get Promise type by default.

[ericdrobinson]

@saschanaz

but they do not remove your problem with incorrect suggested type, right? Because by default DOM types are included and then you will still get Promise type by default.

This issue is not at all concerned with the default and explicitly states as much:

Including these types breaks expectations in properly initialized environments

The DOM is included by default, yes, as it is extremely common to write frontend (browser) code. However, it is possible to customize the environment such that DOM APIs are removed (e.g. "lib": ["es5"]). Node environments are just such a place (and old versions of Node, like some legacy browsers, did not have Promise support).

The basic issue is that when I specify "ECMAScript 5" as my environment, I do not expect to see Promises, TypedArrays, etc. In the case of #16077, pushing them into ES3 actually moves those types farther away from the spec in which they were actually introduced (ECMAScript 6).

As for the default, one might argue based on the current ES6 API coverage in browsers that it should be updated to point to ECMAScript 6. But that is a different issue.

[BurtHarris]

It seems there are no defined semantics associated with use of the --lib switch/config entry, so exactly what --lib es5 should do or include seems arguable. I can't be alone in knowing of and depending on some of the definitions that this issue proposes should be removed, and such dependencies expand every day (see for example comment on this PR). Making a breaking change that would impact others to match some inferred definition of --lib es5 semantics seems unwise.

As a TypeScript user, I expect that an environment configured with "lib": ["es5"] should only resolve types specified in the ECMAScript 5.1 Language Specifications. Any further type declarations necessary should be optionally imported or specified by the TypeScript user. Indeed, this appears to be the suggested usage, as pointed out by @mhegazy in #1995 (emphasis mine):

I think this over-interprets the comment made in #1995, which says "you can" rather than "you should". That comment seems to have been meant as a helpful how-to workaround, not a definition of philosophy of the library files.

[ericdrobinson]

It seems there are no defined semantics associated with use of the --lib switch/config entry, so exactly what --lib es5 should do or include seems arguable.

Do you really think so? You don't think that someone explicitly saying "I want only es5 types" is expecting to see Decorators (a feature which is still in proposal mode)? As a TypeScript user unfamiliar with the internals, would I not be surprised to see Promises and Typed Arrays show up when I attempt to limit the environment to ECMAScript 5.1 built-ins?

I can't be alone in knowing of and depending on some of the definitions that this issue proposes should be removed, and such dependencies expand every day (see for example comment on this PR).

You most assuredly aren't! And I address the "expanding every day" issue in the initial post. If you look at the timeline of those features, you'll see that the es5 library "evolved" out of a core.d.ts file. The comment you linked to doesn't show "Decorators entering the es5 lib", it simply refers to the Decorator type definitions in ES5 after that switch. You can see where those definitions initially came in here, back when the es5 library was known as "core". It should be noted that this all happened before or as TypeScript gained the ability to specify specific libraries for the environment to include by default. The thing is that the ES5 library was never fully cleaned up - it was too difficult at the time to properly pull things apart such that ES5 could be, well, spec-compliant.

But things have improved dramatically since then.

Making a breaking change that would impact others to match some inferred definition of --lib es5 semantics seems unwise.

What are you implying by stating "inferred definition of --lib es5"? Almost every single other library you can specify attempts to match the spec it refers to. The es5 lib is a special case due to it's evolution. This Issue simply calls this out as being odd and something that should be fixed. The es5 standard (ECMAScript 5.1) is well-defined.

I would highly suggest taking a look at the discussion #16077 (and especially the comments starting here. The TL;DR of the discussions there is that work has been done to create an es3 library and the initial approach was to simply move some types (e.g. Typed Arrays) out of es5 into es3. The current working approach is to break out the specific features called out at the top of this issue into their own libraries. With this approach, the default setup would be protected, the language type libraries would actually match the specs, and certain features could be added as needed. Users who currently rely on all features included in --lib es5 would simply add a few extra library references as needed (e.g. es2015.promise or such).

I think this over-interprets the comment made in #1995, which says "you can" rather than "you should". That comment seems to have been meant as a helpful how-to workaround, not a definition of philosophy of the library files.

The portion of the quote that you're referring to was included for context. The portion that does speak towards "philosophy of the library files" is the portion that I emphasized in bold. To reiterate, that is specifically:

The idea is that the lib file reflects the baseline of engines supporting the target of the compilation (ES5 v.s. ES6)

What, then, are Decorators doing in ES5? They haven't even made it out of Draft stage and aren't supported in any current engines (beyond TypeScript itself).

[BurtHarris]

@ericdrobinson, Who says that --lib es5 means "I want only es5 types"? It could mean "I expect the generated code to run on an es5 engine". That's what I meant by an 'inferred definition' of --lib es5.

Of course it --lib es5 shouldn't enable decorators and it doesn't! It just includes some supporting typedefs, which are seem highly unlikely to break anyone's code who's not using decorators. While I agree it may not not conceptually pure, from my point-of-view its not worth making a breaking change over, especially one that isn't be supported by a typical deprecation period.

The statement that "lib file reflects the baseline of engines supporting the target of the compilation" isn't really applicable to the decorator support types anyway, decorators don't need JavaScript engine support. They are handled by TypeScript, and thus decorators can be made to work on es5 engines, if the developer chooses to use them (which is controlled by a separate switch.)

I'm painfully aware that decorators haven't stabilized yet, but that's an entirely separate discussion.

[ericdrobinson]

Who says that --lib es5 means "I want only es5 types"? It could mean "I expect the generated code to run on an es5 engine". That's what I meant by an 'inferred definition' of --lib es5.

@BurtHarris Wait, wait, wait, wait, wait. You're talking about code generation here, yes? That implies you don't mean to talk about the lib switch, but the target one, right? See:

--lib: List of library files to be included in the compilation.
--target: Specify ECMAScript target version: "ES3" (default), "ES5", "ES6"/"ES2015", "ES2016", "ES2017" or "ESNext".

Basically, --lib is "I need the types listed here included in my environment so that the types I use resolve correctly" and --target is "I need the emitted JavaScript to run on engines with this level of ECMAScript support." With the--lib flag you are describing the features that you plan to use in the code that you write, not what should be output to run on a given engine. That's the --target flag's job.

Note further that --lib is a list. You can specify multiple options. The proposed fix for this simply breaks out the non-ES5 types into separate options that you could add to that list.

Apologies if you were already aware of this. I just want to make sure we're on the same page. Your example above talks about expectations of an actual "target running engine". Substituting other --lib options for "es5" in your example (e.g. ES2015.Symbol) starts to break down. I suspect that anyone who sees the list of options (and the description of the --lib flag) will quickly understand that this not about [lib]-engine, but rather [lib]-features. This is how it worked for me.

Of course it --lib es5 shouldn't enable decorators and it doesn't! It just includes some supporting typedefs, which are seem highly unlikely to break anyone's code who's not using decorators. While I agree it may not not conceptually pure, from my point-of-view its not worth making a breaking change over, especially one that isn't be supported by a typical deprecation period.

To be clear, the breaking change you're concerned about here is that once you upgrade to a newer version of TypeScript you'd have to add an extra option or two to the --lib flag (if you even specified it to begin with), correct?

The issue is that having those types show up in auto-correct/IntelliSense can be confusing for people who specifically say "I want ES5-features only, thank you very much." And for less-obvious features, seeing such types may send people on a bit of a wild goose-chase to figure out if there's an option there they can use. Such [repeated] experiences prompted the creation of this very issue.

The statement that "lib file reflects the baseline of engines supporting the target of the compilation" isn't really applicable to the decorator support types anyway, decorators don't need JavaScript engine support. They are handled by TypeScript, and thus decorators can be made to work on es5 engines, if the developer chooses to use them (which is controlled by a separate switch.)

Right. This is where you would specify --target es5, right? If you specify --lib es5, the aforementioned change would simply require that you specify --lib es5,proposals.decorators or the like. The --target would remain unchanged and your emitted JavaScript would work without issue.

It should be noted that it appears as though @rbuckton's current proposal would potentially take "several releases" to fully realize... If your fear is related to a deprecation period, this may actually already be under consideration...

[BurtHarris]

Wait, wait, wait, wait, wait. You're talking about code generation here, yes? That implies you don't mean to talk about the lib switch, but the target one, right?

No @ericdrobinson, I'm not talking about code generation. I'm talking about the ambient definitions assumed present by TypeScript without explicit import statements. This has little to do with which set of parse tree transforms are applied prior to emitting the .js file. The impact of --target when --lib is not specified is documented on the compiler options page, but you seem to be conflating the two.

Basically, --lib is "I need the types listed here included in my environment so that the types I use resolve correctly" and --target is "I need the emitted JavaScript to run on engines with this level of ECMAScript support." With the--lib flag you are describing the features that you plan to use in the code that you write, not what should be output to run on a given engine. That's the --target flag's job.

The issue is that having those types show up in auto-correct/IntelliSense can be confusing for people who specifically say "I want ES5-features only, thank you very much." ...

Here we go again, you writing quoted phrases, and I'm asking where you are quoting from. It looks to me like is that you are taking the definitions you have inferred and treating them as if they are somehow documented intent of the language designers. Now you are citing a PR comment by @rbuckton containing the word "perhaps" as a his current proposal. I think that stretches credibility, especially after he's explicitly told you "You either misread or misquoted me" in that same PR. Please think carefully about the lengths you seem to be going to in order to justify your point of view... you might start by assuming people mean what they say.

The documented meaning of --lib is "List of library files to be included in the compilation"; no more, no less. The intent of each file is best defined by its contents, though the documentation could certainly be improved. But a little lack of documentation is no reason to introduce a breaking change.

To be clear, the breaking change you're concerned about here is that once you upgrade to a newer version of TypeScript you'd have to add an extra option or two to the --lib flag (if you even specified it to begin with), correct?

Yes, that's mostly correct. That is a breaking change. Imposing your interpretations of intent could cause other innocent component to fail. But you are incorrect about it only applying if I've specified a --lib, due to the well documented interaction with --target.

If you really want purity of the ambient namespace, perhaps you should be using the --nolib option, rather than proposing changes that assume everyone else will change their point of view. The --nolib option's documented meaning is "Do not include the default library file" which seems to handle the situation perfectly well, and orthogonally to --target.

[BurtHarris]

I suggest this issue should be closed as by-design.

[saschanaz]

@BurtHarris I think the TS team agrees that we should bring non-ES5 types out. #16077 (comment) #16077 (comment)

[ericdrobinson]

No @ericdrobinson, I'm not talking about code generation. I'm talking about the ambient definitions assumed present by TypeScript without explicit import statements.

Understood. I was clarifying because the statement you used in your example specifically mentioned the engine the code would run in. This was confusing in the moment. I was just trying to make sure we were on the same page. Glad it's cleared up!

The impact of --target when --lib is not specified is documented on the compiler options page, but you seem to be conflating the two.

I am aware of that but perhaps wasn't being as explicit as I should have been. In the discussion on #16077, special care was given to ensure that lib.d.ts (at least) would not be affected by the migration of non-es5 types into their own libraries. I would assume that such a change would also adjust the default libraries injected "when --lib is not specified" such that simply specifying --target es5 or --target es6 would result in no change for the end user.

This still leaves the changes necessary for people simply specifying --lib es5 alone, of course. I'll respond to this below.

Here we go again, you writing quoted phrases, and I'm asking where you are quoting from. What it looks to me like is that you are taking the definitions you have inferred and treating them as if they are somehow documented intent of the language designers.

Hmm... The quotes in the blocks of text that you cite are paraphrases/interpretations - they are not specific quotes. Most of the time I use Markdown Quote formatting when I'm specifically referencing someone or a specific page. The exception to this is the handful of times that I use them to link to a sentence fragment in a previous comment.

In the first block you cite, for example, the quotes are reinterpretations of the [honestly rather thin and unclear] descriptions from the Compiler Options definitions that I directly quoted immediately prior.

Now you are citing a comment by @rbuckton containing the word "perhaps" as a his current proposal, which I think stretches credibility, especially after he's explicitly told you "You either misread or misquoted me" in that same PR. Please think carefully about the lengths you seem to be going to in order to justify your point of view...

I'm not really sure how best to respond to this. I'm not trying to suggest that the status was 'official proposal', only that 'one option has been proposed.' It's simply the latest idea on how it might be effected - one proposal as to how it might be implemented.

Also, if you follow the discussion, the "perhaps" that I quote wherein an option is presented actually follows the "You either misread or misquoted me" quote that you cite. I responded directly to that statement in a follow-up comment to which he then responded in-kind with "I think I see what you are getting at. Perhaps the best breakdown would be..." In short, there was a bit of mutual-misunderstanding. Once that was cleared up, however, the most recent option/proposal (to which I linked) was put forth. It fell out of the constructive discussion.

To be clear, the breaking change you're concerned about here is that once you upgrade to a newer version of TypeScript you'd have to add an extra option or two to the --lib flag (if you even specified it to begin with), correct?

Yes, that's correct

Great! That is valuable feedback! Perhaps a suggestion that would work around all of this would be to add another option to the --lib list called pure.es5. The library loaded when you set --lib es5 could then be built up of pure.es5 and the other potentially broken-out types. That would allow all settings that work today to remain unchanged and avoid any breakage.

If you really want purity of the ambient namespace, perhaps you should be using the --nolib option, rather than proposing changes that assume everyone else will change their point of view. That option's documented meaning is "Do not include the default library file" which seems to handle the situation perfectly well.

Well, but specifying --lib overrides the default library as well. I'm not interested in an empty ambient namespace, just one that fully conforms to ECMAScript 5.1 (and ECMAScript 3, in the case of the upcoming es3.d.ts), with no surprises.

[BurtHarris]

You can accomplish that using --nolib (or using /// <reference no-default-lib="true"/>) by copying and editing the es5 default lib file into your project, naming it pure.es5.1.ts if you want, and explicitly importing it into your modules. You can even publish your definitions on npmjs.com. That way you get what you want and can share it without having an impact on other modules.

[RyanCavanaugh]

From what I've been able to glean from the history, the fact that the above type declarations exist in es5.d.ts is effectively a kludge to more simply support the constantly-evolving DOM API libraries.

This is correct. The DOM doesn't follow the same versioning scheme as ECMAScript, so we're effectively trying to map an N-dimensional versioning space into a single dimension, which is definitely lossy.

My main concern here is that this is a cure that is potentially much worse than the disease. We still don't actually bring in the values of the 'future' runtimes into scope, so you can't really write expression code that tries to access a non-existent value. And arguably it's not actually incorrect to use e.g. an ES6 type in an ES5 codebase -- if you write class Foo implements Promise { you haven't actually done anything wrong, even in an ES3 codebase.

This gets really confusing if you consider libraries that work perfectly in ES3/ES5 environments, but still understand ES6 constructs if given them. If a library knows how to call .then on a Promise when it sees one, but never tries to instantiate the Promise itself, is it wrong to say function fn(x: Promise | string) { in the context of an ES3 program? Note that the solution here cannot be "write interface Promise { }" because now that function has no type safety whatsoever! So you tell ES3 users of this library that they have to include lib.promise.d.ts... which brings in the value, which is the very thing we should be most afraid of, and is already solved by the current solution? That's a regression, not an improvement.

[ericdrobinson]

You can accomplish that using --nolib by copying and editing the es5 default lib file into your project, naming it pure.es5.ts if you want, and explicitly importing it into your modules. That way you get what you want without having an impact on other modules.

Sure. But maintaining that across multiple projects is a headache and would imply extraneous upkeep costs as enhancements/fixes/changes continue in the root es5 default lib. Further, it leaves the issues raised in the last section of the initial post unaddressed.