Bring back `SourceFile.EndOfFileToken` #1257

Andarist · 2025-06-21T13:01:20Z

No description provided.

internal/astnav/tokens_test.go

internal/parser/reparser.go

testdata/baselines/reference/astnav/GetTouchingPropertyName.mapCode.ts.baseline.txt

internal/parser/parser.go

_packages/api/src/node.ts

testdata/baselines/reference/submodule/conformance/jsDeclarationsDefault.js.diff

testdata/baselines/reference/submoduleAccepted/conformance/importTag14.errors.txt.diff

...eference/submoduleAccepted/conformance/jsDeclarationsFunctionPrototypeStatic.errors.txt.diff

sandersn

Still reviewing baselines, but here are some out of order comments.

sandersn · 2025-06-23T14:03:15Z

internal/astnav/tokens.go

@@ -49,6 +49,9 @@ func getTokenAtPosition(
 	left := 0

 	testNode := func(node *ast.Node) int {
+		if node.Kind == ast.KindEndOfFile {
+			return 0


how does this translate from the code in services/utilities.ts? I don't see a specific EOF case in that code so I assume it's simplified from it in some way.

This is a good question. It's an indirect port of the EOF case checked by nodeContainsPosition. The related code in Corsa is rewritten sligthly so it's not easy to see this. There is a chance this second nodeContainsPosition call might have to be ported too... but, so far, I have not found a need for it (at least not in the case of EOF):

// first element whose start position is before the input and whose end position is after or equal to the input if (nodeContainsPosition(children[middle], start, end)) { if (children[middle - 1]) { // we want the _first_ element that contains the position, so left-recur if the prior node also contains the position if (nodeContainsPosition(children[middle - 1])) { return Comparison.GreaterThan; } } return Comparison.EqualTo; }

I don't know either but I suspect that it's the cause of the difference in astnav baselines

internal/astnav/tokens_test.go

internal/parser/parser.go

internal/parser/reparser.go

sandersn · 2025-06-23T16:41:52Z

internal/parser/reparser.go

@@ -102,6 +106,9 @@ func (p *Parser) reparseUnhosted(tag *ast.Node, parent *ast.Node, jsDoc *ast.Nod
 	case ast.KindJSDocImportTag:
 		importTag := tag.AsJSDocImportTag()
 		importClause := importTag.ImportClause
+		if importClause == nil {


I think ModuleSpecifier is actually the critical property to require non-nil, although an import tag that imports no identifiers is pointless, so keep checking importClause for nil too.

I'm not sure if I understand correctly. Have you meant I should adjust it to this?

if importClause == nil || importTag.ModuleSpecifier == nil { break }

All of those work alright-ish right now:

// @checkJs: true // @allowJs: true // @filename: /foo.js /** * @import {A} from */ /** * @import from */ /** * @import */

So I wonder, if the proposed change is actually needed. The module specifier end up being an "empty identifier" (with an error flag set on it) in all three of those cases

Yes, if it works without the new code, please delete it.

sandersn · 2025-06-23T18:00:08Z

testdata/baselines/reference/astnav/GetTouchingPropertyName.mapCode.ts.baseline.txt

@@ -1,3 +1,39 @@
+〚Positions: [729, 730]〛
+【TS: SourceFile [0, 9772)】
+《Go: EndOfFileToken [9770, 9772)》


@andrewbranch I don't know how to read these baselines, but this could be the difference in GetTokenAtPosition that I was worried about.

Later: no, there's no jsdoc at the end; this is just a difference in GetTokenAtPosition or something similar between Strada and Corsa. Based on my reading of the diff--still not an expert--it looks like it is incorrect and Strada is correct.

I think the diff generator has a subtle bug where it’s not showing the actual positions where this is occurring, but all of these (729, 730, 780–783, etc.) are whitespace between top-level statements where Strada returns the SourceFile (seems right enough) and Corsa is returning the EOF token (seems wrong, since its text range is nowhere near the position being triggered).

sandersn

Requesting changes based on my previous comments, since this batch is just guesses for why diffs have happened.

sandersn · 2025-06-23T18:44:32Z

testdata/baselines/reference/astnav/GetTouchingPropertyName.mapCode.ts.baseline.txt

@@ -1,3 +1,39 @@
+〚Positions: [729, 730]〛
+【TS: SourceFile [0, 9772)】
+《Go: EndOfFileToken [9770, 9772)》


Later: no, there's no jsdoc at the end; this is just a difference in GetTokenAtPosition or something similar between Strada and Corsa. Based on my reading of the diff--still not an expert--it looks like it is incorrect and Strada is correct.

sandersn · 2025-06-23T18:48:22Z

testdata/baselines/reference/submoduleAccepted/compiler/uniqueSymbolJs.errors.txt.diff

this error seems more accurate than the Strada one, although it's missing the suggestion to use a mapped type.

This depends on t := c.getTypeFromTypeNode(typeNode). Corsa gets an unresolved any type (an error type) whereas Strada gets unique symbol type.

In what way, you'd like to unify this stuff?

ah, well, in that case the Strada type is more correct. I dn't think there's anything to fix for this PR, just a later fix--it sounds as if this might even be a difference between Corsa and Strada in TS as well.

testdata/baselines/reference/submoduleAccepted/conformance/callbackTag2.types.diff

testdata/baselines/reference/submoduleAccepted/conformance/importTag13.errors.txt.diff

...eference/submoduleAccepted/conformance/jsDeclarationsFunctionPrototypeStatic.errors.txt.diff

jakebailey · 2025-06-23T20:42:16Z

testdata/baselines/reference/submodule/conformance/jsDeclarationsDefaultsErr.js

 //// [index3.d.ts]
 // merge type alias and variable (behavior is borked, see #32366)
 declare const x = 12;
 export { x as default };
-export type default = string | number;
+export type _default = string | number;


Hm, but this is weird, no?

it is, the comment above this says the behavior is borked - and the type alias declaration here is the same in Strada:
https://github.com/microsoft/TypeScript/blob/78c16795cdee70b9d9f0f248b6dbb6ba50994a59/tests/baselines/reference/jsDeclarationsDefaultsErr.js#L81-L84

jakebailey · 2025-06-25T06:11:08Z

Seems like it needs more baseline updates.

Andarist · 2025-06-25T08:15:58Z

...baselines/reference/submoduleAccepted/conformance/jsDeclarationsUniqueSymbolUsage.types.diff


     return value;
 ->value : import("a").WithSymbol
-+>value : any
+>value : import("./a").WithSymbol


interestingly, this seems buggy in Strada - but this is already in the submoduleAccepted here so 🤷

https://github.com/microsoft/TypeScript/blob/e6a50a78619632bba7af4f23b2e7e4940b9cf788/tests/baselines/reference/jsDeclarationsUniqueSymbolUsage.types#L18-L31

This is a known difference across a lot of these baselines, yeah.

Note that "accepted" applies to basically all JS code right now given we're claiming it's "not complete", so it was helpful to ignore these diffs in other PRs just fixing type checking issues.

jakebailey · 2025-06-25T15:58:53Z

Still a test failure?

+--- old.partiallyNamedTuples2.types
++++ new.partiallyNamedTuples2.types
+@@= skipped -37, +37 lines =@@
+ const matches = x.get(id1);
+ >matches : Iterable<[id2: string, object]>
+ >x.get(id1) : Iterable<[id2: string, object]>
+->x.get : <Key extends [id1: string, id2: string] | [id1: string] | []>(...key: Key) => GetResult<[id1: string, id2: string], Key, object>
++>x.get : <Key extends [] | [id1: string] | [id1: string, id2: string]>(...key: Key) => GetResult<[id1: string, id2: string], Key, object>
+ >x : MultiKeyMap<[id1: string, id2: string], object>
+->get : <Key extends [id1: string, id2: string] | [id1: string] | []>(...key: Key) => GetResult<[id1: string, id2: string], Key, object>
++>get : <Key extends [] | [id1: string] | [id1: string, id2: string]>(...key: Key) => GetResult<[id1: string, id2: string], Key, object>
+ >id1 : string

Seems to have been a bad merge with main?

Andarist · 2025-06-25T16:03:40Z

weird, maybe I have forgotten to git add this one single baseline after syncing with main. But either way, this is still blocked on the GetTouchingPropertyName difference - I hope to fix this before Friday but we'll see how it goes

jakebailey · 2025-06-25T16:04:23Z

internal/transformers/declarations/transform.go

@@ -1767,3 +1772,12 @@ func (tx *DeclarationTransformer) transformJSDocOptionalType(input *ast.JSDocOpt
 	tx.EmitContext().SetOriginal(replacement, input.AsNode())
 	return replacement
 }
+
+func (tx *DeclarationTransformer) getGeneratedDefaultExportIdentifier() *ast.Node {


This function feels wrong to me; this means we're putting the same node into the tree multiple times, which is a no-no.

(I'm not 100% sure this fix is required for the rest of the PR, so would probably be worth a follow-up as a d.ts improvement?)

Ah, ye - I meant to ask if I should clone this before returning and forgot.

(I'm not 100% sure this fix is required for the rest of the PR, so would probably be worth a follow-up as a d.ts improvement?)

yes, it's a little bit out of scope of this PR. This has just popped up in the diffs after EOF got included. I'll remove it from this PR

Bring back SourceFile.EndOfFileToken

2d49b7c

Andarist commented Jun 21, 2025

View reviewed changes

internal/astnav/tokens_test.go Outdated Show resolved Hide resolved

Andarist commented Jun 21, 2025

View reviewed changes

internal/parser/reparser.go Outdated Show resolved Hide resolved

Andarist commented Jun 21, 2025

View reviewed changes

testdata/baselines/reference/astnav/GetTouchingPropertyName.mapCode.ts.baseline.txt Show resolved Hide resolved

Andarist added 5 commits June 21, 2025 22:15

update assert

2374111

regenerate fourslash test correctly

6607da5

add endOfFileToken getter

1571ba6

define childProperties for SourceFile

21b6a94

add endOfFileToken to the TS type

6210c99

Andarist mentioned this pull request Jun 22, 2025

Don't ignore completions at source file locations microsoft/TypeScript#61909

Merged

inclue reparsed nodes at eof

a7b21cf

Andarist commented Jun 22, 2025

View reviewed changes

internal/parser/parser.go Show resolved Hide resolved

Andarist commented Jun 22, 2025

View reviewed changes

_packages/api/src/node.ts Show resolved Hide resolved

Andarist marked this pull request as ready for review June 22, 2025 10:02

sandersn self-requested a review June 23, 2025 13:54

sandersn self-assigned this Jun 23, 2025

Andrew magic

9f798cf

jakebailey reviewed Jun 23, 2025

View reviewed changes

testdata/baselines/reference/submodule/conformance/jsDeclarationsDefault.js.diff Outdated Show resolved Hide resolved

jakebailey reviewed Jun 23, 2025

View reviewed changes

testdata/baselines/reference/submoduleAccepted/conformance/importTag14.errors.txt.diff Show resolved Hide resolved

jakebailey reviewed Jun 23, 2025

View reviewed changes

...eference/submoduleAccepted/conformance/jsDeclarationsFunctionPrototypeStatic.errors.txt.diff Show resolved Hide resolved

sandersn reviewed Jun 23, 2025

View reviewed changes

sandersn requested changes Jun 23, 2025

View reviewed changes

Andarist added 3 commits June 23, 2025 21:19

address some CR comments

7e25056

Merge remote-tracking branch 'origin/main' into eof

603ed8a

fixed the default export merge

7feb9e7

jakebailey reviewed Jun 23, 2025

View reviewed changes

Andarist added 2 commits June 24, 2025 12:34

Merge remote-tracking branch 'origin/main' into eof

b44dc32

add extra test case

5bb73c4

Andarist force-pushed the eof branch from ab5e83b to f20c7a0 Compare June 25, 2025 08:12

Andarist commented Jun 25, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into eof

705ec33

Andarist force-pushed the eof branch from f20c7a0 to 705ec33 Compare June 25, 2025 16:01

jakebailey reviewed Jun 25, 2025

View reviewed changes

Bring back SourceFile.EndOfFileToken #1257

Are you sure you want to change the base?

Bring back SourceFile.EndOfFileToken #1257

Conversation

Andarist commented Jun 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sandersn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andarist Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandersn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakebailey commented Jun 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakebailey commented Jun 25, 2025

Uh oh!

Andarist commented Jun 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Bring back `SourceFile.EndOfFileToken` #1257

Bring back `SourceFile.EndOfFileToken` #1257

Andarist Jun 24, 2025 •

edited

Loading