Skip to content

Commit 381df6b

Browse files
committed
Refactors tokenizer to emit a Token double-linked-list
This refactors the tokenizer, simplifying both tokenizer and parser code slightly (fewer utility functions) and returns tokens as part of the AST's `loc` information. Tokens also have gained a few new properties: `line`/`column` for the 1-indexed position the token begins and `prev`/`next` to navigate the dll of Tokens. Finally, this introduces two new Token types: `<SOF>` for the empty token at beginning of the file stream, and `Comment` for ignored comment tokens. These tokens are *included* in the Token list allowing for easier navigation of the parsed document in GraphQL tools and as a basis for parsing comment descriptions in the schema language. Providing tokens as part of each AST node's `loc` provides a linkage between the abstract AST and the concrete token list which provides more useful information to GraphQL language tools. Thanks to the addition of the `<SOF>` token, The AST `Document` node returned from `parse()` now definitionally points to `<SOF>` as its `startToken` and `<EOF>` as its `endToken`, effectively the head and tail of the token double linked list. This also removes the `noSource` parser option in favor of `source` being non-enumerable (the new `startToken` and `endToken` properties are also non-enumerable) so that they do not appear in JSON.stringify/util.inspect.
1 parent 3cf66f3 commit 381df6b

File tree

8 files changed

+906
-724
lines changed

8 files changed

+906
-724
lines changed

src/index.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ export {
121121
visitInParallel,
122122
visitWithTypeInfo,
123123
Kind,
124+
TokenKind,
124125
BREAK,
125126
} from './language';
126127

0 commit comments

Comments
 (0)