Skip to content

Commit 9633548

Browse files
committed
Refactors tokenizer to emit a Token double-linked-list
This refactors the tokenizer, simplifying both tokenizer and parser code slightly (fewer utility functions) and returns tokens as part of the AST's `loc` information. Tokens also have gained a few new properties: `line`/`column` for the 1-indexed position the token begins and `prev`/`next` to navigate the dll of Tokens. Finally, this introduces three new Token types: `<SOF>` for the empty token at beginning of the file stream, `,` for ignored comma tokens, and `Comment` for ignored comment tokens. These tokens are *included* in the Token list allowing for easier navigation of the parsed document in GraphQL tools and as a basis for parsing comment descriptions in the schema language.
1 parent 3cf66f3 commit 9633548

File tree

8 files changed

+905
-724
lines changed

8 files changed

+905
-724
lines changed

src/index.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ export {
121121
visitInParallel,
122122
visitWithTypeInfo,
123123
Kind,
124+
TokenKind,
124125
BREAK,
125126
} from './language';
126127

0 commit comments

Comments
 (0)