Remove remaining references to gyb

Author: ahoppen

CodeGeneration/README.md

@@ -2,18 +2,6 @@
 
 This directory contains file to generate source code that is part of the SwiftSyntax package. If you are looking to generate Swift code yourself, you might be interested in [SwiftSyntaxBuilder](../Sources/SwiftSyntaxBuilder).
 
-SwiftSyntax currently uses two mechanisms to generate source code: gyb and SwiftSyntaxBuilder. Both can be run using 
-
-```bash
-./build-script.py generate-source-code --toolchain /path/to/toolchain.xctoolchain/usr
-```
-
-## `gyb`
-
-Files generated using `gyb` have a `.gyb` suffix. [ChangingSwiftSyntax.md](../Sources/SwiftSyntax/Documentation.docc/ChangingSwiftSyntax.md) contains documentation how `gyb` works. This directory contains the definitions that back most of gyb-generation in `gyb_syntax_support`.
-
-## `SwiftSyntaxBuilder`
-
 Some source code inside SwiftSyntax is generated using [SwiftSyntaxBuilder](../Sources/SwiftSyntaxBuilder), a Swift library whose purpose is to generate Swift code using Swift itself. This kind of code generation is performed by the Swift package defined in this directory.
 
-This directory is a standalone package that uses a pinned version of SwiftSyntaxBuilder. It is thus NOT using SwiftSyntaxBuilder of the parent directory. This guarantees that when `generate-swiftsyntaxbuilder` is run, it can't break its own build.
+This directory is a standalone package that uses a pinned version of SwiftSyntaxBuilder. It is thus NOT using SwiftSyntaxBuilder of the parent directory. This guarantees that when `generate-swiftsyntaxbuilder` is run, it can't break its own build.

CodeGeneration/Sources/Utils/SyntaxBuildableType.swift

@@ -26,7 +26,7 @@ public struct SyntaxBuildableType: Hashable {
   public init(syntaxKind: String, isOptional: Bool = false) {
     self.isOptional = isOptional
     if syntaxKind.hasSuffix("Token") {
-      // There are different token kinds but all of them are represented by `Token` in the Swift source (see `kind_to_type` in `gyb_syntax_support`).
+      // There are different token kinds but all of them are represented by `Token` in the Swift source (see `kindToType` in `SyntaxSupport/Utils.swift`).
       self.syntaxKind = "Token"
       self.tokenKind = syntaxKind
     } else {

CodeGeneration/Sources/Utils/Utils.swift

@@ -22,7 +22,7 @@ public func dedented(string: String) -> String {
 }
 
 /// Creates a single-line documentation string from indented
-/// documentation as written in `gyb_syntax_support`.
+/// documentation as written in `CodeGeneration`.
 public func flattened(indentedDocumentation: String) -> String {
   dedented(string: indentedDocumentation)
     .replacingOccurrences(of: "\n", with: "")

CodeGeneration/Sources/generate-swiftsyntax/templates/swiftsyntax/SyntaxNodeFile.swift

@@ -15,7 +15,7 @@ import SwiftSyntaxBuilder
 import SyntaxSupport
 import Utils
 
-/// This gyb-file generates the syntax nodes for SwiftSyntax. To keep the generated
+/// This file generates the syntax nodes for SwiftSyntax. To keep the generated
 /// files at a managable file size, it is to be invoked multiple times with the
 /// variable `emitKind` set to a base kind listed in
 /// It then only emits those syntax nodes whose base kind are that specified kind.

Sources/SwiftParser/README.md

@@ -27,44 +27,3 @@ assert(sourceFile.description == sourceText)
 // Visualize the complete syntax tree.
 dump(sourceFile)
 ```
-
-## Implementation Status
-
-There are a number of implementation tasks involved in realizing the parser, which include:
-
-* [ ] Parsing
-  * [x] Rough parse of full grammar
-  * [x] Recovery from unexpected syntax
-  * [x] Insert missing syntax where needed
-  * [ ] SIL syntax
-  * [ ] [Regular expression literals](https://github.com/apple/swift-experimental-string-processing)
-  * [x] Parse into raw syntax nodes
-* [ ] Diagnostics
-  * [ ] Render diagnostics to a terminal with color, highlights, etc.
-  * [ ] Fix-Its to add/remove/replace syntax
-  * [ ] Emit diagnostics in the same binary format that the Swift compiler/driver do
-  * [x] Syntax visitor to emit diagnostics for all missing and unexpected syntax
-* [ ] Tooling and automation
-  * [ ] Improve test harness to easily check the expected parse trees + diagnostics
-  * [ ] Migrate [attribute definitions](https://github.com/apple/swift/blob/main/include/swift/AST/Attr.def) from C++ to Swift
-  * [x] Migrate [grammar definition](https://github.com/apple/swift/tree/main/utils/gyb_syntax_support) to the swift-syntax repository
-  * [ ] Migrate grammar definition from Python to Swift
-  * [ ] Replace uses of gyb with SwiftSyntax-based generation
-* [ ] Auditing the grammar to ensure that it accurately represents the Swift grammar
-  * [ ] Attributes
-  * [ ] Declarations
-  * [ ] Statements
-  * [ ] Expressions
-* [ ] AST generation within the Swift compiler itself
-
-### Testing methodology
-
-We'll employ a number of different techniques to help test the new parser at each phase, including:
-
-* [x] Unit tests for specific parser behaviors, added as we implement pieces and uncover bugs.
-
-* [ ] Self-parse test to ensure that all Swift files in the package can be parsed, that the syntax tree is well-formed, and that it always reproduces the input source.
-
-* [ ] Parsing tests from the Swift compiler repository to check that the parsers accept the same code.
-
-* [ ] Fuzzing tests that ensure that the parser does not crash, and always reproduces the input source.

Sources/SwiftSyntax/Documentation.docc/ChangingSwiftSyntax.md

@@ -6,74 +6,18 @@ Discover the steps necessary to add, remove, and update syntax nodes
 
 The surface area of the entire Swift programming language is represented in the
 Swift Syntax API and its associated data structures. In order to keep these
-structures in sync, this project makes use of a code generation utility called
-[gyb][GYB] - short for "Generate Your Boilerplate". 
-
-## Introduction to GYB
-
-The gyb utility allows for [literate metaprogramming][LiterateProgramming] of 
-files in any human-editable format. The standard convention is to attach the
-`.gyb` suffix to a file. For example, this repository uses `.swift.gyb` files
-that generate `.swift` code. The metaprogramming language that gyb uses is
-python. Templating code is delimited by `%` and can be used inline with `${...}`
-blocks. 
-
-Here is the gyb definition for all of the cases of the <doc:TokenKind> enum.
-
-```swift
-public enum TokenKind: Hashable {
-  case eof
-% for token in SYNTAX_TOKENS:
-%   kind = token.swift_kind()
-%
-%   # Tokens that don't have a set text have an associated value that
-%   # contains their text.
-%   if not token.text:
-%     kind += '(String)'
-%   end
-  case ${kind}
-% end
-}
-```
-
-- Note: gyb files act as the source of truth in this repository. Always edit
-        gyb files instead of the corresponding Swift files, or your changes will
-        be overwritten the next time those files are regenerated.
+structures in sync, this project makes generates the code using the Swift package 
+`CodeGeneration`, located in the root of this repository. 
 
 ## Regenerating Files
 
-While working on gyb files, it can be helpful to regenerate the files in this
-repository to see the effects of local changes. SwiftSyntax comes with a script
-called `build-script.py` that automates the process of regenerating, validating,
-and building all of the files in this repository. 
-
-To invoke `build-script.py` to regenerate source code, run
-
-```
-$ ./build-script.py generate-source-code --toolchain /path/to/toolchain/usr
-```
-
-where `/path/to/toolchain` can point either to the `XcodeDefault.xctoolchain`
-installed with Xcode, or to a [toolchain downloaded from swift.org][SwiftToolchains].
-
-- Note: To see the raw commands that `build-script.py` is running, use the `-v`
-        flag. This can be used to dump the command(s) necessary to regenerate
-        one file at a time.
-
-This script also comes with a verification command that is run by Swift's
-CI/CD systems to ensure that you regenerate files before they are checked into
-the version control system. 
-
-To invoke `build-script.py` in verification mode, run
-
-```
-$ ./build-script.py verify-source-code --toolchain /path/to/toolchain/usr
-```
+To re-generate the files after changing `CodeGeneration` run the `generate-swiftsyntax` 
+target of `CodeGeneration` and pass `path/to/swift-syntax/Sources` as the argument.
 
 ## Adding and Removing Syntax Nodes
 
 The files containing the definition of all of the syntax nodes are available
-under the [gyb_syntax_support][gyb_syntax_support] directory. These files
+under the [SyntaxSupport][SyntaxSupport] directory. These files
 are roughly divided according to broad syntactic categories in the Swift
 programming language. That is, the syntax nodes for classes, structs, and actors
 are defined in `DeclNodes.py`, while the syntax nodes for string literals, 
@@ -92,74 +36,29 @@ nodes among these categories for easier processing.
 
 The node for a source file is reproduced below:
 
-```python
-Node('SourceFile', name_for_diagnostics='source file', kind='Syntax',
-     traits=['WithStatements'],
-     children=[
-         Child('Statements', kind='CodeBlockItemList',
-               collection_element_name='Statement'),
-         Child('EOFToken', kind='EOFToken')
-     ]),
-```
-
-Notice that the `Statements` child node contains a `collection_element_name`
-field. This is because it is defining a syntax collection with the type
-`CodeBlockItemList` with elements that are `Statement` nodes.
-
-- Warning: Always remember to run `build-script.py` to regenerate files after
-           you make changes to the syntax nodes.
-
-
-## Adding and Removing Attributes
-
-The file that defines all of the attributes respected by the Swift compiler 
-is available under the [gyb_syntax_support][gyb_syntax_support] directory
-in AttributedKinds.py. 
-
-### Type Attributes
-
-To define a new attribute that applies to types, add a new `TypeAttribute` to
-the `TYPE_ATTR_KINDS` list with the name of the attribute spelled without the
-leading '@' sign.
-
-```python
-TypeAttribute('_myAttribute'), # Becomes @_myAttribute in the compiler
-```
+```swift
+  Node(
+    name: "SourceFile",
+    nameForDiagnostics: "source file",
+    kind: "Syntax",
+    traits: [
+      "WithStatements"
+    ],
+    parserFunction: "parseSourceFile",
+    children: [
+      Child(
+        name: "Statements",
+        kind: .collection(kind: "CodeBlockItemList", collectionElementName: "Statement")
+      ),
+      Child(
+        name: "EOFToken",
+        kind: .node(kind: "EOFToken")
+      ),
+    ]
+  ),
 
-### Declaration Attributes
-
-To define a new attribute that applies to declarations, add a new 
-`DeclAttribute` to the `DECL_ATTR_KINDS` list. Declaration attributes are
-richer than type attributes, and require a bit more metadata to define them.
-A `DeclAttribute` consists of the attribute's name, the name of the 
-corresponding C++ class, a series of metadata tags, and a stable serialization
-code.
-
-To add an attribute, identify an unused serialization code and place the
-attribute in the correct position in the `DECL_ATTR_KINDS` list. Then,
-fill out the corresponding metadata and tags. At a minimum, every attribute
-should come with ABI and API stability tags. But there are also tags that
-alter the behavior of the AST printer and the parser that may be desirable.
-
-The definition of the `_silgen_name` declaration attribute is reproduced below
-as it provides a good example of many of these tags:
-
-```python
-DeclAttribute('_silgen_name', 'SILGenName', # Spelled @_silgen_name, becomes SILGenNameAttr in C++
-              OnAbstractFunction, # Only applies to functions - parser will diagnose other usages automatically
-              LongAttribute,  UserInaccessible, # Attribute should be printed on its own line since it's long.
-              ABIStableToAdd,  ABIStableToRemove, APIStableToAdd,  APIStableToRemove, # Stability tags
-              code=0), # Serialization code
 ```
 
-Once a new attribute has been added, you will need to regenerate SwiftSyntax to
-account for it. If the attribute includes a non-trivial set of arguments, 
-consider adding a dedicated syntax node for it as well. To use the new attribute
-with the Swift compiler, ensure that swift-syntax and swift are checked out
-alongside each other and invoke the appropriate build tool to rebuild the Swift
-compiler. The compiler's build system automatically detects changes to the 
-Attr.def.gyb gyb file and the python defined here. 
-
 ## Committing Changes
 
 When it is time to commit changes to the Swift Syntax repository, most cases
@@ -186,10 +85,9 @@ and is a much more involved process. [The Swift Forums][CompilerDev] are a
 great source for guidance on how to extend the C++ compiler to accomodate new 
 syntax and semantics for the Swift programming language.
 
-[GYB]: https://github.com/apple/swift/blob/main/utils/gyb.py
 [LiterateProgramming]: https://en.wikipedia.org/wiki/Literate_programming
 [SwiftToolchains]: https://www.swift.org/download/
-[gyb_syntax_support]: https://github.com/apple/swift-syntax/tree/main/Sources/generate-swift-syntax-builder/gyb_syntax_support
+[SyntaxSupport]: https://github.com/apple/swift-syntax/tree/main/CodeGeneration/Sources/SyntaxSupport
 [swift-stress-tester]: https://github.com/apple/swift-stress-tester
 [Swiftc]: https://github.com/apple/swift
 [ParserBasics]: https://github.com/apple/swift-syntax/tree/main/Sources/SwiftParser/SwiftParser.docc/ParsingBasics.md

build-script.py

@@ -118,7 +118,7 @@ def realpath(path: Optional[str]) -> Optional[str]:
 
 
 # -----------------------------------------------------------------------------
-# Generating gyb Files
+# Generating Files
 
 
 def run_code_generation(
@@ -515,16 +515,6 @@ def test_command(args: argparse.Namespace) -> None:
 _DESCRIPTION = """
 Build and test script for SwiftSyntax.
 
-Build SwiftSyntax by generating all necessary files form the corresponding
-.swift.gyb files first. For this, SwiftSyntax needs to be check out alongside
-the main swift repo (http://github.com/apple/swift/) in the following structure
-
-- (containing directory)
-  - swift
-  - swift-syntax
-
-It is not necessary to build the compiler project.
-
 The build script can also drive the test suite included in the SwiftSyntax
 repo. This requires a custom build of the compiler project since it accesses
 test utilities that are not shipped as part of the toolchains. See the Testing

format.py

@@ -89,9 +89,7 @@ def get_files_to_format() -> List[Path]:
     files_to_format = package_dir.glob('**/*.swift')
 
     def should_exclude(path: Path) -> bool:
-        if 'gyb_generated' in path.parts:
-            return True
-        elif 'lit_tests' in path.parts:
+        if 'lit_tests' in path.parts:
             return True
         elif 'generated' in path.parts:
             return True