Skip to content

Improve CONTRIBUTING.md #1993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 5, 2023
Merged

Improve CONTRIBUTING.md #1993

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7ce21f8
Improve CONTRIBUTING.md
ahoppen Aug 3, 2023
e8fc865
Update ChangingSwiftSyntax.md
ahoppen Aug 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 28 additions & 83 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,109 +1,54 @@
# Contributing

This guide refers to Swift Development Snapshots. These can be downloaded from [swift.org/download](https://swift.org/download/#snapshots).
If you are seeing issues, it is generally a good idea to re-try with the latest Swift Development Snapshot.
## Building & Testing

## Building
swift-syntax is a SwiftPM package, so you can build and test it using anything that supports packages - opening in Xcode, Visual Studio Code with [Swift for Visual Studio Code](https://github.com/swift-server/vscode-swift) installed, or through the command line using `swift build` and `swift test`.

The easiest option to build SwiftSyntax is to open it in Xcode.
> 💡 Tip: Running swift-syntax’s self-parse tests takes the majority of testing time. If you want to iterate quickly, you can skip these tests:
> - In Xcode
> 1. Product -> Scheme -> Edit Scheme…
> 2. Select the Arguments tab in the Run section
> 3. Add a `SKIP_LONG_TESTS` environment variable with value `1`
> - On the command line: Set the `SKIP_LONG_TESTS` environment variable to `1` when running tests, e.g by running `SKIP_LONG_TESTS=1 swift test`

Alternatively you can also build it from the command line using `build-script.py`. To do this, perform the following steps

1. Check `swift-syntax` and `swift-argument-parser` out side by side:
```
- (enclosing directory)
- swift-argument-parser
- swift-syntax
```
2. Execute the following command
```bash
swift-syntax/build-script.py build --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr
```
> 💡 Tip: If you are only modifying the `SwiftParser` module, you can run only the parser tests by selecting the `SwiftParserTest` target.
> - In Xcode: Select the `SwiftParserTest` scheme. If you can’t find it in your Schemes, you need to manually add it using Product -> Scheme -> New Scheme…
> - On the command line: Run `swift test --test-product SwiftParserTests`

## Formatting

SwiftSyntax is being formatted using [swift-format](http://github.com/apple/swift-format) to ensure a consistent style.

To format your changes run `format.py` at the root of this repository. If you have a `swift-format` executable ready, you can pass it to `format.py`. If you do not, `format.py` will build its own copy of `swift-format` in /tmp/swift-format.

CI will ensure that all hand-written source code is correctly formatted. Generated source code is not formatted to make it easier to spot changes when re-running code generation.

## Testing

Because of SwiftSyntax’s integration with the Swift compiler project, testing certain parts of the project is a little bit more involved than others.

The `SwiftParser` tests have no dependencies, other `XCTests` require you to use a matching Swift Development Snapshot, `lit`-based test require a compiler build.

Run the tests that you believe are necessary locally. CI will always run all test before allowing the PR to be merged.

### SwiftParser
swift-syntax is formatted using [swift-format](http://github.com/apple/swift-format) to ensure a consistent style.

If you are only modifying the `SwiftParser` module, you can just run the tests using Xcode by testing the `SwiftParserTest` target.
To format your changes run `format.py` at the root of this repository. If you have a `swift-format` executable ready, you can pass it to `format.py`. If you do not, `format.py` will build its own copy of `swift-format` in `/tmp/swift-format`.

If you can’t find it in your Schemes, you need to manually add it using Product -> Scheme -> New Scheme…
If you are seeing surprising formatting results, you most likely have a `swift-format` installed on your system that’s not the most recent version built from the `main` branch. To fix this, clone [swift-format](http://github.com/apple/swift-format), build it using `swift build` and pass the freshly built executable to `format.py` as `--swift-format path/to/swift-format/.build/debug/swift-format`. Alternatively, you can uninstall `swift-format` on your system and `format.py` will build it from scratch.

### XCTests
Generated source code is not formatted to make it easier to spot changes when re-running code generation.

To run the tests in Xcode, select the SwiftSyntax-Package scheme and hit Product -> Test.

You can also run the tests from the command line using
```bash
./build-script.py test --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr --skip-lit-tests
```
## Generating Source Code

If you are seeing issues regarding a mismatched parser library, try the following
1. Update your Swift Development Snapshot
2. Revert your swift-syntax checkout to the date of your Swift Development Snapshot
3. Wait for a new Swift Development Snapshot
4. If the above options are not possible, build your own Swift toolchain locally and use that toolchain as the `--toolchain` parameter for SwiftSyntax’s `build-script.py` invocations
- Note: Building your own toolchain will take more than 1 hour and even longer if you are running into any issues. We do not recommend building your own Swift toolchain unless you are familiar with Swift compiler development.
If you want to modify the generated files, open the [CodeGeneration](CodeGeneration) package and run the `generate-swift-syntax` executable.

Tip: Running SwiftSyntax’s self-parse tests takes the majority of testing time. If you want to iterate quickly, you can skip these tests using the following steps:
1. Product -> Scheme -> Edit Scheme…
2. Select the Arguments tab in the Run section
3. Add a `SKIP_LONG_TESTS` environment variable with value `1`
## Additional Verification

### Additional Verification
swift-syntax has additional verification methods (see the sections below) that provide more extensive validation. They have a significant runtime impact on swift-syntax and are thus not enabled by default when building swift-syntax, but are enabled in CI. If CI fails and you are unable to reproduce the failure locally, make sure that `SKIP_LONG_TESTS` is not set and try enabling these validations.

swift-syntax has two additional verification methods (see the two sections below) that provide more extensive validation. They have significant runtime impact on swift-syntax and are thus not enabled by default when building swift-syntax, but are enabled in CI. If CI fails and you are unable to reproduce the failure locally, make sure that `SKIP_LONG_TESTS` is not set and try enabling these validations.
### RawSyntax Validation

#### RawSyntax Validation
When the `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` environment variable is set while building swift-syntax, SwiftSyntax will perform additional validation that the layout of the syntax tree is correct. It validates that
1. every child of a syntax node has the correct kind, which should be guaranteed by the Swift type system in most cases
2. each token only has one of the token kinds that is specified in the syntax tree layout of the `CodeGeneration` package.

When the `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` environment variable is set while building swift-syntax (or the check for that variable has been changed to always return `true` in Package.swift), SwiftSyntax will perform additional validation that the layout of the syntax tree is correct. It validates that every child of a syntax node has the correct kind (which should be guaranteed by the Swift type system in most cases) and, more importantly, validates that each token only has one of the token kinds that is specified in the syntax tree layout of the `CodeGeneration` package.
If this validation hits an assertion failure that a token is not accepted at a certain position in the syntax tree, double-check if the token kind that is being stored in the syntax tree actually makes sense here. If it does not, check if there is a parser bug or whether you need to remap the token kind. If it does make sense, add the token kind to `.token(choices:)` of the syntax node in CodeGeneration, re-generate that source code and run tests again.

If this validation hits an assertion failure that a token is not accepted at a certain position in the syntax tree, double check if the token kind that is being stored in the syntax tree actually makes sense here. If it does not, check if there is a parser bug or whether you need to remap the token kind. If it does make sense, add the token kind to `.token(choices:)` of the syntax node in CodeGeneration, re-generate that source code and run tests again.
> 💡 Tip: If you want to run tests in Xcode with RawSyntax validation enabled, you can also modify Package.swift and replace the check for `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` with `true`.

#### Test Case Mutation
### Test Case Mutation

When the `SWIFTPARSER_ENABLE_ALTERNATE_TOKEN_INTROSPECTION` environment variable is set while building swift-syntax (or the check for that variable has been changed to always return `true` in Package.swift), SwiftParser records alternative tokens that the parser was looking for at specific offsets in the source file (e.g. whether it also checked for a `struct` keyword when the source code contained a `class` keyword). It will then use that information to mutate the test case by e.g. substituting `class` by `struct`.
When the `SWIFTPARSER_ENABLE_ALTERNATE_TOKEN_INTROSPECTION` environment variable is set while building swift-syntax, SwiftParser records alternative tokens that the parser was looking for at specific offsets in the source file, e.g. whether it also checked for a `struct` keyword when the source code contained a `class` keyword. It will then use that information to mutate the test case by e.g. substituting `class` with `struct`.

When testing finds one of these failures, it will show you the syntax tree that produced the failure. Create a new test case with the source code the failure gives you and fix the failure.

### `lit`-based Tests

A few tests are based LLVM’s `lit` and `FileCheck` tools.
To run these, build `FileCheck`, e.g. by building the Swift compiler and run the tests using the following command:
```bash
./build-script.py test --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr --filecheck-exec /path/to/build/Release+Asserts/llvm-macosx-x86_64/bin/FileCheck
```

## Generating Source Code

If you want to modify the code-generated files, perform the following steps:

1. Check out `swift` next to `swift-syntax`
```
- (enclosing directory)
- swift
- swift-argument-parser
- swift-syntax
```
2. Run the following command
```bash
swift-syntax/build-script.py generate-source-code --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr
```
3. The new source-generated file will be written into your `Sources` directory.

## Swift Version

We require that SwiftSyntax builds with the latest released compiler and the previous major version (e.g. with Swift 5.8 and Swift 5.7).
We require that swift-syntax builds with the latest released compiler and the previous major version (e.g. with Swift 5.8 and Swift 5.7).
75 changes: 28 additions & 47 deletions Sources/SwiftSyntax/Documentation.docc/Contributing/ChangingSwiftSyntax.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,31 +12,27 @@ structures in sync, this project generates code using the Swift package
## Regenerating Files

To re-generate the files after changing `CodeGeneration` run the `generate-swiftsyntax`
target of `CodeGeneration` and pass `path/to/swift-syntax/Sources` as the argument.
target of `CodeGeneration`.

On the command line, this would be
```bash
swift run --package-path CodeGeneration generate-swiftsyntax Sources
swift run --package-path CodeGeneration generate-swiftsyntax
```

Or if you open the `CodeGeneration` package in Xcode, you can add the
`generate-swift syntax ../Sources` arguments using Product -> Scheme -> Edit Scheme…

## Adding and Removing Syntax Nodes

The files containing the definition of all of the syntax nodes are available
under the [SyntaxSupport][SyntaxSupport] directory. These files
under the [SyntaxSupport](SyntaxSupport) directory. These files
are roughly divided according to broad syntactic categories in the Swift
programming language. That is, the syntax nodes for classes, structs, and actors
are defined in `DeclNodes.py`, while the syntax nodes for string literals,
arrays, and tuples is defined in `ExprNodes.py`.
are defined in `DeclNodes.swift`, while the syntax nodes for string literals,
arrays, and tuples is defined in `ExprNodes.swift`.

To add a node to these files, it can be helpful to copy an existing node and
alter its definition to suit the needs of the new syntax being defined. A syntax
node consists of a name, a kind, and a list of child nodes. The name of a node
not only identifies it uniquely to Swift Syntax, it also provides a handle that
other nodes can use to refer to it. The kind of a syntax node defines the
class of syntax the node belongs to. All nodes are at least of the `Syntax`
node consists of a kind (which also defines the node’s name), a base kind, and a list of
child nodes. The base kind of a syntax node defines the class of syntax the node belongs to.
All nodes are at least of the `Syntax`
kind, though some nodes may have a more specific kind like `Stmt` for
statements or `Expr` for expressions. The SwiftSyntax library expands these
kinds into protocol conformances and allows for casting strongly typed syntax
Expand All @@ -45,33 +41,29 @@ nodes among these categories for easier processing.
The node for a source file is reproduced below:

```swift
Node(
name: "SourceFile",
nameForDiagnostics: "source file",
kind: "Syntax",
traits: [
"WithStatements"
],
parserFunction: "parseSourceFile",
children: [
Child(
name: "Statements",
kind: .collection(kind: "CodeBlockItemList", collectionElementName: "Statement")
),
Child(
name: "EOFToken",
kind: .node(kind: "EOFToken")
),
]
),

Node(
kind: .sourceFile,
base: .syntax,
nameForDiagnostics: "source file",
parserFunction: "parseSourceFile",
traits: ["WithStatements"],
children: [
Child(
name: "Statements",
kind: .collection(kind: .codeBlockItemList, collectionElementName: "Statement")
),
Child(
name: "EndOfFileToken",
deprecatedName: "EOFToken",
kind: .token(choices: [.token(tokenKind: "EndOfFileToken")])
),
]
)
```

## Committing Changes

When it is time to commit changes to the Swift Syntax repository, most cases
of adding syntax will require a single PR to swift-syntax. Added
syntactic elements will require corresponding changes to the included
Added syntactic elements will require corresponding changes to the included
SwiftParser library. For an introduction on parsing Swift nodes, see
[the article on Parsing Basics][ParserBasics].

Expand All @@ -82,21 +74,10 @@ SwiftParser library. For an introduction on parsing Swift nodes, see

When updating nodes, certain clients of SwiftSyntax that are relying upon those
nodes will need to be changed in tandem. For example, the
[swift-stress-tester][swift-stress-tester] uses SwiftSyntax, and the CI/CD
[swift-stress-tester][swift-stress-tester] uses SwiftSyntax, and the CI
system will not allow changes to SwiftSyntax that break `swift-stress-tester`
without a paired change to that repository.

When adding nodes or removing existing nodes, it is often the case that you
will need a paired PR to the [Swift Compiler repository][Swiftc] as well. This
step currently involves some amount of duplication with the SwiftParser library
and is a much more involved process. [The Swift Forums][CompilerDev] are a
great source for guidance on how to extend the C++ compiler to accommodate new
syntax and semantics for the Swift programming language.

[LiterateProgramming]: https://en.wikipedia.org/wiki/Literate_programming
[SwiftToolchains]: https://www.swift.org/download/
[SyntaxSupport]: https://github.com/apple/swift-syntax/tree/main/CodeGeneration/Sources/SyntaxSupport
[swift-stress-tester]: https://github.com/apple/swift-stress-tester
[Swiftc]: https://github.com/apple/swift
[ParserBasics]: https://github.com/apple/swift-syntax/tree/main/Sources/SwiftParser/SwiftParser.docc/ParsingBasics.md
[CompilerDev]: https://forums.swift.org/c/development/compiler/