[ASTGen] Move regex literal parsing from SwiftCompilerSources to ASTGen #69838

rintaro · 2023-11-14T03:38:01Z

(Continuation from #68892)

ASTGen always builds with the host Swift compiler, without requiring bootstrapping, and is enabled in more places. Move the regex literal parsing logic there so it is enabled in more host environments, and makes use of CMake's Swift support. Enable all of the regex literal tests when ASTGen is built, to ensure everything is working.

Remove the "AST" and "Parse" Swift modules from SwiftCompilerSources, because they are no longer needed.

rintaro · 2023-11-14T03:38:16Z

@swift-ci Please smoke test

rintaro · 2023-11-14T03:42:13Z

lib/Parse/Lexer.cpp

lib/Parse already depends on ASTGen (e.g. swift_ASTGen_buildTopLevelASTNodes) Instead of injecting the function, just use the @_cdecl function in ASTGen

rintaro · 2023-11-14T06:45:33Z

@swift-ci Please smoke tests

SwiftCompilerSources/CMakeLists.txt

lib/ASTGen/Package.swift

lib/ASTGen/Sources/ASTGen/Regex.swift

SwiftCompilerSources/Sources/Parse/Parse.swift

rintaro · 2023-11-14T14:31:06Z

@swift-ci Please smoke test

rintaro · 2023-11-15T23:59:56Z

swiftlang/swift-experimental-string-processing#699
@swift-ci Please smoke test

rintaro · 2023-11-16T00:56:04Z

swiftlang/swift-experimental-string-processing#699
@swift-ci Please smoke test

compnerd · 2023-11-16T03:21:08Z

C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-experimental-string-processing\Sources\_RegexParser\Regex\Parse\SourceLocation.swift:136:19: error: cannot specialize non-generic type 'Source.LocatedError'
    return Source.LocatedError<Self>(self, loc)
                  ^           ~~~~~~
                              

error: fatalError

rintaro · 2023-11-16T05:07:31Z

swiftlang/swift-experimental-string-processing#699
@swift-ci Please smoke test

compnerd · 2023-11-16T16:48:42Z

Failed Tests (3):
  Swift(windows-x86_64) :: StringProcessing/Parse/regex.swift
  Swift(windows-x86_64) :: StringProcessing/Parse/regex_parse_end_of_buffer.swift
  Swift(windows-x86_64) :: StringProcessing/Parse/regex_parse_error.swift

ASTGen always builds with the host Swift compiler, without requiring bootstrapping, and is enabled in more places. Move the regex literal parsing logic there so it is enabled in more host environments, and makes use of CMake's Swift support. Enable all of the regex literal tests when ASTGen is built, to ensure everything is working. Remove the "AST" and "Parse" Swift modules from SwiftCompilerSources, because they are no longer needed.

swiftrt.obj is required for making things work

This reverts commit 5d0c5a6.

rintaro · 2023-11-16T19:04:25Z

@swift-ci Please smoke test

finagolfin · 2023-11-28T03:59:41Z

This pull broke a test on the Android AArch64 CI, which doesn't have a host Swift compiler so it does a full bootstrap, ie not building the swift-syntax parser and now regex literals after this pull:

FAIL: Swift(android-aarch64) :: SourceKit/Sema/sema_regex.swift (8100 of 17117)
******************** TEST 'Swift(android-aarch64) :: SourceKit/Sema/sema_regex.swift' FAILED ********************
Script:
--
: 'RUN: at line 6';   /home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/buildbot_linux/swift-linux-x86_64/bin/sourcekitd-test -module-cache-path '/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/buildbot_linux/swift-linux-x86_64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache' -req=sema /home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift -- /home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift -Xfrontend -enable-bare-slash-regex -Xfrontend -disable-availability-checking | /usr/bin/python3.10 /home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/utils/PathSanitizingFileCheck --allow-unused-prefixes --sanitize BUILD_DIR=/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/buildbot_linux/swift-linux-x86_64 --sanitize SOURCE_DIR=/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift --use-filecheck /home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/buildbot_linux/llvm-linux-x86_64/bin/FileCheck  /home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift
--
Exit Code: 1

Command Output (stderr):
--
{
  key.request: source.request.editor.open,
  key.name: "/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift",
  key.compilerargs: [
    "-module-cache-path",
    "/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/buildbot_linux/swift-linux-x86_64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache",
    "/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift",
    "-Xfrontend",
    "-enable-bare-slash-regex",
    "-Xfrontend",
    "-disable-availability-checking"
  ],
  key.sourcefile: "/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift",
  key.primary_file: "/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift"
}
/home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift:24:15: error: CHECK-NOT: excluded string found in input
// CHECK-NOT: key.severity:
              ^
<stdin>:20:2: note: found here
 key.severity: source.diagnostic.severity.error,
 ^~~~~~~~~~~~~

Input file: <stdin>
Check file: /home/ubuntu/jenkins/workspace/oss-swift-RA-linux-ubuntu-16.04-android-arm64/swift/test/SourceKit/Sema/sema_regex.swift

-dump-input=help explains the following input dump.

Input was:
<<<<<<
        .
        .
        .
       15: [ 
       16:  { 
       17:  key.line: 2, 
       18:  key.column: 3, 
       19:  key.filepath: "SOURCE_DIR/test/SourceKit/Sema/sema_regex.swift", 
       20:  key.severity: source.diagnostic.severity.error, 
not:24      !~~~~~~~~~~~~                                    error: no match expected
       21:  key.id: "use_nonmatching_operator", 
       22:  key.description: "'/' is not a prefix unary operator", 
       23:  key.diagnostic_stage: source.diagnostic.stage.swift.sema 
       24:  }, 
       25:  { 
        .
        .
        .
>>>>>>

The issue is that this test requires swift_in_compiler, which is always defined unless bootstrapping is completely turned off. Previously, regex literals were still built in the full bootstrapping mode, but after this pull, they no longer are, so the features associated with swift_in_compiler have now changed. This will affect any platform that doesn't have a host Swift compiler, not just the Android CI.

This test was missed in swiftlang#69838. Update to depend on swift_swift_parser (ie. "have a host Swift compiler").

DougGregor · 2023-11-28T20:58:56Z

This test should be using swift_swift_parser, which is covered by #70073.

However, in the near future we're going to require a Swift compiler on the host to build the Swift compiler, with "full bootstrap" going away entirely, so that we can fully switch over to (e.g.) the new Swift parser and driver, and remove the C++ implementations.

rintaro requested review from CodaFi, DougGregor, ahoppen, bnbarham, eeckstein, hamishknight, hborla, xedin and zoecarver as code owners November 14, 2023 03:38

rintaro marked this pull request as draft November 14, 2023 03:38

rintaro commented Nov 14, 2023

View reviewed changes

DougGregor approved these changes Nov 14, 2023

View reviewed changes

rintaro force-pushed the astgen-regex branch from 1c69924 to f32dfbc Compare November 14, 2023 06:44

hamishknight approved these changes Nov 14, 2023

View reviewed changes

SwiftCompilerSources/CMakeLists.txt Outdated Show resolved Hide resolved

lib/ASTGen/Package.swift Outdated Show resolved Hide resolved

lib/ASTGen/Sources/ASTGen/Regex.swift Outdated Show resolved Hide resolved

SwiftCompilerSources/Sources/Parse/Parse.swift Outdated Show resolved Hide resolved

rintaro force-pushed the astgen-regex branch from 764ceb5 to 0b9dff3 Compare November 14, 2023 14:27

rintaro added 3 commits November 16, 2023 10:59

[Windows] Link host tools with 'swiftrt.obj'

abd4164

swiftrt.obj is required for making things work

Revert "ASTGen: add a workaround for CSC conformance visibility"

24b44e3

This reverts commit 5d0c5a6.

rintaro force-pushed the astgen-regex branch from 0b9dff3 to 24b44e3 Compare November 16, 2023 19:03

rintaro marked this pull request as ready for review November 16, 2023 21:37

rintaro merged commit 66e4487 into swiftlang:main Nov 16, 2023

rintaro mentioned this pull request Nov 16, 2023

[5.10][ASTGen] Move regex literal parsing from SwiftCompilerSources to ASTGen #69937

Merged

bnbarham added a commit to bnbarham/swift that referenced this pull request Nov 28, 2023

[Test] Regex tests depend on host Swift compiler

000a808

This test was missed in swiftlang#69838. Update to depend on swift_swift_parser (ie. "have a host Swift compiler").

bnbarham mentioned this pull request Nov 28, 2023

[Test] Regex tests depend on host Swift compiler #70073

Merged

bnbarham added a commit to bnbarham/swift that referenced this pull request Nov 28, 2023

[Test] Regex tests depend on host Swift compiler

230c09a

This test was missed in swiftlang#69838. Update to depend on swift_swift_parser (ie. "have a host Swift compiler").

bnbarham mentioned this pull request Nov 28, 2023

[5.10][Test] Regex tests depend on host Swift compiler #70075

Merged

[ASTGen] Move regex literal parsing from SwiftCompilerSources to ASTGen #69838

[ASTGen] Move regex literal parsing from SwiftCompilerSources to ASTGen #69838

Uh oh!

Conversation

rintaro commented Nov 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rintaro commented Nov 14, 2023

Uh oh!

rintaro Nov 14, 2023

Choose a reason for hiding this comment

Uh oh!

rintaro commented Nov 14, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rintaro commented Nov 14, 2023

Uh oh!

rintaro commented Nov 15, 2023

Uh oh!

rintaro commented Nov 16, 2023

Uh oh!

compnerd commented Nov 16, 2023

Uh oh!

rintaro commented Nov 16, 2023

Uh oh!

compnerd commented Nov 16, 2023

Uh oh!

rintaro commented Nov 16, 2023

Uh oh!

finagolfin commented Nov 28, 2023

Uh oh!

DougGregor commented Nov 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rintaro commented Nov 14, 2023 •

edited

Loading