[SWT-0002] A stable JSON-based ABI for tools integration (#479)

One of the core components of Swift Testing is its ability to
interoperate with Xcode 16, VS Code, and other tools. Swift Testing has
been fully open-sourced across all platforms supported by Swift, and can
be added as a package dependency (or—eventually—linked from the Swift
toolchain.)

Because Swift Testing may be used in various forms, and because
integration with various tools is critical to its success, we need it to
have a stable interface that can be used regardless of how it's been
added to a package.

Read the full proposal
[here](https://github.com/apple/swift-testing/blob/main/Documentation/Proposals/0002-json-abi.md).

### Checklist:

- [x] Code and documentation should follow the style of the [Style
Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md).
- [x] If public symbols are renamed or modified, DocC references should
be updated.

Author: grynspan

Documentation/ABI/JSON.md

@@ -11,13 +11,9 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors
 -->
 
 This document outlines the JSON schemas used by the testing library for its ABI
-entry point and for the `--experimental-event-stream-output` command-line
-argument. For more information about the ABI entry point, see the documentation
-for [ABI.EntryPoint_v0](https://github.com/search?q=repo%3Aapple%2Fswift-testing%EntryPoint_v0&type=code).
-
-> [!WARNING]
-> This JSON schema is still being developed and is subject to any and all
-> changes including removal from the package.
+entry point and for the `--event-stream-output-path` command-line argument. For
+more information about the ABI entry point, see the documentation for
+[ABIv0.EntryPoint](https://github.com/search?q=repo%3Aapple%2Fswift-testing%EntryPoint&type=code).
 
 ## Modified Backus-Naur form
 
@@ -103,8 +99,8 @@ encoded as a single [JSON Lines](https://jsonlines.org) value.
 
 A stream consists of a sequence of values encoded as [JSON Lines](https://jsonlines.org).
 A single instance of `<output-stream>` is defined per test process and can be
-accessed by passing `--experimental-event-stream-output` to the test executable
-created by `swift build --build-tests`.
+accessed by passing `--event-stream-output-path` to the test executable created
+by `swift build --build-tests`.
 
 ```
 <output-stream> ::= <output-record>\n | <output-record>\n <output-stream>

Documentation/Proposals/0002-json-abi.md

@@ -0,0 +1,123 @@
+//
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2023 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See https://swift.org/LICENSE.txt for license information
+// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
+//
+
+#if canImport(Foundation) && !SWT_NO_ABI_ENTRY_POINT
+private import _TestingInternals
+
+extension ABIv0 {
+  /// The type of the entry point to the testing library used by tools that want
+  /// to remain version-agnostic regarding the testing library.
+  ///
+  /// - Parameters:
+  ///   - configurationJSON: A buffer to memory representing the test
+  ///     configuration and options. If `nil`, a new instance is synthesized
+  ///     from the command-line arguments to the current process.
+  ///   - recordHandler: A JSON record handler to which is passed a buffer to
+  ///     memory representing each record as described in `ABI/JSON.md`.
+  ///
+  /// - Returns: Whether or not the test run finished successfully.
+  ///
+  /// - Throws: Any error that occurred prior to running tests. Errors that are
+  ///   thrown while tests are running are handled by the testing library.
+  public typealias EntryPoint = @convention(thin) @Sendable (
+    _ configurationJSON: UnsafeRawBufferPointer?,
+    _ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
+  ) async throws -> Bool
+
+  /// The entry point to the testing library used by tools that want to remain
+  /// version-agnostic regarding the testing library.
+  ///
+  /// The value of this property is a Swift function that can be used by tools
+  /// that do not link directly to the testing library and wish to invoke tests
+  /// in a binary that has been loaded into the current process. The value of
+  /// this property is accessible from C and C++ as a function with name
+  /// `"swt_abiv0_getEntryPoint"` and can be dynamically looked up at runtime
+  /// using `dlsym()` or a platform equivalent.
+  ///
+  /// The value of this property can be thought of as equivalent to a call to
+  /// `swift test --event-stream-output-path` except that, instead of streaming
+  /// JSON records to a named pipe or file, it streams them to an in-process
+  /// callback.
+  public static var entryPoint: EntryPoint {
+    return { configurationJSON, recordHandler in
+      try await Testing.entryPoint(
+        configurationJSON: configurationJSON,
+        recordHandler: recordHandler
+      ) == EXIT_SUCCESS
+    }
+  }
+}
+
+/// An exported C function that is the equivalent of
+/// ``ABIv0/entryPoint-swift.type.property``.
+///
+/// - Returns: The value of ``ABIv0/entryPoint-swift.type.property`` cast to an
+///   untyped pointer.
+@_cdecl("swt_abiv0_getEntryPoint")
+@usableFromInline func abiv0_getEntryPoint() -> UnsafeRawPointer {
+  unsafeBitCast(ABIv0.entryPoint, to: UnsafeRawPointer.self)
+}
+
+// MARK: - Xcode 16 Beta 1 compatibility
+
+/// An older signature for ``ABIv0/EntryPoint-swift.typealias`` used by Xcode 16
+/// Beta 1.
+///
+/// This type will be removed in a future update.
+@available(*, deprecated, message: "Use ABIv0.EntryPoint instead.")
+typealias ABIEntryPoint_v0 = @Sendable (
+  _ argumentsJSON: UnsafeRawBufferPointer?,
+  _ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
+) async throws -> CInt
+
+/// An older signature for ``ABIv0/entryPoint-swift.type.property`` used by
+/// Xcode 16 Beta 1.
+///
+/// This function will be removed in a future update.
+@available(*, deprecated, message: "Use ABIv0.entryPoint (swt_abiv0_getEntryPoint()) instead.")
+@_cdecl("swt_copyABIEntryPoint_v0")
+@usableFromInline func copyABIEntryPoint_v0() -> UnsafeMutableRawPointer {
+  let result = UnsafeMutablePointer<ABIEntryPoint_v0>.allocate(capacity: 1)
+  result.initialize { configurationJSON, recordHandler in
+    try await entryPoint(
+      configurationJSON: configurationJSON,
+      eventStreamVersionIfNil: -1,
+      recordHandler: recordHandler
+    )
+  }
+  return .init(result)
+}
+
+/// A common implementation for ``ABIv0/entryPoint-swift.type.property`` and
+/// ``copyABIEntryPoint_v0()`` that provides Xcode 16 Beta 1 compatibility.
+///
+/// This function will be removed (with its logic incorporated into
+/// ``ABIv0/entryPoint-swift.type.property``) in a future update.
+private func entryPoint(
+  configurationJSON: UnsafeRawBufferPointer?,
+  eventStreamVersionIfNil: Int? = nil,
+  recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
+) async throws -> CInt {
+  var args = try configurationJSON.map { configurationJSON in
+    try JSON.decode(__CommandLineArguments_v0.self, from: configurationJSON)
+  }
+
+  // If the caller needs a nil event stream version to default to a specific
+  // JSON schema, apply it here as if they'd specified it in the configuration
+  // JSON blob.
+  if let eventStreamVersionIfNil, args?.eventStreamVersion == nil {
+    args?.eventStreamVersion = eventStreamVersionIfNil
+  }
+
+  let eventHandler = try eventHandlerForStreamingEvents(version: args?.eventStreamVersion, forwardingTo: recordHandler)
+  let exitCode = await entryPoint(passing: args, eventHandler: eventHandler)
+  return exitCode
+}
+#endif

Sources/Testing/ABI/EntryPoints/ABIEntryPoint.swift

@@ -25,8 +25,8 @@ private import _TestingInternals
 ///     to this function.
 ///
 /// External callers cannot call this function directly. The can use
-/// ``copyABIEntryPoint_v0()`` to get a reference to an ABI-stable version of
-/// this function.
+/// ``ABIv0/entryPoint-swift.type.property`` to get a reference to an ABI-stable
+/// version of this function.
 func entryPoint(passing args: __CommandLineArguments_v0?, eventHandler: Event.Handler?) async -> CInt {
   let exitCode = Locked(rawValue: EXIT_SUCCESS)
 
@@ -200,7 +200,7 @@ public struct __CommandLineArguments_v0: Sendable {
   /// The value of the `--xunit-output` argument.
   public var xunitOutput: String?
 
-  /// The value of the `--experimental-event-stream-output` argument.
+  /// The value of the `--event-stream-output-path` argument.
   ///
   /// Data is written to this file in the [JSON Lines](https://jsonlines.org)
   /// text format. For each event handled by the resulting event handler, a JSON
@@ -215,18 +215,18 @@ public struct __CommandLineArguments_v0: Sendable {
   ///
   /// The file is closed when this process terminates or the test run completes,
   /// whichever occurs first.
-  public var experimentalEventStreamOutput: String?
+  public var eventStreamOutputPath: String?
 
   /// The version of the event stream schema to use when writing events to
-  /// ``experimentalEventStreamOutput``.
+  /// ``eventStreamOutput``.
   ///
   /// The corresponding stable schema is used to encode events to the event
   /// stream (for example, ``ABIv0/Record`` is used if the value of this
   /// property is `0`.)
   ///
   /// If the value of this property is `nil`, the testing library assumes that
   /// the newest available schema should be used.
-  public var experimentalEventStreamVersion: Int?
+  public var eventStreamVersion: Int?
 
   /// The value(s) of the `--filter` argument.
   public var filter: [String]?
@@ -252,8 +252,8 @@ extension __CommandLineArguments_v0: Codable {
     case quiet
     case _verbosity = "verbosity"
     case xunitOutput
-    case experimentalEventStreamOutput
-    case experimentalEventStreamVersion
+    case eventStreamOutputPath
+    case eventStreamVersion
     case filter
     case skip
     case repetitions
@@ -288,7 +288,8 @@ func parseCommandLineArguments(from args: [String]) throws -> __CommandLineArgum
   // NOTE: While the output event stream is opened later, it is necessary to
   // open the configuration file early (here) in order to correctly construct
   // the resulting __CommandLineArguments_v0 instance.
-  if let configurationIndex = args.firstIndex(of: "--experimental-configuration-path"), !isLastArgument(at: configurationIndex) {
+  if let configurationIndex = args.firstIndex(of: "--configuration-path") ?? args.firstIndex(of: "--experimental-configuration-path"),
+     !isLastArgument(at: configurationIndex) {
     let path = args[args.index(after: configurationIndex)]
     let file = try FileHandle(forReadingAtPath: path)
     let configurationJSON = try file.readToEnd()
@@ -302,12 +303,14 @@ func parseCommandLineArguments(from args: [String]) throws -> __CommandLineArgum
   }
 
   // Event stream output (experimental)
-  if let eventOutputIndex = args.firstIndex(of: "--experimental-event-stream-output"), !isLastArgument(at: eventOutputIndex) {
-    result.experimentalEventStreamOutput = args[args.index(after: eventOutputIndex)]
+  if let eventOutputIndex = args.firstIndex(of: "--event-stream-output-path") ?? args.firstIndex(of: "--experimental-event-stream-output"),
+     !isLastArgument(at: eventOutputIndex) {
+    result.eventStreamOutputPath = args[args.index(after: eventOutputIndex)]
   }
   // Event stream output (experimental)
-  if let eventOutputVersionIndex = args.firstIndex(of: "--experimental-event-stream-version"), !isLastArgument(at: eventOutputVersionIndex) {
-    result.experimentalEventStreamVersion = Int(args[args.index(after: eventOutputVersionIndex)])
+  if let eventOutputVersionIndex = args.firstIndex(of: "--event-stream-version") ?? args.firstIndex(of: "--experimental-event-stream-version"),
+     !isLastArgument(at: eventOutputVersionIndex) {
+    result.eventStreamVersion = Int(args[args.index(after: eventOutputVersionIndex)])
   }
 #endif
 
@@ -404,9 +407,9 @@ public func configurationForEntryPoint(from args: __CommandLineArguments_v0) thr
 
 #if canImport(Foundation)
   // Event stream output (experimental)
-  if let eventStreamOutputPath = args.experimentalEventStreamOutput {
+  if let eventStreamOutputPath = args.eventStreamOutputPath {
     let file = try FileHandle(forWritingAtPath: eventStreamOutputPath)
-    let eventHandler = try eventHandlerForStreamingEvents(version: args.experimentalEventStreamVersion) { json in
+    let eventHandler = try eventHandlerForStreamingEvents(version: args.eventStreamVersion) { json in
       try? _writeJSONLine(json, to: file)
     }
     configuration.eventHandler = { [oldEventHandler = configuration.eventHandler] event, context in
@@ -489,7 +492,7 @@ func eventHandlerForStreamingEvents(version: Int?, forwardingTo eventHandler: @e
   case nil, 0:
     ABIv0.Record.eventHandler(forwardingTo: eventHandler)
   case let .some(unsupportedVersion):
-    throw _EntryPointError.invalidArgument("--experimental-event-stream-version", value: "\(unsupportedVersion)")
+    throw _EntryPointError.invalidArgument("--event-stream-version", value: "\(unsupportedVersion)")
   }
 }
 

Sources/Testing/EntryPoints/EntryPoint.swift renamed to Sources/Testing/ABI/EntryPoints/EntryPoint.swift

@@ -15,7 +15,7 @@ extension ABIv0.Record {
   ///
   /// - Parameters:
   ///   - eventHandler: The event handler to forward events to. See
-  ///     ``ABIv0/EntryPoint`` for more information.
+  ///     ``ABIv0/EntryPoint-swift.typealias`` for more information.
   ///
   /// - Returns: An event handler.
   ///
@@ -74,7 +74,7 @@ extension EventAndContextSnapshot: Codable {}
 ///
 /// - Parameters:
 ///   - eventHandler: The event handler to forward events to. See
-///     ``ABIEntryPoint_v0`` for more information.
+///     ``ABIv0/EntryPoint-swift.typealias`` for more information.
 ///
 /// - Returns: An event handler.
 ///

Sources/Testing/EntryPoints/SwiftPMEntryPoint.swift renamed to Sources/Testing/ABI/EntryPoints/SwiftPMEntryPoint.swift

@@ -9,4 +9,5 @@
 //
 
 /// A namespace for ABI version 0 symbols.
-enum ABIv0: Sendable {}
+@_spi(ForToolsIntegrationOnly)
+public enum ABIv0: Sendable {}