Skip to content

Conversation

@artemcm
Copy link
Contributor

@artemcm artemcm commented Nov 18, 2022

Building on top of #621, on an incremental build, load a prior dependency graph, check if it is still valid:

  1. import set of the current module has not changed
  2. input files of every module in the prior graph are older than the output and if it is, re-use it, skipping invoking the dependency scanner again.

More concretely, computation of the initial incremental state in computeIncrementalStateForPlanning will attempt to read out and validate a prior serialized inter-module dependency graph. If validated, such graph will become a part of the initial incremental state itself.

During build planning, at the stage of computing inter-module dependencies, the dependency scanning action will be skipped if the initial incremental state contains a still-valid prior inter-module dependency graph.

In both cases, if a prior graph is being re-used, or if a new graph was computed via a dependency scanning action, the graph will become a component of the incremental build state to be serialized after the build is complete.

This change does not yet result in skipping the actual module build jobs in the planning output, which can be done with this, but this is left to a followup PR.

There is also some notable overlap between this and what is already done for detecting changes in input files to the current compilation:

  • Suppose a Swift module interface of a module dependency has changed. This means that, according to the above mechanism, a dependency scanning action must be performed on an incremental build. This will result in a new dependency graph and a subsequent re-build of the dependency. But, moreover, there is an existing mechanism in place to ensure that source-files also get re-built on changes to previously-recorded inputs to compilation so the source-files will also be re-built when a dependency changes.

Once explicit builds become the default, we can consolidate the two approaches into one.

Partially resolves rdar://66801475

@_spi(Testing) public extension Driver {
/// Scan the current module's input source-files to compute its direct and transitive
/// module dependencies.
mutating func gatherModuleDependencies()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code just moved from Planning.swift it is unchanged.

@artemcm artemcm force-pushed the IncrementalExplicitBuildReScan branch 2 times, most recently from 72fbcc7 to c4029ab Compare November 18, 2022 22:59
///
/// FIXME: This is a little ridiculous. We could probably just replace the
/// build record outright with a serialized format.
var driverDependencyGraphPath: VirtualPath? {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was duplicated in BuildRecordInfo and this instance was not used anywhere.

@artemcm artemcm force-pushed the IncrementalExplicitBuildReScan branch 3 times, most recently from 5016cf4 to aeacb87 Compare November 29, 2022 23:29
@artemcm artemcm changed the title [Explicit Module Builds] Serialize the inter-module dependency graph as incremental build state [Explicit Module Builds] Serialize and re-use the inter-module dependency graph as incremental build state Nov 29, 2022

/// A build-record-relative path to the location of a serialized copy of the
/// driver's inter-module dependency graph.
var interModuleDependencyGraphPath: VirtualPath {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively we can also make this a part of the .swiftdeps file, but for now it seemed simpler to keep it separate.

@artemcm
Copy link
Contributor Author

artemcm commented Nov 29, 2022

@swift-ci test

Thread.sleep(forTimeInterval: 1)
print("*** touching \(name) ***", to: &stderrStream); stderrStream.flush()
let (path, contents) = try! XCTUnwrap(inputPathsAndContents.filter {$0.0.pathString.contains(name)}.first)
try! localFileSystem.writeFileContents(path) { $0 <<< contents }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this to actually just re-write the file's contents with what they already are, so it actually just updates the timestamp. Previously touch was overwriting file contents with a pre-determined initial state for that input, instead of what the test may have modified it to be at that point.

@artemcm artemcm marked this pull request as ready for review November 30, 2022 16:09
Copy link
Contributor

@nkcsgexi nkcsgexi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed with Artem about this offline. LGTM as the first step for handling incremental explicit module build!

…as incremental build state

On an incremental build, load a prior dependency graph, check if it is still valid:
1. import set of the current module has not changed
2. input files of every module in the prior graph are older than the output
and if it is, re-use it, skipping invoking the dependency scanner again.

Partially resolves rdar://66801475
@artemcm artemcm force-pushed the IncrementalExplicitBuildReScan branch from aeacb87 to 9c6e137 Compare December 1, 2022 18:50
@artemcm
Copy link
Contributor Author

artemcm commented Dec 1, 2022

@swift-ci test

@artemcm
Copy link
Contributor Author

artemcm commented Dec 1, 2022

Thanks @tshortli

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants