Skip to content

Conversation

@nojaf
Copy link
Contributor

@nojaf nojaf commented Feb 14, 2023

I'm just curious at this point.

@vzarytovskii
Copy link
Member

Compiler build is deterministic, but it problem is still there, right? Pdb may have differences when graph checking is enabled?

@nojaf
Copy link
Contributor Author

nojaf commented Feb 15, 2023

Compiler build is deterministic, but it problem is still there, right? Pdb may have differences when graph checking is enabled.

I'm not quite sure. I was able to generate the same pdb about 50 times in a row on my machine.
For various projects (FSharp.Core, FCS, Fantomas.Core, my reproduction of #14703 project GraphSample)
These all worked for me: mvid is stable, file-hash of binary and pdb file are stable.

Used script:
#r "nuget: CliWrap, 3.6.0"
#r "nuget: System.Reflection.Metadata"
#r "System.Security.Cryptography"

open System
open System.IO
open System.Reflection.Metadata
open System.Reflection.PortableExecutable

let getMvid refDll =
    use embeddedReader = new PEReader(File.OpenRead refDll)
    let sourceReader = embeddedReader.GetMetadataReader()
    let loc = sourceReader.GetModuleDefinition().Mvid
    let mvid = sourceReader.GetGuid(loc)
    mvid

let getFileHash filename =
    use sha256 = System.Security.Cryptography.SHA256.Create()
    use stream = File.OpenRead(filename)
    let hash = sha256.ComputeHash(stream)
    BitConverter.ToString(hash).Replace("-", "")

open CliWrap

let argsFile =
    // FileInfo(@"C:\Users\nojaf\Projects\main-fantomas\src\Fantomas.Core\Fantomas.Core.args.txt")
// FileInfo(@"C:\Users\nojaf\Projects\fsharp\src\Compiler\FSharp.Compiler.Service.args.txt")
    // FileInfo(@"C:\Users\nojaf\Projects\fsharp\src\FSharp.Core\FSharp.Core.args.txt")
    FileInfo(@"C:\Users\nojaf\Projects\graph-sample\GraphSample.args.txt")

let total = 50

type CompilationResultInfo =
    {
        Mvid: Guid
        BinaryFileHash: string
        PdbFileHash: string option
    }

    override x.ToString() =
        let mvid = x.Mvid.ToString("N")

        let pdb =
            match x.PdbFileHash with
            | None -> ""
            | Some pdb -> $", pdb: {pdb}"

        $"mvid: {mvid}, binary: {x.BinaryFileHash}{pdb}"

[<RequireQualifiedAccess>]
type CompilationResult<'TResult when 'TResult: equality> =
    | None
    | Stable of result: 'TResult * times: int
    | Unstable of initial: 'TResult * times: int * variant: 'TResult

let oldFiles (argsFile: FileInfo) =
    let outputFile =
        File.ReadAllLines(argsFile.FullName)
        |> Array.tryPick (fun line ->
            if not (line.StartsWith("-o:")) then
                None
            else
                let objPath = line.Replace("-o:", "")

                let objPath =
                    if File.Exists objPath then
                        objPath
                    else
                        Path.Combine(argsFile.Directory.FullName, objPath)

                FileInfo(objPath) |> Some
        )

    match outputFile with
    | None -> Seq.empty
    | Some outFile ->
        seq {
            yield! Directory.EnumerateFiles(outFile.Directory.FullName, "*.dll")
            yield! Directory.EnumerateFiles(outFile.Directory.FullName, "*.pdb")
        }

for file in oldFiles argsFile do
    File.Delete(file)

let runs =
    (CompilationResult.None, [ 1..total ])
    ||> List.fold (fun (prevResult: CompilationResult<CompilationResultInfo>) idx ->
        match prevResult with
        | CompilationResult.Unstable _ -> prevResult
        | _ ->

        try
            let args = $"@{argsFile.Name}"

            Cli
                .Wrap(
                    @"C:\Users\nojaf\Projects\safesparrow-fsharp\artifacts\bin\fsc\Release\net7.0\win-x64\publish\fsc.exe"
                )
                .WithWorkingDirectory(argsFile.DirectoryName)
                .WithArguments($"\"{args}\" --test:GraphBasedChecking --test:DumpCheckingGraph --debug:portable") // --debug-
                .ExecuteAsync()
                .Task.Wait()

            let binaryPath =
                let binaryPath = File.ReadAllLines(argsFile.FullName).[0].Replace("-o:", "")

                if File.Exists binaryPath then
                    binaryPath
                else
                    Path.Combine(argsFile.DirectoryName, binaryPath)

            let binary = FileInfo(binaryPath)
            let binaryHash = getFileHash binary.FullName
            let mvid = getMvid binary.FullName

            let pdbFile: FileInfo option =
                let path = Path.ChangeExtension(binary.FullName, ".pdb")

                if not (File.Exists path) then
                    None
                else
                    Some(FileInfo(path))

            let result =
                {
                    Mvid = mvid
                    BinaryFileHash = binaryHash
                    PdbFileHash = pdbFile |> Option.map (fun fi -> getFileHash fi.FullName)
                }

            printfn $"Compiled %02i{idx}, write date %A{binary.LastWriteTime}, result: {result}"

            let renameToRun (file: FileInfo) =
                let differentPath =
                    Path.Combine(
                        file.Directory.FullName,
                        $"{Path.GetFileNameWithoutExtension(file.Name)}-{idx}{file.Extension}"
                    )

                File.Move(file.FullName, differentPath)

            renameToRun binary
            Option.iter renameToRun pdbFile

            match prevResult with
            | CompilationResult.Unstable _
            | CompilationResult.None _ -> CompilationResult.Stable(result, 1)
            | CompilationResult.Stable(prevResult, times) ->
                if prevResult <> result then
                    CompilationResult.Unstable(prevResult, times, result)
                else
                    CompilationResult.Stable(prevResult, times + 1)
        with ex ->
            printfn "%s" ex.Message
            prevResult
    )

printfn "%A" runs

At this point, I really really want someone to point out to me where there still is an actual problem.
For all we know is the problem we saw pre #14703 is just related to the usage of Proto or some other weird compiler-only factor.

@nojaf
Copy link
Contributor Author

nojaf commented Feb 15, 2023

Determinism failed for the following binaries:
FSharp.Compiler.Service.dll (old hash: 5DE8920DD1BD7BD52C169EAC0AFD8C10; new hash: 040C66612A969FBD913FD36A05F2B3A1)
Archiving failure information
Please send D:\a_work\1\s\artifacts\log\Debug\determinism.zip to compiler team for analysis

Ok, that settles it 🙃

@vzarytovskii
Copy link
Member

Determinism failed for the following binaries:
FSharp.Compiler.Service.dll (old hash: 5DE8920DD1BD7BD52C169EAC0AFD8C10; new hash: 040C66612A969FBD913FD36A05F2B3A1)
Archiving failure information
Please send D:\a_work\1\s\artifacts\log\Debug\determinism.zip to compiler team for analysis

Ok, that settles it 🙃

Oh...Let me download it and dig around pdbs. I suspect it may be as simple as ordering of locals for document.

@vzarytovskii
Copy link
Member

Okay, in the specific run it was failing, the difference is in optimization data for sure, havent' looked at PDB yet

@nojaf
Copy link
Contributor Author

nojaf commented Feb 15, 2023

Can that optimization data be written to disk as a separate file?

@vzarytovskii
Copy link
Member

Can that optimization data be written to disk as a separate file?

You can save it from ILSpy

@vzarytovskii
Copy link
Member

vzarytovskii commented Feb 15, 2023

Ok, my bad, it was not an optimization data, but signature data, again 1 byte difference (with offset of one AB -> AC).

@vzarytovskii
Copy link
Member

DLLs themselves differ in MVID and a bunch more (PDB perhaps)

@vzarytovskii
Copy link
Member

Alright, PDBs have a handful of changes, not sure which parts just yet:
image

@nojaf
Copy link
Contributor Author

nojaf commented Feb 15, 2023

Ok, when compiling for Release, I see inconsistent results for FSharp.Compiler.Service locally.
Sometimes the mvid, occasionally only the file hash of the binary.

@nojaf
Copy link
Contributor Author

nojaf commented Apr 4, 2023

Superseded by #15003

@nojaf nojaf closed this Apr 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

2 participants