Running Compile tutorial code produces StackOverflowException in ImplicitLoadIfAllowed

[adamkennedy]

I have been experimenting with dynamic assembly compilation and the results of my tests have been going really well, while I was launching the compiler from F# Interactive sessions.

However, when switching to a Visual Studio Command Line F# project with the FSharp.Compiler.Service 0.76 NuGet package (so I can move to building a production version) I am getting StackOverflowExceptions from infinite loops in the compiler service.

This occurs regardless of the code being compiled.

For example, creating a project with the Hosted Compiler example code results in the same StackOverflowException as my actual project code.

// Learn more about F# at http://fsharp.net
// See the 'F# Tutorial' project for more help.

open Microsoft.FSharp.Compiler.SimpleSourceCodeServices
open System.IO

let scs = SimpleSourceCodeServices()

[<EntryPoint>]
let main argv = 
    let fn = Path.GetTempFileName()
    let fn2 = Path.ChangeExtension(fn, ".fs")
    let fn3 = Path.ChangeExtension(fn, ".dll")

    File.WriteAllText(fn2, """
    module M

    type C() = 
       member x.P = 1

    let x = 3 + 4
    """)

    let errors2, exitCode2, dynAssembly2 = 
        scs.CompileToDynamicAssembly([| "-o"; fn3; "-a"; fn2 |], execute=None)

    // return an integer exit code
    0

I have traced the infinite loop to the following set of stack entries.

FSharp.Compiler.Service.dll!Microsoft.FSharp.Compiler.Build.TcImports.RegisterAndImportReferencedAssemblies(Microsoft.FSharp.Core.FSharpOption<Microsoft.FSharp.Core.FSharpFunc<string,Microsoft.FSharp.Core.Unit>> displayPSTypeProviderSecurityDialogBlockingUI, Microsoft.FSharp.Collections.FSharpList<Microsoft.FSharp.Compiler.Build.AssemblyResolution> nms) Line 4408   F#
    FSharp.Compiler.Service.dll!Microsoft.FSharp.Compiler.Build.tryFile@4420(Microsoft.FSharp.Compiler.Build.TcImports tcImports, Microsoft.FSharp.Compiler.Range.range m, string speculativeFileName) Line 4425    F#
    FSharp.Compiler.Service.dll!Microsoft.FSharp.Compiler.Build.TcImports.ImplicitLoadIfAllowed(Microsoft.FSharp.Compiler.Range.range m, string assemblyName, bool lookupOnly) Line 4431    F#
    FSharp.Compiler.Service.dll!Microsoft.FSharp.Compiler.Build.TcImports.FindCcuInfo(Microsoft.FSharp.Compiler.Range.range m, string assemblyName, bool lookupOnly) Line 3806  F#
    FSharp.Compiler.Service.dll!Microsoft.FSharp.Compiler.Build.RegisterAndPrepareToImportReferencedDll@4363.Invoke(Microsoft.FSharp.Core.Unit unitVar0) Line 4363  F#
    FSharp.Compiler.Service.dll!Microsoft.FSharp.Compiler.Build.TcImports.RegisterAndImportReferencedAssemblies(Microsoft.FSharp.Core.FSharpOption<Microsoft.FSharp.Core.FSharpFunc<string,Microsoft.FSharp.Core.Unit>> displayPSTypeProviderSecurityDialogBlockingUI, Microsoft.FSharp.Collections.FSharpList<Microsoft.FSharp.Compiler.Build.AssemblyResolution> nms) Line 4408   F#

The relevant code is this, from build.fs around line 4414. assemblyName is "FSharp.Core"

    member tcImports.ImplicitLoadIfAllowed (m, assemblyName, lookupOnly) = 
        CheckDisposed()
        // If the user is asking for the default framework then also try to resolve other implicit assemblies as they are discovered.
        // Using this flag to mean 'allow implicit discover of assemblies'.
        let tcConfig = tcConfigP.Get()
        if not lookupOnly && tcConfig.implicitlyResolveAssemblies then 
            let tryFile speculativeFileName = 
                let foundFile = tcImports.TryResolveAssemblyReference (AssemblyReference (m, speculativeFileName, None), ResolveAssemblyReferenceMode.Speculative)
                match foundFile with 
                | OkResult (warns, res) -> 
                     ReportWarnings warns
                     tcImports.DoRegisterAndImportReferencedAssemblies(None,res)
                     true
                | ErrorResult (_warns, _err) -> 
                    // Throw away warnings and errors - this is speculative loading
                    false

            if tryFile (assemblyName + ".dll") then ()
            else tryFile (assemblyName + ".exe")  |> ignore

If there is something that I'm doing wrong here, it's not obvious to me.

[dsyme]

I believe you have an FSharp.Core.dll alongside your HostedCompiler.exe. The compilation is implicitly referencing this FSharp.Core.dll, and if you do that then you need an FSharp.Core.{sigdata,optdata} alongside.

You either need to

(a) remove that FSharp.Core.dll (Private=false in your build) OR

(b) keep FSharp.Core.dll but also put an FSharp.Core.sigdata and FSharp.Core.optdata alongside your application OR

(c) explicitly reference an existing FSharp.Core.dll and mscorlib.dll in your compilation, e.g.

let errors2, exitCode2, dynAssembly2 = 
    scs.CompileToDynamicAssembly([| "fsi.exe"; "--noframework"; "-r"; @"C:\Program Files (x86)\Reference Assemblies\Microsoft\FSharp\.NETFramework\v4.0\4.3.1.0\FSharp.Core.dll"; "-r"; @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorlib.dll"; "-o"; fn3; "-a"; fn2 |], execute=None)

OR

let errors2, exitCode2 = 
    scs.Compile([| "fsc.exe"; "--noframework"; "-r"; @"C:\Program Files (x86)\Reference Assemblies\Microsoft\FSharp\.NETFramework\v4.0\4.3.1.0\FSharp.Core.dll"; "-r"; @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscorlib.dll"; "-o"; fn3; "-a"; fn2 |])

If you do the latter then you need to find an SDK-installed FSharp.Core.dll in a cross-platform way. Some helpers to do this are here: https://github.com/fsharp/FSharp.Compiler.Service/blob/8a943dd3b545648690cb3bed652a469bdb6dd869/tests/service/Common.fs#L54, or you can rely on cracking an F# project file. It sort of depends on whether you expect F# SDK tools to be on the target machine running your application or not.

[dsyme]

Based on this discussion I added some documentation about this issue, see #260 (and please review and/or correct it)

[moserware]

Thanks for the info Don!

I ran into this issue on my machine as well and indeed it came down to having FSharp.Core.dll referenced as private=true since its intended use is on machines that will not have the FSharp SDK installed. When I explicitly added the FSharp.Core.sigdata and FSharp.Core.optdata to the project and set them as Copy If Newer (to the output directory), everything worked great.

In addition to the documentation updates, I suggest:

1. Have an explicit compiler check for this scenario to have a better error message than a StackOverflowException (which is challenging to debug).
2. Have someone on the FSharp.Compiler.Service project (e.g. @tpetricek ?) consider adding these two references (FSharp.Core.sigdata and FSharp.Core.optdata) to the NuGet package as Copy If Newer (perhaps as an MSBUILD rule?), or maybe creating an altogether new package that depends on FSharp.Compiler.Service and FSharp.Core and could be used in these self-hosting scenarios?

Alternatively, come up with a way for self-hosted scenarios to not need these two files.

[adamkennedy]

As @moserware says, we'll go with the extra dll approach (although down the track a bit further, I want to optionally prefer the installed version if available, so that the F# files will run the same with the tool and in whatever interactive they have available)

[dsyme]

Out of curiosity can I ask what you're building? No problem if you can't say :)

[adamkennedy]

It's a build system, superficially similar to make or FAKE but intended for data science workloads rather than software/ops work, for projects up to the 100-1000 CPU hour range where parallelism, shortcutting, and custom failure tolerance are critical, but below the scale where you need to trade up to centralized server resources such as Hadoop.

It's intended for support extremely rapid iterative experimental data science workflows. We currently use make for this, but we're reaching the limits of how far we can scale it for complexity and we're struggling with load management at high CPU core counts.

The repository is currently private, and the code is experimental grade, but we've got it in a repository that we can switch to public and Open Source it when we've got something Alpha grade.

I imagine we'll end up borrowing bits and pieces of FAKE along the way.

[adamkennedy]

If you're interested, I'm quite sure @moserware would be happy to add you to the repo so you can observe/comment :)

[dsyme]

Very interesing!! Don't add me to a private repo, but it sounds great!