Help wanted: how to run ExprTests in CompilerService, keep getting The exception has been reported. This internal exception should now be caught at an error recovery point on the stack. Original message: Exception of type 'FSharp.Compiler.ErrorLogger+UnresolvedPathReferenceNoRange' was thrown.)

[abelbraaksma]

Blocks: #9549

I've tried several ways to run these tests. From the commandline, I believe they're part of the FSharpQA tests, and they crash, plus the failing tests just say "failed". When running them from the Tests window in VS, I found that they all fail in GetOptimizedAssemblyContents() with the error:

The exception has been reported. This internal exception should now be caught at an error recovery point on the stack. Original message: Exception of type 'FSharp.Compiler.ErrorLogger+UnresolvedPathReferenceNoRange' was thrown.)

I don't understand this error...

Repro steps

Use this PR: #9549. Then:

Rebuild All -> Run test "Test Operator Declarations for Byte" (or any other test in that file).

The error does not occur if I use a master branch, I assume that's because the generated IL is different. But instead of getting an error on the expected output, I get the error shown above.

Expected behavior

That there's at least one way of running these tests properly, and seeing output on what fails and why. CI raises the same exception, see for instance this log dump: https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/715859/logs/117 (search for FSharp.Compiler.ErrorLogger+ReportedError).

Actual behavior

The following line (bottom) always seems to balk at me:

fsharp/tests/service/ExprTests.fs

Lines 941 to 951 in b17ecf2

	let args = mkProjectCommandLineArgsSilent (dllName, [fileName])
	let options = checker.GetProjectOptionsFromCommandLineArgs (projFileName, args)
	let wholeProjectResults = exprChecker.ParseAndCheckProject(options) |> Async.RunSynchronously
	for e in wholeProjectResults.Errors do
	printfn "%s Operator Tests error: <<<%s>>>" dnName e.Message
	wholeProjectResults.Errors.Length |> shouldEqual 0
	let fileUnoptimized = wholeProjectResults.AssemblyContents.ImplementationFiles.[0]
	let fileOptimized = wholeProjectResults.GetOptimizedAssemblyContents().ImplementationFiles.[0]

Known workarounds

Not yet...

Related information

I also notice that more often than not, Visual Studio keeps a lock on DummyProviderForLanguageServiceTesting.dll, which makes debug-fix-rebuild impossible (closing VS doesn't always help here...). This sounds like there needs to be a try-finally added somewhere.

Running the tests through Ctrl+F5 in a new VS 2019 instance leads to the same error.

I think this function and tests were introduced here: #3784, @forki, do you know why I keep getting the UnresolvedPathReferenceNoRange and what to do about it? It sounds like some reference is not what is expected.

[cartermp]

These tests should be run by the FCS test suite, not fsharpQA. Though I've found that they're very troublesome in the past. I think we should disable them.

[abelbraaksma]

These tests should be run by the FCS test suite, not fsharpQA.

That's probably my misunderstanding. It's quite hard to find out what sets of tests are run with what command, from the command line.

Though I've meanwhile established that the error is just extremely misleading, and probably should say something like "expected code to be so and so, but found code to be so and so".

I think we should disable them.

@cartermp While that would certainly allow certain PRs to pass more easily, if I understand these tests correctly, they seem to check resulting code after optimizations (but I fail to see exactly how it does so, hence I've no good idea how to fix them).

And that's probably the main problem here, it's just excruciatingly hard to even understand how these tests run and what they do. There's no single responsibility principle with these tests. Maybe there's an instruction hidden somewhere, but certainly not in that file. If I knew what the ranges do in these tests, and how they resolve, and how to check results when they fail, then at least I (and others) can maintain them.

The current error still has me puzzled, I found it that it's thrown by the optimizer, as a result of the data given from the test (which is turned into an ad-hoc project in a temp dir). But it's that data that I don't understand.

[cartermp]

I think these tests are of dubious usefulness. They don’t actually test optimizations in the compiler, just that some data gets emitted a certain way with the Expr API.

[abelbraaksma]

@cartermp Aha! So it's about Expr (I could've gotten that from the name ExprTests.fs 😆). Which means it should have nothing to do with the change in emitted IL.

With that in mind, I took another approach:

• Change a test using master branch. I get a normal Assert.Fail dumping the strings that are different and showing a trace from shouldEqual. This is expected.
• Change back to my branch which changed the string function. Removed the lines with Operators.ToString in Expr.Tests.fs. This still gives me the UnresolvedPathReferenceNoRange error.

Back to the drawing board, and back to the idea that indeed a reference is missing. This is also true for the CI build. Which suggests this may be a genuine error that needs to be fixed.

Seems that what goes around comes around, I had exactly the same issue 4 years ago: #1641. And there appear to be more issues around UnresolvedPathReferenceNoRange. If only I knew what path is unresolved, or what range (whatever Range means here).

[abelbraaksma]

@cartermp, I finally managed to find where this originates. It turns out that the test cannot find System.Numerics.BigInteger:

(and I'll see if I can change the tests that if this happens, this info is at least dumped to the console, regardless of whether they are going to be removed at some point)

[abelbraaksma]

And exactly the same stack trace here, plus a hint of how to solve it: fsprojects/FSharp.Formatting#218

[abelbraaksma]

@cartermp Ok, this is worrysome:

• System.Numerics should be on any system path, and in fact it is
• If I add a literal path where I know that there's a System.Numerics.dll, it says "Assembly reference ' System.Numerics.dll' was not found or is invalid".

This is how literal I tried to be to solve it:

let args = [|
    yield! mkProjectCommandLineArgsSilent (dllName, [fileName])
    yield @"-r: System.Numerics.dll"
    yield @"-I:""C:\Users\Abel\.nuget\packages\microsoft.netframework.referenceassemblies.net472\1.0.0-preview.2\build\.NETFramework\v4.7.2"""

let options =  checker.GetProjectOptionsFromCommandLineArgs (projFileName, args)
|]

I'm now officially lost. All this (and variants of the above) work fine from a commandline. Other places in this test project use similar methods. It feels more like there's something wrong in finding System.Numerics itself, as really, it should be there...

[abelbraaksma]

There's more at hand here. Other tests in the CompilerService directory have related problems:

• If a test compiles with a dependency on FSharp.Core.dll in CSharpProjectAnalysis.fs, they fail with an error "Cannot print exception string because Exception.ToString() failed". I may have found the cause for this: the type exception by default does not override ToString nor Message. The inherited Message calls GetClassName(), which calls GetType() on the exception. This type is then not available, leading to the very confusing error from .NET. I think this is both a mistake on F#'s side (not overriding Message) and .NET's side, calling GetType() without safety guards.
• If a test compiles that uses something from FSharp.Core.dll that itself depends on another assembly, like here System.Numerics, this leads to errors, as the reference is not automatically resolved in these tests, maybe because there's -noframework present.
• A lot of tests here are disabled for .NET Core, probably precisely for these, or similar reasons?

  #if NETCOREAPP
  [<Ignore("SKIPPED: need to check if these tests can be enabled for .NET Core testing of FSharp.Compiler.Service")>]
  #endif

My troubles with ExprTests.fs seem to fall in the same category, as I run them with .NET Core. EDIT: wrong, I run them on .NET Framework, that seems to be the default for these tests.

Apart from these tests being problematic (I think there's stuff we can do to make them more predictable, or give better info when they fail), these are real problems that need to be solved. I will try to fix the references issue, as that will go a long way.

The problem with the exception type is of a different nature and is not entirely unique to F#, see this report: dotnet/runtime#6299, and specifically this comment: dotnet/runtime#6299 (comment).

[abelbraaksma]

Ok, ok, ok, ok, sigh! Watch this:

yield @"-r: System.Numerics.dll"

And this:

yield @"-r:System.Numerics.dll"

The devil exists and he's called "space devil" 😈

Why oh why aren't spaces stripped? Ah well, that's for another issue...

I've now also fixed some of these tests to show meaningful output on fail. I'll commit stuff to a PR tomorrow.

[abelbraaksma]

Just a reminder to myself: the fix helps with a bunch of issues with these tests, but not all. Still more work to do, I'll discuss that with @vzarytovskii.