Crash in FSharpDocumentHighlightsService causes VS crash and System.NullReferenceException in Roslyn

[cartermp]

Okay, so I don't know how to reproduce this. I was mostly threading a g value for TcGlobals through NicePrint.fs functions and into SymbolHelpers.fs. VS got into a crash loop and crashed 7 times in ~15 minutes according to the Windows Event Viewer.

The first had this stack trace:

Application: fsiAnyCpu.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileLoadException
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at System.AppDomain.ExecuteAssemblyByName(AssemblyName assemblyName, String[] args)
   at System.AppDomain.ExecuteAssemblyByName(System.Reflection.AssemblyName, System.String[])
   at Sample.FSharp.Compiler.Interactive.Main.MainMain(System.String[])

The other 6 had this stack trace coming from Roslyn:

Application: devenv.exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Collections.Immutable.ImmutableArray.CreateRange[TSource,TResult](ImmutableArray`1 items, Func`2 selector)
   at Microsoft.CodeAnalysis.ExternalAccess.FSharp.Internal.DocumentHighlighting.FSharpDocumentHighlightsService.<GetDocumentHighlightsAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Editor.ReferenceHighlighting.ReferenceHighlightingViewTaggerProvider.<ProduceTagsAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Editor.Tagging.AbstractAsynchronousTaggerProvider`1.TagSource.<RecomputeTagsAsync>d__70.MoveNext()
Stack:
   at System.Environment.FailFast(System.String, System.Exception)
   at Microsoft.CodeAnalysis.FailFast.OnFatalException(System.Exception)
   at Microsoft.CodeAnalysis.ErrorReporting.WatsonReporter.ReportFatal(System.Exception)
   at Microsoft.CodeAnalysis.ErrorReporting.FatalError.Report(System.Exception, System.Action`1<System.Exception>)
   at Microsoft.CodeAnalysis.ErrorReporting.FatalError.Report(System.Exception)
   at Roslyn.Utilities.TaskExtensions.ReportFatalErrorWorker(System.Threading.Tasks.Task, System.Object)
   at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.Finish(Boolean)
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskExtensions+VoidResult, System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetException(System.Object)
   at System.Threading.Tasks.UnwrapPromise`1[[System.Threading.Tasks.TaskExtensions+VoidResult, System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetFromTask(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.UnwrapPromise`1[[System.Threading.Tasks.TaskExtensions+VoidResult, System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].InvokeCore(System.Threading.Tasks.Task)
   at System.Threading.Tasks.UnwrapPromise`1[[System.Threading.Tasks.TaskExtensions+VoidResult, System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Invoke(System.Threading.Tasks.Task)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.Finish(Boolean)
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetException(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetException(System.Exception)
   at Microsoft.CodeAnalysis.Editor.Tagging.AbstractAsynchronousTaggerProvider`1+TagSource+<RecomputeTagsAsync>d__70[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.Finish(Boolean)
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetException(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetException(System.Exception)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetException(System.Exception)
   at Microsoft.CodeAnalysis.Editor.ReferenceHighlighting.ReferenceHighlightingViewTaggerProvider+<ProduceTagsAsync>d__11.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.Finish(Boolean)
   at System.Threading.Tasks.Task`1[[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.DocumentHighlighting.DocumentHighlights, Microsoft.CodeAnalysis.Features, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]], System.Collections.Immutable, Version=1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].TrySetException(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.DocumentHighlighting.DocumentHighlights, Microsoft.CodeAnalysis.Features, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]], System.Collections.Immutable, Version=1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].SetException(System.Exception)
   at Microsoft.CodeAnalysis.ExternalAccess.FSharp.Internal.DocumentHighlighting.FSharpDocumentHighlightsService+<GetDocumentHighlightsAsync>d__3.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task`1[[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.ExternalAccess.FSharp.DocumentHighlighting.FSharpDocumentHighlights, Microsoft.CodeAnalysis.ExternalAccess.FSharp, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]], System.Collections.Immutable, Version=1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].TrySetResult(System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.ExternalAccess.FSharp.DocumentHighlighting.FSharpDocumentHighlights>)
   at System.Threading.Tasks.TaskCompletionSource`1[[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.ExternalAccess.FSharp.DocumentHighlighting.FSharpDocumentHighlights, Microsoft.CodeAnalysis.ExternalAccess.FSharp, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]], System.Collections.Immutable, Version=1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].TrySetResult(System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.ExternalAccess.FSharp.DocumentHighlighting.FSharpDocumentHighlights>)
   at Microsoft.VisualStudio.FSharp.Editor.RoslynHelpers+StartAsyncAsTask@113-4[[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.ExternalAccess.FSharp.DocumentHighlighting.FSharpDocumentHighlights, Microsoft.CodeAnalysis.ExternalAccess.FSharp, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]], System.Collections.Immutable, Version=1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].Invoke(System.Exception)
   at Microsoft.FSharp.Control.AsyncPrimitives+StartWithContinuations@916-2.Invoke(System.Runtime.ExceptionServices.ExceptionDispatchInfo)
   at Microsoft.FSharp.Control.Trampoline.Execute(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.Unit,Microsoft.FSharp.Control.AsyncReturn>)
   at Microsoft.FSharp.Control.TrampolineHolder.ExecuteWithTrampoline(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.Unit,Microsoft.FSharp.Control.AsyncReturn>)
   at <StartupCode$FSharp-Core>.$Async+-ctor@163-1.Invoke(System.Object)
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

It's unclear right now where the issue is:

• F# External Access not handling a bad value correctly
• F# tooling itself sending nonsense data back

We should fix this, since I can't imagine I'm the first one to have seen this before.

[cartermp]

I actually can't continue my work unless this is resolved, heh

[cartermp]

Okay, here's what's going on:

1. I managed to to get my editor into a state (due to really messing around with stuff and causing error recovery to go wild) such that there was no reasonable way we could parse out the tree and select symbols to highlight in the DocumentHighlights service.
2. In our code, if we have nothing to highlight, we return ImmutableArray<FSharpDocumentHighlights>.Empty
3. In the F# Roslyn bindings, this data is passed into a call to ImmutableArray.CreateRange: http://sourceroslyn.io/#Microsoft.CodeAnalysis.Workspaces/Collections/ImmutableArrayExtensions.cs,155
4. This triggers an NRE because it violates this contract: https://github.com/dotnet/runtime/blob/master/src/libraries/System.Collections.Immutable/src/System/Collections/Immutable/ImmutableArray.cs#L228

What I'm unclear on is why this is happening though. Why would that contract fail? Is some data corrupted?

[abelbraaksma]

We should fix this, since I can't imagine I'm the first one to have seen this before.

You're not alone ;). Though it's just really hard to get to a good repro. I got it a few times yesterday with the Fsharp.sln, when playing with the tokenizer code (usually VS restarts itself, sometimes it just disappears on me). I didn't really look into it, but maybe I still have the reports in the event viewer and could check tomorrow if they're similar.

(unfortunately, I've nothing sensible to add to your analysis, the only thing that comes to mind is memory corruption because something gets to big, but that's a wild guess and should've given an OOM before FailFast. In fact, I don't see any condition there that should trigger a FailFast, I didn't think NRE qualifies).

[abelbraaksma]

Oops, I already forgot, the latest "disappearing VS" situation was actually analyzed by me and you and ended up being caused by a weirdly behaving "Manage NuGet references" window, when opened from FSharp.sln: #9518

Please disregard the previous message, it is not applicable (though the observation the VS grinds to a halt or crashes when playing with the tokenizer-related files still holds for me though).

[vadim-kor]

Is it fixed? I started getting it on the last VS with all extensions disabled

[cartermp]

No, this can still happen from time to time. There's a very subtle bug somewhere that causes us to send corrupted data to VS.

[vadim-kor]

@cartermp
I found why it happens on my side: I created a new file that had namespace and a pasted a module that has a lot of functions connected by the "and" keyword.

The file was missing the open namespace section.
So it crashes instead of saying that there are several namespaces are missing.
It started working after I put the required namespaces.

The interesting thing is that it will not crash if I remove the namespace definition and openings and keep only module definition

[cartermp]

@vadim-kor is your code sharable at all? If you have a reliable repro and the ability to share then that would be helpful. One of the issues when I filed this was that I had no ability to re-trace back to a repro. It was after an hour or two of heavy modifications across the compiler codebase.