FSharpQA CI run failing with StackOverflowException

[T-Gro]

I am in the process of enabling language features from preview to v10.
I noticed that FSharpQA failed 3 tests with a stackoverflow, worth mentioning it uses a stateful hosted compiler process.

It could have been bad luck, but then also one of the features enabled was type subsumption cache.
@majocha : Do you think it could have had impact? All the FsharpQA tests are running within the same host, so the number of cache entries can grow (possibly even lead to large arrays that cannot be allocated due to memory fragmentation? Maybe we need to be more aware of array sizes?)

+++ Conformance\Expressions\ExpressionQuotations\Regressions (QuoteAddressOf01.fs) +++
--------------------------------------------------------
Results from hosted compiler
msg: Compiling
cmd: fsc   --test:ErrorRanges E_QuoteAddressOf01.fs 
Exit code: 1
Error:     2
Microsoft (R) F# Compiler version 13.9.100.0 for F# 10.0
Copyright (c) Microsoft Corporation. All Rights Reserved.
(1,1-1,1): error FS0229: Error opening binary file 'D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll': Exception of type 'System.OutOfMemoryException' was thrown.
(1,1-1,1): error FS3160: Problem reading assembly 'D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll': The exception has been reported. This internal exception should now be caught at an error recovery point on the stack. Original message: Error opening binary file 'D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll': Exception of type 'System.OutOfMemoryException' was thrown.)
(1,1-1,1): error FS0073: internal error: no import of D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll
--------------------------------------------------------
Microsoft (R) F# Compiler version 13.9.100.0 for F# 10.0
Copyright (c) Microsoft Corporation. All Rights Reserved.
(1,1-1,1): error FS0229: Error opening binary file 'D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll': Exception of type 'System.OutOfMemoryException' was thrown.
(1,1-1,1): error FS3160: Problem reading assembly 'D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll': The exception has been reported. This internal exception should now be caught at an error recovery point on the stack. Original message: Error opening binary file 'D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll': Exception of type 'System.OutOfMemoryException' was thrown.)
(1,1-1,1): error FS0073: internal error: no import of D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll

*** The following necessary lines were never matched:
***	\(7,16-7,17\):.+error FS0462:.+Quotations cannot contain this kind of type
***	\(16,17-16,18\):.+error FS0462:.+Quotations cannot contain this kind of type


*** The following necessary lines were incorrectly matched:
***	(1,1-1,1): error FS0073: internal error: no import of D:\a\_work\1\s\tests\fsharpqa\testenv\bin\FSharp.Core.dll


Unexpected Compiler Output 
FAIL
+++ Conformance\Expressions\ExpressionQuotations\Regressions (E_QuoteDynamic01.fs) +++
--------------------------------------------------------
Results from hosted compiler
msg: Compiling
cmd: fsc   --langversion:4.6 --test:ErrorRanges E_QuoteDynamic01.fs 
Exit code: 1
Error:     2
Microsoft (R) F# Compiler version 13.9.100.0 for F# 10.0
Copyright (c) Microsoft Corporation. All Rights Reserved.
(0,0-0,0): error FS3160: Problem reading assembly 'C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Windows.Forms.dll': Exception of type 'System.OutOfMemoryException' was thrown.
--------------------------------------------------------
Microsoft (R) F# Compiler version 13.9.100.0 for F# 10.0
Copyright (c) Microsoft Corporation. All Rights Reserved.
(0,0-0,0): error FS3160: Problem reading assembly 'C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.8\System.Windows.Forms.dll': Exception of type 'System.OutOfMemoryException' was thrown.

*** The following necessary lines were never matched:
***	\(19,17-19,18\):.+error FS0458:.+Quotations cannot contain expressions that make member constraint calls, or uses of operators that implicitly resolve to a member constraint call
***	\(22,13-22,30\):.+error FS0458:.+Quotations cannot contain expressions that make member constraint calls, or uses of operators that implicitly resolve to a member constraint call


*** The following necessary lines were incorrectly matched:

Unexpected Compiler Output 
FAIL
+++ Conformance\Expressions\ExpressionQuotations\Regressions (E_QuotationOperators02.fs) +++
--------------------------------------------------------
Results from hosted compiler
msg: Compiling
cmd: fsc   --test:ErrorRanges E_QuotationOperators02.fs 
Exit code: 0
Error:     2
--------------------------------------------------------

*** The following necessary lines were never matched:
***	\(6,9-6,13\):.+error FS0010:.+Unexpected infix operator in binding


*** The following necessary lines were incorrectly matched:

Unexpected Compiler Output 

[T-Gro]

I will rerun it now and see if it happens again.

[majocha]

There was a env var to limit cache sizes. My guess is we need to set it manually for the fsharpqa. Also, isn't it running in 32bits? If so, its certainly not helping.

[majocha]

Yeah, the env variable to limit cache size is named FSHARP_CACHE_OVERRIDE. It needs to be set manually for fsharpQA and testVs because both of them run out of memory now.