12322: Fix deep recursive expression processing

[dsyme]

#12322 is a related to processing very large expressions recursively, where these expressions are not already recognised as "linear".

Normally when processing F# expressions the depth is small, e.g. 10, though for some known "linear" sequences such as long lists of let bindings, the depth may be arbitrarily large. We already code the compiler to handle these.

However some potentially-linearizable expressions are not detected and it's really hard to detect and handle them properly systematically. Some examples include

• Very long pipelines A |> A |> A ...
• Very long computation expressions thing { 1;2;3;4;.. .}
• Very deep hand-written expressions M(M(M(M(M....)

We can at least guard against this kind of problem by jumping to a new thread stack when non-linear processing of deeply recursive is detected. This isn't ideal but it works.

Overall this protects us against the real problem of #12322 which is that stack depth varies on the six configurations of running our compiler

• 32-bit .NET Framework fsc.exe
• 64-bit .NET Framework fscAnyCpu.exe/devenv.exe
• 64-bit .NET 6 fsc.dll

It does this by making much more of our deep-stack processing independent of actual available stack size.

Testing:

• I've done manual testing to uncover the problematic recursive processing for the case of long computation expressions from 64-bit fscAnyCpu.exe uses more stack than 32-bit fsc.exe (was: Project that builds in VS2019 fails to build in VS2022) #12322
• Add stress tests for each of the three cases above
• Run them in both debug and release mode
• Run them when generating both optimized and debug code
• Run them in all three of net472 fsc.exe, net472 fscAnyCpu.exe and .NET 6 fsc.dll
• Run them in all three of net472 fsi.exe, net472 fsiAnyCpu.exe and .NET 6 fsi.dll
• Test with "full" and "portable" PDB generation

Manual testing notes for the whole matrix (except Windows v. Linux v. macOS)

# Debug compiler, release codegen, no debug 
del test.exe && artifacts\bin\fscAnyCpu\Debug\net472\fscAnyCpu.exe tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && artifacts\bin\fsc\Debug\net472\fsc.exe tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && dotnet artifacts\bin\fsc\Debug\net5.0\fsc.dll tests/fsharp/regression/12322/test.fsx  && dir test.exe

# Debug compiler, debug codegen, debug full
del test.exe && artifacts\bin\fscAnyCpu\Debug\net472\fscAnyCpu.exe --optimize- --debug:full --tailcalls- tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && artifacts\bin\fsc\Debug\net472\fsc.exe --optimize- --debug:full --tailcalls- tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && dotnet artifacts\bin\fsc\Debug\net5.0\fsc.dll --optimize- --debug:full --tailcalls- tests/fsharp/regression/12322/test.fsx  && dir test.exe

# Debug compiler, debug codegen, debug portable
del test.exe && artifacts\bin\fscAnyCpu\Debug\net472\fscAnyCpu.exe --optimize- --debug:portable --tailcalls- --define:PORTABLE_PDB tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && artifacts\bin\fsc\Debug\net472\fsc.exe --optimize- --debug:portable --tailcalls- --define:PORTABLE_PDB tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && dotnet artifacts\bin\fsc\Debug\net5.0\fsc.dll --optimize- --debug:portable --tailcalls- --define:PORTABLE_PDB tests/fsharp/regression/12322/test.fsx  && dir test.exe

# Debug fsi, normal codegen
artifacts\bin\fsi\Debug\net472\fsi.exe tests/fsharp/regression/12322/test.fsx  
artifacts\bin\fsiAnyCpu\Debug\net472\fsiAnyCpu.exe tests/fsharp/regression/12322/test.fsx  
dotnet artifacts\bin\fsi\Debug\net5.0\fsi.dll tests/fsharp/regression/12322/test.fsx  

# Release compiler, release codegen, no debug 
del test.exe && artifacts\bin\fscAnyCpu\Release\net472\fscAnyCpu.exe tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && artifacts\bin\fsc\Release\net472\fsc.exe tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && dotnet artifacts\bin\fsc\Release\net5.0\fsc.dll tests/fsharp/regression/12322/test.fsx  && dir test.exe

# Release compiler, debug codegen, debug full
del test.exe && artifacts\bin\fscAnyCpu\Release\net472\fscAnyCpu.exe --optimize- --debug:full --tailcalls- tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && artifacts\bin\fsc\Release\net472\fsc.exe --optimize- --debug:full --tailcalls- tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && dotnet artifacts\bin\fsc\Release\net5.0\fsc.dll --optimize- --debug:full --tailcalls- tests/fsharp/regression/12322/test.fsx  && dir test.exe

# Release compiler, debug codegen, debug portable
del test.exe && artifacts\bin\fscAnyCpu\Release\net472\fscAnyCpu.exe --optimize- --debug:portable --tailcalls- --define:PORTABLE_PDB tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && artifacts\bin\fsc\Release\net472\fsc.exe --optimize- --debug:portable --tailcalls- --define:PORTABLE_PDB tests/fsharp/regression/12322/test.fsx  && dir test.exe
del test.exe && dotnet artifacts\bin\fsc\Release\net5.0\fsc.dll --optimize- --debug:portable --tailcalls- --define:PORTABLE_PDB tests/fsharp/regression/12322/test.fsx  && dir test.exe

# Release fsi, normal codegen
artifacts\bin\fsi\Release\net472\fsi.exe tests/fsharp/regression/12322/test.fsx
artifacts\bin\fsiAnyCpu\Release\net472\fsiAnyCpu.exe tests/fsharp/regression/12322/test.fsx
dotnet artifacts\bin\fsi\Release\net5.0\fsi.dll tests/fsharp/regression/12322/test.fsx

[dsyme]

OK, this is ready, subject to being green. There is the automated test matrix in the PR (see tests.fs) and I've been through the same manual steps up above

[dsyme]

I took the opportunity to cleanup the test configuration matrix used in test\fsharp\tests.fs

[kentcb]

Curious that the latest preview (17.1.0 Preview 2) doesn't seem to contain this fix. Is that expected?

[baronfel]

@kentcb you'll want to check what version of the SDK is bundled with 17.1.0 Preview 2. I tested this manually with the 6.0.2xx nightly previews (which you can download from the table at the dotnet/installer repo, and the sample code in the linked issue #12322 compiled successfully for me. I don't believe this fix made it into 'servicing' for the 6.0.1xx series of SDKs, but others may have more insight there.

[kentcb]

Sorry to ping you @baronfel, but do you know who I should be bugging about this? I believe this to be a fundamental flaw that needs to be patched - I am blocked from using my project on VS22 as a result.

[vzarytovskii]

It should be in 6.0.200, an probably next dev17 preview. @brettfo do you know if it made it there?

[kentcb]

Hmm, I can successfully build my repro project from the CLI with 6.0.200-preview.21617.4, but building from within VS2022 still reproduces the issue. From what I can tell, VS should be using the same SDK as when building from CLI so I'm very confused.

[dsyme]

@kentcb By default VS uses the .NET Framework compiler. There's a way to adjust your project to use the .NET SDK compiler, (@KevinRansom can give the details or search this repo)

[mattiasw2]

Is this problem solved? I get

MSB6006 "fscAnyCpu.exe" exited with code -1073741571.

with no reference to which file the compilation fails. I am using VS2022 17.1.1

The workaround is to continue to use VS2019. Also, my command line compilations continue to work.

It is a big F# project, used to be Ocaml when created 20 years ago. Some files are very big switch statements or big functions with internal functions.

[mattiasw2]

Still not working in VS2022 17.3, solved it by adding the folloiwing to the project file for the problematic project. The source is most like a very big "switch" statement

<FSharpPreferNetFrameworkTools>true</FSharpPreferNetFrameworkTools>
<FSharpPrefer64BitTools>false</FSharpPrefer64BitTools>