Most performant way of getting typed AST of single file edited after full parsing

[alfonsogarciacaro]

One of the selling points of Fable is it works very well in watch mode as frontend developers are used to work. However, I still don't know what's the best way to get the typed AST of a single file edited after fully parsing a project. Right now I'm using ParseAndCheckProject for both cases (first project load and file edits after that), the only difference is I reuse the ProjectOptions for watch compilations.

Is there a better way to do this? I've seen there're some methods like ParseFileInProject but it doesn't give the FSharpImplementationFileContents. Also, should I set LoadTime in ProjectOptions to any specific value?

Thanks! (Pinging @dsyme for help 😄 )

[dsyme]

@alfonsogarciacaro If you want FSharpImplementationFileContents then ParseAndCheckProject is correct right now

If you want a single file - I suppose we could add FSharpImplementationFileContents to the results from ParseFileInProject

[alfonsogarciacaro]

Probably being able to parse a single file would be useful though some cases may be a bit complicated (for example, the edited file may have to access the AST of another file to inline a method, which may or may have not have changed as well).

My question is mainly if there's any performance trick I may be overlooking as I don't know well how the cache works. Does it depend on the LastWriteTime of the source files? Does the LoadTime property of ProjectOptions affect the cache somehow? If so, what's the proper way to set it?

[alfonsogarciacaro]

@dsyme How difficult would it be to add FSharpImplementationFileContents to ParseFileInProject? Would it be able to also detect changes in dependent files?

[dsyme]

@dsyme How difficult would it be to add FSharpImplementationFileContents to ParseFileInProject?

I think it would be easy to add it to CheckFileInProject

Would it be able to also detect changes in dependent files?

Yes, if they have been saved to disk

[alfonsogarciacaro]

That's great news! It's not urgent but probably parsing a single file will bring a big performance gain, as currently Fable merges sources of all dependencies in a single file so parsing the whole project on every file save can get a bit slow.

I'll try to have a look at it myself after FableConf and will ask for pointers if necessary 😄

[vasily-kirichenko]

Wow, it should be fixed.

[forki]

@vasily-kirichenko go for it!

[forki]

I personally think this would be super valuable for Fable.
In our usual projects we have many "fable base libraries" that we import via nuget source code packages. These libs never change during watch mode and are always imported at the top. So even if I change the very first line of my own code these libs are not touched. So a version of the FCS which only starts rechecking from that line already will be a big perf win.

[vasily-kirichenko]

Changing single file results with the following output:

webpack: Compiling...
Parsing ./EsExplorer.fsproj...
fable: Compiled src\App.fs
Hash: 37c448bdf0c57a51d758
Version: webpack 3.6.0
Time: 6317ms
        Asset     Size  Chunks                    Chunk Names
    bundle.js  2.31 MB       0  [emitted]  [big]  main
bundle.js.map  2.32 MB       0  [emitted]         main
 [250] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
 [255] ./src/App.fs 4.05 kB {0} [built]
    + 464 hidden modules
webpack: Compiled successfully.

Does it mean a single file has been type checked? Or it still checked the entire project?

[alfonsogarciacaro]

Every time you see the Parsing XXX.fsproj... it means the whole project is being checked. I didn't find another way to get the FSharpImplementationFileContents which contains the AST :/

[vasily-kirichenko]

How to debug the compiler? So I have Fable.sln opened in Rider. It builds successfully. How can I replace the compiler which is used in a project with the one just built and attach with the debugger to it? Or the dev flow is completely different?

[alfonsogarciacaro]

I don't use the debugger, because when I compile Fable with Debug config I get ehem SO exceptions more often than not (it works if I target .NET Framework, I removed the .fsproj files for that recently but I can add them again).

My workflow with FAKE is usually to make the modifications in the code and then run QuickFableCompilerTest which compiles Fable and then uses it to compile src/tools/QuickTest.fs (there're some more comments in that file) so I can check the output (src/tools/temp/QuickTest.js) or use printfn statements were necessary. With VS for Mac I have this dotnet-fable.fsproj.user file so I just need to press F5:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup Condition=" '$(RunConfiguration)' == 'Default' ">
    <StartAction>Project</StartAction>
    <StartArguments>npm-run rollup --port free</StartArguments>
    <StartWorkingDirectory>..\..\tools</StartWorkingDirectory>
    <ExternalConsole>true</ExternalConsole>
  </PropertyGroup>
</Project>

[vasily-kirichenko]

After some thinking about it I've come to conclusion that it's impossible to speed up this. This is how I see the current implementation:

1. A *.fs file changed and saved to disk.
2. The mailbox processor got a message which path field is set to the file https://github.com/fable-compiler/Fable/blob/master/src/dotnet/dotnet-fable/StateUtil.fs#L199
3. The MB calls StateUtil.updateState, which calls ParseAndCheckProject and pass the result to transformFile along with the file name https://github.com/fable-compiler/Fable/blob/master/src/dotnet/dotnet-fable/StateUtil.fs#L153 https://github.com/fable-compiler/Fable/blob/master/src/dotnet/dotnet-fable/StateUtil.fs#L75
4. It finds the related FSharpImplementationFileContents in checkedProject.AssemblyContents.ImplementationFiles and uses it to generate JavaScript from it https://github.com/fable-compiler/Fable/blob/master/src/dotnet/Fable.Compiler/FSharp2Fable.fs#L1328

Why the optimization won't work:

1. Instead of calling ParseAndCheckProject we call ParseAndCheckFileInProject, which does not compile files below this one.
2. We store FSharpImplementationFileContents (which is not a part of FSharpFileCheckResults yet) somewhere in a map by file name.
3. We transform the file impl content to js.
4. Now we have a broken project: if *.fs files below the current one are now non-compilable due to changes made in the current file (i.e. it's API has changed), then ParseAndCheckFileInProject won't fail like ParseAndCheckProject would, and we have inconsistent JavaScript code.

Does it make sense or I don't understand something?

/cc @dsyme

[vasily-kirichenko]

(AFAIK ParseAndCheckProject is smart and rechecks files starting and below the top changed file only)