Proposal: Type provider design-time DLLs should be chosen more appropriately

[dsyme]

Recent PRs in the Type Provider SDK repo (e.g. fsprojects/FSharp.TypeProviders.SDK#139) are a big step towards completing our type provider story for .NET Core With this work, properly written type providers using the latest version of the TPSDK can now always produce target code for either .NET Core or .NET Framework – i.e. they properly adjust for the target reference assemblies being used by the compilation. This applies to both generative and erasing type providers.

One final piece of the type provider puzzle is to have the F# compiler load an appropriate design-time DLL depending on whether the compiler is running in .NET Core or .NET Framework.

Currently, the compiler looks for design time DLL alongside the referenced runtime DLLs. (For simple type providers, these DLLs are the same)

We will do something like what is specified her https://docs.microsoft.com/en-us/nuget/schema/analyzers-conventions,

Rough proposal

When a referenced assembly as the usual TypeProviderAssembly attribute, indicating it wants design-time type provider component “MyDesignTime.dll”, then

1. When executing using .NET Core the compiler looks in this order

   ...\typeproviders\fsharpNN\netcoreapp2.0\ARCH\MyDesignTime.dll
   ...\typeproviders\fsharpNN\netcoreapp2.0\MyDesignTime.dll 
   ...\typeproviders\fsharpNN\netstandard2.0\ARCH\MyDesignTime.dll
   ...\typeproviders\fsharpNN\netstandard2.0\MyDesignTime.dll 
   MyDesignTime.dll 

2. When executing using .NET Framework the compiler looks in this order

    ...\typeproviders\fsharpNN\net461\ARCH\MyDesignTime.dll
    ...\typeproviders\fsharpNN\net461\MyDesignTime.dll
    ...\typeproviders\fsharpNN\netstandard2.0\ARCH\MyDesignTime.dll
    ...\typeproviders\fsharpNN\netstandard2.0\MyDesignTime.dll
    MyDesignTime.dll 

relative to the location of the runtime DLL, which is presumed to be in a nuget package.

• When we use ... we mean a recursive upwards directory search looking for a directory names typeproviders, stopping when we find a directory name packages

• WHen we use fsharpNN we mean a successive search backwards for fsharp42, fsharp41 etc. Putting a TPDTC in fsharp41 means the TPDTC is suitable to load into F# 4.1 tooling and later, and has the right to minimally assume FSharp.Core 4.4.1.0

Some examples:

The idea is that a package can contain a type provider design-time for both .NET Core and .NET Framework. This will allow it to load into both compiler, F# Interactive and tools running .NET Framework OR .NET Core.

Example 1 - the ultimate simplest form

Going forward, the very simplest type providers will eventually use a single .NET Standard 2.0 DLL for both the runtime library and the design-time component. Layout:

      lib\netstandard2.0\MyRuntimeAndDesignTime.dll 

As of today however you will also need to ship facades for netstandard.dll, System.Runtime.dll and System.Reflection.dll alongside this component and as such you should either

• adopt one of the layouts below to ensure facade extra files don't end up in the lib directory, OR
• put the facades in a well-known relative location (not under the lib directory in the package) and added AssemblyResolve events to go and find them

Example 2 - FSharp.Data

FSharp.Data would lay out as follows:

    lib\net45\FSharp.Data.dll 
    lib\netstandard2.0\FSharp.Data.dll 
    lib\portable-whatever\FSharp.Data.dll 

    typeproviders\fsharp40\net461\FSharp.Data.DesignTime.dll  
    typeproviders\fsharp40\netcoreapp2.0\FSharp.Data.DesignTime.dll  

Here we are assuming FSharp.Data wants to use different design time DLLs for the two host tooling contexts we care about. Example 3 deals with the case where FSHarp.Data wants to use a single DLL.

Example 3 - FSharp.Data (with .NET Standard design-time DLL)

A layout like this may also be feasible if shipping facades to create a .NET Standard 2.0 TPDTC

    lib\net45\FSharp.Data.dll 
    lib\netstandard2.0\FSharp.Data.dll 
    lib\portable-whatever\FSharp.Data.dll 

    typeproviders\fsharp41\netstandard2.0\FSharp.Data.DesignTime.dll  

See note on facades above.

Example 4 - type providers with 32/64-bit dependencies

A Python type provider may have different dependencies for 32 and 64-bit for both runtime and design-time (the directory names may not be exactly right)

    lib\netstandard2.0\x86\FPython.Runtime.dll 
    lib\netstandard2.0\x64\FPython.Runtime.dll 

    typeproviders\fsharp41\netstandard2.0\x86\FPython.Runtime.dll 
    typeproviders\fsharp41\netstandard2.0\x86\cpython32.dll # some 32-bit DLL needed to run at design-time

    typeproviders\fsharp41\netstandard2.0\x64\FPython.Runtime.dll 
    typeproviders\fsharp41\netstandard2.0\x64\cpython64.dll # some 64-bit DLL needed to run at design-time

plus facades as mentioned above

Non Example 5

Going forward, we should not be happy with type provider packages that look like this - these will be unusable when the compiler executes using .NET Core

    lib\net45\MyTypeProvider.dll 

or even

    lib\net45\MyTypeProvider.dll 
    typeproviders\fsharp40\net45\MyTypeProvider.DesignTime.dll

Again if your TPDTC component is a solitary NET Framework component then it will be unusable when a host tool executes using .NET Core.

Remember, we must be able to load type provider design-time DLLs into many different tooling contexts, notable

• FsAutoComplete.exe .NET Framework running 64-bit
• fsc.exe running on .NET Core 2.0 as either 64-bit or 32-bit
• fsi.exe running on .NET Framework as 32-bit
• fsiAnyCpu.exe running on .NET Framework as either 64-bit or 32-bit
• fsc.exe running on .NET Framework as either 64-bit or 32-bit
• devenv.exe running on .NET Framework 32-bit

and any context that uses FSharp.Compiler.Service.dll as either netstandard 2.0 or a .NET Framework component.

Note the above is effectively a proposed package architecture for any cross-generating compiler plugins. If we ever support other kinds of compiler plugins then we could follow a similar pattern.

[matthid]

In (C#) Roslyn-Analyzer world is it really the compiler figuring out the paths?
To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...
What about dependencies of the Design-Time assembly?

[dsyme]

In (C#) Roslyn-Analyzer world is it really the compiler figuring out the paths?

No, it's just a related example

To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...

I think its inevitable in some form. The referenced runtime DLL and the design-time DLL live in the same package and we really have to separate them, but equally be able to locate the latter from the former.

What about dependencies of the Design-Time assembly?

The design-time DLL is loaded using LoadFrom and must carry all its dependencies in the same directory as the DLL (or loadable using standard binding rules after a LoadFrom). It must be a standalone "reflection-loaded adding".

One huge advantage of the above is that each design-time configuration (e.g. x86 and x64) can now carry different dependencies.

Note the situation today is horrendous - the dependencies of the design-time components all end up in the "lib" directory, and mistakenly get added as references. And people hack the search through neighouring packages in terrible ways, e.g. https://github.com/BlueMountainCapital/FSharpRProvider/blob/master/src/RProvider/RProvider.fsx

[dsyme]

To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...

Note the other alternative would be to have logic in our MSBuild targets file and add a first-class /typeproviderreference:MyDesignTime.dll flag to the compiler.

However I think that's very obscure and won't work for tools like F# Interactive (fsi.exe), which must locate type provider design-time components without resorting to using MSBuild. In particular I don't really want people to have to use

#r "packages\FSharp.Data\lib\net45\FSharp.Data.dll"
#tpr "packages\FSharp.Data\typeproviders\netstandard2.0\FSharp.Data.DesignTime.dll"

in their F# scripts - though admittedly it's being very honest about what's going on. We would end up baking just the same logic in to the compiler in order to give people a good error message telling them where to find the design-time DLL of they forget to add the #tpr reference

[matthid]

Note the other alternative would be to have logic in our MSBuild targets file and add a first-class "/typeproviderreference:MyDesignTime.dll" flag to the compiler. However I think that's very obscure and won't work for tools like F# Interactive (fsi.exe),

To be honest that sounds a lot better and if we add a corresponding #typeproviderr (or however we call it) to F# scripting, we can extend Paket to properly add those into the load scripts including the proper dependencies. This way we leave resolving dependencies to the caller (as it should be).

which must locate type provider design-time components without resorting to using MSBuild.

Yes but a compiler solution introduces a lot of other problems

• what if multiple type providers bundle the same dependencies?
• Framework compatibility and fallback is not too simple to implement correctly, all that needs to be duplicated in the F# compiler (or otherwise we will miss a lot in our compatibility list)
• and it feels wrong

On the other side thinking about supporting design time and runtime dependencies in Paket makes my head burn. (In practice a unified resolution will most likely be fine / we might add special supports in our groups). Also there is a problem with detecting those dependencies: Is a dependency of a type provider package a runtime or a designtime dependency? But at least that will not be a problem of the compiler.

[matthid]

which must locate type provider design-time components without resorting to using MSBuild.

And it will be solved via #2483 ;)

[smoothdeveloper]

@matthid:

To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...

I see it not as NuGet implementation, but as providing a stable layout relative to the runtime assembly.

To be honest that sounds a lot better and if we add a corresponding #typeproviderr (or however we call it) to F# scripting, we can extend Paket to properly add those into the load scripts including the proper dependencies. This way we leave resolving dependencies to the caller (as it should be).

This could be done leveraging #2483:

#r "typeprovider:path/to/FSharp.Data.dll"

paket in the load script generation would identify assemblies exposing type providers and add the reference in that way rather than simple assembly reference.

The typeprovider resolution handler would be implemented with the conventions @dsyme was sketching, it would help conceal all the implementation details, and keep the manual work from a simple script to get the reference working minimal (unlike @dsyme sample in #3736 (comment))

Yes but a compiler solution introduces a lot of other problems

Lots of valid concerns.

[dsyme]

On the other side thinking about supporting design time and runtime dependencies in Paket makes my head burn. (In practice a unified resolution will most likely be fine / we might add special supports in our groups). Also there is a problem with detecting those dependencies: Is a dependency of a type provider package a runtime or a designtime dependency? But at least that will not be a problem of the compiler.

I don't expect Paket or Nuget will ever seriously get into the business of managing dependencies of compile-time tools and add-ins. At least not in a timeframe that's useful. Indeed I don't think they could - they just don't know enough about host tooling. We could maybe push something bespoke into Paket, but its only use cases would be for F# type-provider add-ins, so there's not much point.

I'm also not terribly keen in hiding resolution logic in MSBuild for similar reasons. On the whole I'd rather have this stuff "close to home" where we have control over it, rather than pushing the problem off to Paket, Nuget etc. If it inspires Nuget to come up with a way of managing compile-time tool dependency chains then that's good and we could eventually switch to using the results, but I think that's unlikely. Even for analyzers mentioned above there is no compile-time dependency management - I think it's assumed each analyzer is independently loadable

what if multiple type providers bundle the same dependencies?

That problem exists in any situation. Type provider design-time components (let's call those TPDTCs) are loaded into host tooling using Assembly.LoadFrom (regardless of how they are specified and found). There is no other choice (unless we either isolate them in a process - AppDomains are no longer an option on .NET Core).

This causes problems - for example, TPDTCs may also not have binding redirects (except ones implemented via adhoc AssemblyResolve events, and ones provided by host tooling, e.g. a redirect to a recent version of FSharp.Core can generally be assumed) .

I don't think Paket and Nuget can help us here - how will Paket know of the binding configuration of Ionide, or VS2019, or FSC, or ... , which I believe it needs to know to do useful dependency management of compile-time addins.

Perhaps in the longer term we can run TPDTCs isolated, e.g. in a separate process (cf. FsAutoComplete.exe). However it is not totally easy to remote the System.Type/Method/... API through a process boundary.

Perhaps there is more we can learn from how Roslyn analyzers are loaded.

all that needs to be duplicated in the F# compiler (or otherwise we will miss a lot in our compatibility list)

The good news is that in practice the design-time component of most TPs can be relatively simple .NET Standard 2.0 components (e.g. this works for FSharp.Data). There are cases where the component has significant non-.NET Standard dependencies (e.g. RProvider) - over time we really just have to hope that these dependencies become available as .NET Standard components.

[dsyme]

which must locate type provider design-time components without resorting to using MSBuild.

And it will be solved via #2483 ;)

I don't think this solves type provider references.

Say that separate TPDTCs are available for .NET Framework 4.5 and .NET Standard 2.0. Which type TPDTC reference is emitted into the load script when running in FSI.EXE (.NET Framework 4.6 32 bit)? What about when type checking in devenv.exe (.NET Framework 32 bit)? When running in the F# Compiler (.NET Core 2.0 64 bit)? The package manager library (Paket.Core.dll) would have to look at the configuration of the tool it is running in (i.e,. FSI.EXE on .NET Core 2.0 x64) and do resolution based on that, but again presumably looking under some equivalent of the typeprovider directories mentioned above (these TPDTC components can't live in lib directories since they are not intended to be referenced in the normal way).

But that's just adding some fairly bespoke resolution rules into Paket. And we'd need another implementation in MSBuild/nuget.

I understand there's some perfect world where the compiler relies on external tooling to tell it about TPDTCs. But relying on external tools causes a lot of procedural problems too, and I'm not sure those tools know that much that's useful, apart from the framework compatibility relationship - which is trivial if all we care about is netstandard2.0, netcoreapp2.0 and net4x.

Note that we made a decision not to rely on external tooling for the resolution of TPDTCs in 2012 - and given the vast churn in nuget/msbuild since then that feels like a wise decision - it feels to me that external package manager tooling actually churns faster than anything else in the dotnettyverse :) - and any bespoke feature we might have put in back then would have been waaaay out of date by now. Better to have control over our destiny and put the rules in the F# compiler.. :)

One reason I'm in favour of the proposal above is that it represents a small variation on where we are today: just 20-50 lines of code in ExtensionTyping.fs, and I think it will get us to a stable place for the next 2-3 years. If we are going to change things in a major way I feel we should be more radical, e.g. look at interpreting type providers, or isolating them, or ... Basically getting rid of type providers as binary components. But that doesn't solve dependencies and so on, unless we use a full IL interpreter. (Also, we could look at what Scala tooling does for macros - the problems are very similar and I know there has been a lot of engineering challenges there).

[matthid]

Perhaps there is more we can learn from how Roslyn analyzers are loaded.

Yes please let's figure out their solution and go down the same/a similar route.
It seems they have to specify all deps by command line: dotnet/roslyn#14866 (comment)

Also it might be that we want to support language agnostic analyzers sooner or later one way or another, but in the mean-time we could even use the unused "fsharp" folder in there:
https://docs.microsoft.com/en-us/nuget/schema/analyzers-conventions

I'll try to respond to the rest of your response as well (as it contains valid critics) but I just wanted to get this part out.

[dsyme]

we could even use the unused "fsharp" folder in there:
https://docs.microsoft.com/en-us/nuget/schema/analyzers-conventions

TPDTCs are not analyzers, so putting them in the analyzers directory seems problematic. Adding "fsharp" to the path makes sense

I should also add this: when executing the F# compiler with .NET Core TPDTCs are loaded using AssemblyLoadContext.Default.LoadFromAssemblyPath - which I assume is the equivalent of Assembly.LoadFrom (untested).

[enricosada]

@dsyme, can we just use a nuget package to ship the design assemblies?
so TP libraries can depend on that package, and that package add the right design.time based on runtime
As a note, that package can also bundle the ProvidedTypes.fs file, so TP libraries just need to add that package and a stuff like

<Compile Include="$(ProvidedTypesSourcePath)" />

the ProvidedTypesSourcePath property is provided by the FSharp.TypeProviders.SDK package, and the source file is inside that.
So for TP lib, is a normal library. and can add deps too (by runtime or native too)

[dsyme]

@dsyme, can we just use a nuget package to ship the design assemblies?

@enricosada No - that would mean the TPDTCs get referenced as normal assemblies (-r) for the compilation, which is wrong. Also, tools like nuget and paket would select the DLLs based on the target runtime, not the compile-time runtime (the runtime and platform being used to execute the F# compiler or other host tool).