Plan for updating the Type Provider Authoring SDK

[dsyme]

We need to update the Type Provider Authoring SDK (currently called FSharp.TypeProviders.StarterPack).

This is most urgently driven by two needs: using type providers when compiling PCL and .NET Core components. We also need our type providers to be "portable" where possible, so that they can at least be used regardless of whether the F# compiler is run using .NET Core or .NET Framework.

Here are the steps to happiness:

1. Support the authoring of type providers which can do PCL code generation when running in a .NET Framework hosted compiler. See PR Support for authoring type providers which target portable profiles  #78. Also put in place basic testing and better CI in this repo, see also PR Support for authoring type providers which target portable profiles  #78
2. Support the authoring of type providers which can do CoreCLR (or “platform standard”) code generation when running in a .NET Framework hosted compiler. This should just be a matter of finding the right reference DLLs.
3. Support the authoring of type providers which can do CoreCLR (or “platform standard”) code generation when running in a CoreCLR hosted compiler. This may be trickier as all the code in the SDK needs to be made CoreCLR-friendly
4. Support the authoring of type providers which are themselves PCL or CoreCLR or “platform standard” components, and can be used in binary form in either compilers, and can generate code for any target. This should drop out for free from step 3, but we need to check & test
5. Rename the SDK to "FSharp.TypeProviders.Authoring" or the like
6. Greatly improve the docs, for example addressing Include samples/tests from old f# sample pack #66. We would like the docs to be at least as good as the MSDN docs and walkthroughs.

As part of this I will improve and update the docs in that repo so that they are at least up to the standard of an F# community project like FSharp.Data. There are also good quality docs on MSDN and some other good resources, linked here. Updating the MSDN content would also be great, if only to link across.

[jaredpar]

Another possible step to consider:

• Add multiple platform targets to the Type Provider NuGet for both the Desktop and NetStandard targeting SDKs. NuGet would then pick the right one based on the target of the type provider being authored.

[KevinRansom]

@dsyme

I think the right place for this API is in the compiler. The re-mapping is going to be dependent on the set of referenced assemblies, I would prefer the compiler ensured the right thing happened rather than a library SDK.

I am also not sure about the purpose of : typeof in the example. There should be no circumstance where referencing a type depends on the version of mscorlib that is loaded in process.
I think you may be using the typeof in order to pass into the Api the name of the type, A string is probably the better way of doing that. In a world containing reference assemblies we shouldn't expect the type providers to load and execute the types they are specifying. They may want to reify and use real types but we shouldn't require it. After all the type from a reference assembly has absolutely no behavior. Also for the coreclr, the set of assemblies will be the minimum set necessary to run the compiler, and the version of any assembly loaded and executed will be the one in the coreclr directory.
The type provider even with load contexts may have no opportunity to specify alternate versions of assemblies already executing. One final thing when we do AOT compilation, we will probably want to restrict reflectable apis, to the absolute minimum necessary set for the compiler to operate.

I think this location is fine for VS2015, but moving on, I think would prefer if the compiler managed this and the API did not have types as input values.

Kevin

[dsyme]

I think the right place for this API is in the compiler. The re-mapping is going to be dependent on the set of referenced assemblies, I would prefer the compiler ensured the right thing happened rather than a library SDK.

I'm sympathetic to this and some of this work can be seen as a preliminary cut of that for some future compiler addition. It seems best that the compiler provide a type/member binding facility.

I am also not sure about the purpose of : typeof in the example. There should be no circumstance where referencing a type depends on the version of mscorlib that is loaded in process.

This problem runs deep with how type providers are have always been authored, and it's considerably worse than typeof: Quotation literals <@ ... @> are like typeof - they produce quotation values which reference System.Type and other System.Reflection objects which are associated with the running mscorlib. Quotations literals are used by pretty much all type providers to describe provided code. Let's call these QRunning quotations. There are QRunning quotations in the example in the README, and in every other type provider sample on the web.

Let's call quotation values created directly QTarget quotations (these are created using the quotation API Expr.Call etc. without using quotation literals, and if done carefully can be independent of the running mscorlib and instead reference the set of target assemblies) . The compiler expects to receive QTarget quotations from the type provider.

The primary contribution of this PR is to allow the creation and manipulation of QTarget quotation values and types in the SDK code (see AssemblyReader, AssemblyReaderReflection).

For now, the API for authoring remains largely unchanged: the author uses typeof and QRunning quotations and the AssemblyReplacer remaps QRunning to QTarget behind the scenes (based on searching the set of target referenced assemblies for matching type and method names).

I agree that the use of QRunning quotations to author quotations is wrong. However, it is done for three reasons

• Remapping gives a transition story for existing type providers like FSharp.Data which make very, very widespread use of typeof and QRunning quotations
• Remapping works well in the cases where you are targeting a PCL/.NETCore subset
• Most difficult is that fact that authoring type providers without using typeof and QRunning quotations and remapping is brutally hard (both to write the code and get it right). It's not impossible and we will need to document how to do it. But specifying types by string names without even using typeof<Foo>.FullName (the moral equivalent of manual remapping) is a nightmare. Specifying code this way is even worse.

With regard to AOT - if we really want to eliminate the compiler reliance on receiving even QTarget System.Type/System.MethodInfo/quotations values, then we would need to change the ITypeProvider API to avoid transacting these types altogether, and instead use a completely independent representation of types, symbols and code. There would be some advantages to this (it would reduce the compiler exposure to .NET primitives greatly and clarify exactly which parts of System.Type etc. are relevant - a much smaller subset than the entire System.Type API). I'll think this over, thanks.

[dmitry-a-morozov]

I have 2 items in wish list

1. Productization of https://github.com/fsprojects/TPCombinators.
2. Ability to emit generics from type providers.

[mavnn]

I'm planning on being at the post F# Exchange (London) hackday on the 16th April. I'll have some distractions, but I was planning to spend some of the time doing some maintenance of this repo and getting a framework for testing and documentation in place.

It would be an excellent time for people to either meet up, or if you can ping me pull requests/specific infrastructure requests in advance I'll try to fit in as much as I can. Unfortunately I don't really have the spare bandwidth at the moment to help out with the actual changes to the APIs required.

[mavnn]

As an aside I'm planning to kill the (somewhat unreliable) source NuGet package and recommend paket file references for all TP development. If anyone currently uses the NuGet and has it as a preferred option, please let me know how and why.

[KevinRansom]

I would appreciate understanding what the unreliability of nuget is caused by. Since they both use the same service for providing the packages. It would seem that service unreliability was the only real variable in the system, so it would be nice to know why the "nuget client" is unreliable, perhaps it can get fixed, regardless of the choice of dependency management for TypeProvider development within the fsharp. repos.

Thanks

Kevin

[mavnn]

@KevinRansom the problem is not in the package, it's in the fact that nuget packages have very limited support for content files and how to handle them. Meanwhile, "github referencing" a file in paket is a first class concept.

There have been several reported issues with the nuget package over time, such as what order it adds content files, having no control over which content files it adds (it's all or nothing) and similar issues.

updates for .net core?

[alfonsogarciacaro]

I just tried to compile an erasure Type Provider with .NET Core. Most of the Reflection methods can be polyfilled with packages like System.Reflection.Extensions or System.Reflection.TypeExtensions but the showstopper is the helper types inherit from classes like System.Type which don't have a public constructor in NETStandard. Would there be any hack to overcome this obstacle for the time being?

[dsyme]

@alfonsogarciacaro The Type() constructor appears to be protected, so should be usable from a subclass AFAIK https://github.com/dotnet/coreclr/blob/master/src/mscorlib/src/System/Type.cs#L56.

Does this match what you're seeing? Could you provide an error log?

[alfonsogarciacaro]

I get many errors so only the first one hundred are printed. I cannot see the error concerning System.Type but there're some for other types from System.Reflection like:

C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1248,5): error FS0509: Method or object constructor 'ParameterInfo' not found
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1266,5): error FS0509: Method or object constructor 'ParameterInfo' not found
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1354,17): error FS0855: No abstract or interface member was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1356,17): error FS0856: This override takes a different number of arguments to the corresponding abstract member
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1357,17): error FS0856: This override takes a different number of arguments to the corresponding abstract member
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1358,17): error FS0859: No abstract property was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1360,17): error FS0859: No abstract property was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1361,17): error FS0855: No abstract or interface member was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1362,17): error FS0855: No abstract or interface member was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1284,5): error FS0509: Method or object constructor 'ConstructorInfo' not found
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1440,17): error FS0855: No abstract or interface member was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1454,17): error FS0859: No abstract property was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1459,17): error FS0856: This override takes a different number of arguments to the corresponding abstract member
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1460,17): error FS0859: No abstract property was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1461,17): error FS0855: No abstract or interface member was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1462,17): error FS0855: No abstract or interface member was found that corresponds to this override
C:\tmp\FSharp.Data.GraphQL.Client\ProvidedTypes.fs(1365,5): error FS0509: Method or object constructor 'MethodInfo' not found

The constructor of ParameterInfo is also protected but still cannot inherit from it.

You can reproduce the issue by cloning the FSharp.Data.GraphQL repo and trying to build the following project.json:
https://github.com/bazingatechnologies/FSharp.Data.GraphQL/blob/dev/src/netcore/FSharp.Data.GraphQL.Client/project.json