Improve Paket.Bootstrapper.exe and Paket.exe for use in F# Scripting

[cloudRoutine]

I know there's an example of how to use Paket inside a fsx, but I think the usage of Paket.Bootstrapper and Paket in F# scripts could be greatly improved by modifying the exposure of their API for consumption and the addition of a few more functions for ease of use.

After a minor tweak to expose the main function of the bootstrapper setting up a Fake build script intended to be run with fsi and appropriate for inclusion in a github repo was as simple as

#r "./.paket/paket.bootstrapper.exe"
Paket.Bootstrapper.Program.Main [|""|]
#r "./.paket/paket.exe"
let dependencies = Paket.Dependencies.Locate __SOURCE_DIRECTORY__
dependencies.Restore()
#r "packages/Fake/FakeLib.dll"
open Fake

Adding a parsing function that takes the same strings as main does on the commandline that writes the output to console would make this easy to use by anyone already familiar with the bootstrapper.

The surface area of Paket's public API seems a bit excessive

but maybe that's not too important

Like the bootstrapper Paket.exe should expose a parsing function taking commandline args as strings and that writes its output to console.

At the same time all of the commands should be exposed in a strongly typed manner to allow more complex programmatic construction and execution of paket's functions. Maybe it'd be useful to more concretely model the domain space for the types returned by these functions?

It could also be useful to implement help message functions for the exposed scripting apis that give back intellisense-like or explanatory information about the functions based on their names since there won't be any intellisense provided by an xml file shipped with the .exe

[cdrnet]

What's the advantage of scripting against Paket.exe instead of using the Paket.Core package which comes with a nice public api?

[cloudRoutine]

If you're cloning a repo with the kind of build script I was describing above all you're going to have to start is the bootstrapper. the bootstrapper gets you paket.exe, which I suppose you could use to get paket.core afterwards then reference that as well and do even more with it, but why download essentially the same thing again?

Paket.exe is always around and if you want to use it to do something programatically; it's easily at hand without adding an extra dependency.

[brianary]

This may not be the place, but it seems related…

I was thinking that a separate syntax to parse dependencies directly from an .fsx file would be nice.

//#paket source https://www.nuget.org/api/v2
#I __SOURCE_DIRECTORY__
#I "packages/p̼a̼c̼k̼a̼g̼e̼n̼a̼m̼e̼/lib/net40" // would imply "nuget packagename"
#r "pkgassembly"
#r "packages/p̼a̼c̼k̼a̼g̼e̼n̼a̼m̼e̼/lib/net40/pkgassembly.dll" // alternate form, though less reliable

This way, existing reference infrastructure would eliminate duplication of effort in listing the dependencies, and no overhead to build the separate dependencies file would be needed for single-.fsx-file solutions.

Currently, a fully independent .fsx file needs to perform a lot of ceremony before installing dependencies:

#I __SOURCE_DIRECTORY__
let ``paket.exe`` = IO.Path.Combine(__SOURCE_DIRECTORY__,"paket.exe")
if not (IO.File.Exists ``paket.exe``) then
    use wc = new Net.WebClient()
    wc.DownloadFile(wc.DownloadString("http://fsprojects.github.io/Paket/stable"),``paket.exe``)
let ``paket.dependencies`` = IO.Path.Combine(__SOURCE_DIRECTORY__,"paket.dependencies")
if not (IO.File.Exists ``paket.dependencies``) then
    IO.File.WriteAllLines(``paket.dependencies``,["source https://nuget.org/api/v2"])
#r "paket.exe"
open Paket
let dependencies = Dependencies.Locate(__SOURCE_DIRECTORY__)
dependencies.Add "FSharp.Data.SqlClient"
dependencies.Add "FSharp.Charting"
dependencies.Restore()

#I @"packages\FSharp.Data.SqlClient\lib\net40"
#r "FSharp.Data.SqlClient"
open FSharp.Data
#I @"packages\FSharp.Charting\lib\net40"
#r "FSharp.Charting"
open FSharp.Charting

This could be reduced to:

#I __SOURCE_DIRECTORY__
let ``paket.exe`` = IO.Path.Combine(__SOURCE_DIRECTORY__,"paket.exe")
if not (IO.File.Exists ``paket.exe``) then
    use wc = new Net.WebClient()
    wc.DownloadFile(wc.DownloadString("http://fsprojects.github.io/Paket/stable"),``paket.exe``)
#r "paket.exe"
open Paket
let dependencies = Dependencies.Locate(__SOURCE_FILE__)
dependencies.Restore()

//#paket source https://www.nuget.org/api/v2
#I @"packages\FSharp.Data.SqlClient\lib\net40"
#r "FSharp.Data.SqlClient"
open FSharp.Data
#I @"packages\FSharp.Charting\lib\net40"
#r "FSharp.Charting"
open FSharp.Charting

If the user first runs paket restore --script s̼c̼r̼i̼p̼t̼n̼a̼m̼e̼.fsx themselves, it becomes trivial:

//#paket source https://www.nuget.org/api/v2
#I @"packages\FSharp.Data.SqlClient\lib\net40"
#r "FSharp.Data.SqlClient"
open FSharp.Data
#I @"packages\FSharp.Charting\lib\net40"
#r "FSharp.Charting"
open FSharp.Charting

[matthid]

@brianary can you take a look at https://github.com/matthid/FAKE/blob/coreclr/help/fake-dotnetcore.md and tell me if this is what you are searching for? And if it isn't then why?

[brianary]

No, I'm not talking about Fake (or builds or .NET core or bash).

One of the main differences between a .fs file and a .fsx file is the .fsx file has no separate .sln file to track the library references it uses, so F# scripts added the #r (and #I) pragmas to keep everything in one file, because a .fsx script idiomatically doesn't expect external infrastructure.

An .fsx file should be adequate on its own, and distributable as a single file, rather than requiring a separate paket.dependencies file when the package dependency information already exists within the .fsx.

It's currently possible to download paket and reference/use the paket API within the script, but in an .fsx that adds even more overhead than the dependencies file (see above).

It would be convenient if paket could just parse the existing #I references that start with "packages/" in the .fsx file, rather than having to duplicate the list of package names in another file.

[matthid]

@brianary

No, I'm not talking about Fake (or builds or .NET core or bash).

You should free yourself from thinking that FAKE is for build scripts only. This is playing with ideas. Whether this functionality will be part of Paket or FAKE is not clear jet.

One of the main differences between a .fs file and a .fsx file is the .fsx file has no separate .sln file to track the library references it uses, so F# scripts added the #r (and #I) pragmas to keep everything in one file, because a .fsx script idiomatically doesn't expect external infrastructure.

That is not correct. fsx scripts expect a working fsi installed on the current system. The same way a working FAKE is expected for the linked proposal (but no other dependencies).
Actually what you are proposing still requires a working paket.exe if I understood that correctly.

It would be convenient if paket could just parse the existing #I references that start with "packages/" in the .fsx file, rather than having to duplicate the list of package names in another file.

Granted it looks differently and its called FAKE instead of Paket but what is the fundamental difference of your suggestion to embedding the paket.dependencies file to the top of the script as suggested?
Again in your proposal one still needs paket.exe and a working fsi. In my proposal one only needs the new FAKE (or whatever we like to call it, eg: fsiext).

There is also no reason why the script couldn't work with regular "fsi" for example by having a command to only restore the dependencies, which - I think - is quite close to what you are suggesting.

[brianary]

I guess I should clarify semantically that .fsx files may require executables, but they don't require local project configuration files. The infrastructure they expect is system-wide, general-purpose.

Is the expectation that the Fake build.fsx script would start with the paket dependencies header? The linked article only mentions "script", but I think you're talking about the Fake script? If that's the case, that also means synchronizing a build script to .fsx files, which doesn't differ much from synchronizing a dependencies file to it, but which could be quite a hassle if you're managing a utility directory full of .fsx scripts, each with separate package requirements.

If I've misread this, and the header is applicable to any script (which could be the case, although this may require formulating fsi.exe-targeted scripts to structure them for Fake), that could be viable. Certainly it makes sense preprocessing to extract dependencies, call paket to resolve them, and maybe exec straight into fsi.exe.

[matthid]

@brianary

Is the expectation that the Fake build.fsx script would start with the paket dependencies header? The linked article only mentions "script", but I think you're talking about the Fake script? If that's the case, that also means synchronizing a build script to .fsx files, which doesn't differ much from synchronizing a dependencies file to it, but which could be quite a hassle if you're managing a utility directory full of .fsx scripts, each with separate package requirements.

I'm not sure I understand this paragraph completely. I'm also not sure why a directory of scripts is a problem.
My proposal (the linked docs) has two modes of operation:

• embedded paket.dependencies into the fsx file
• external paket.dependencies

You can choose the mode based on your requirements. The first mode would make sense if your scripts in your directory should all be stand-alone. The second mode makes sense if you have a complete project or all your scripts share the same dependencies (for example a usual build script).

Whatever the mode is: FAKE first restores the dependencies and then launches the script (and use a local cache if applicable).

and the header is applicable to any script

Again I'm not quite sure what you mean. Are you talking about #loading scripts recursively or something like that? fsi will not understand the header right away you definitely need to start the script with FAKE.

[brianary]

An embedded dependencies file accomplishes most of what I was interested in, but it does still mean duplicating the list of pakages the script will use. Listing them once in the embedded depenedencies, and again with the references. As the script evolves and adds or drops package dependencies, both lists must be kept in sync, since there's no single point of truth. (Of course extracting simple package names from the references has limitations like not supporting specific version requirements.) Embedding the header does allow managing the dependencies for a bunch of scripts independently, though.

On top of this, using Fake does add some overhead. It's more moving parts, the way that it produces output, produces errors, and accepts arguments are all significantly different than a standard fsi.exe script (piping I/O would be pretty tricky to support, I think), and the script body would probably need to be structured as a Fake target (or set of targets) AFAICT.

[matthid]

There is no references file in the embedding case, we simply load all packages of a group.

Fake and Fake.vnext work without Targets as well (in fact you can think of it as if they are just calling fsi behind the scenes). If there are differences in piping and stuff you might want to consider opening a bug. I agree that there should be flag in vnext to make FAKE completely silent and only get script output. I'm more and more thinking that I tried to capture your exact use case with the proposal. My advertisement might be sub-optimal though.

[brianary]

By "references", I meant the `#r` & `#I` pragmas in the .fsx file.

…

On Sun, Jan 8, 2017, 12:09 Matthias Dittrich ***@***.***> wrote: There is no references file in the embedding case, we simply load all packages of a group. Fake and Fake.vnext work without Targets as well (in fact you can think of it as if they are just calling fsi behind the scenes). If there are differences in piping and stuff you might want to consider opening a bug. I agree that there should be flag in vnext to make FAKE completely silent and only get script output. I'm more and more thinking that I tried to capture your exact use case with the proposal. My advertisement might be sub-optimal though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1759 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH8mJ5x9CoNfUsvTzJIxmp7TLLXKhTSdks5rQUKGgaJpZM4I-eBt> .

[matthid]

You only need #load "./.fake/build.fsx/loadDependencies.fsx" and that file is generated by FAKE.

[brianary]

How does Fake extract which libraries to reference? Does it look at the `open` statements and examine the dependencies using reflection, or does it just reference all libraries in the packages listed?

…

On Sun, Jan 8, 2017, 12:23 Matthias Dittrich ***@***.***> wrote: You only need #load "./.fake/build.fsx/loadDependencies.fsx" and that file is generated by FAKE. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1759 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH8mJ9uLFi3c09mH4Dxc5tN1BDgi2bmoks5rQUXIgaJpZM4I-eBt> .

[matthid]

actually this is a feature of Paket itself: Paket can generate loading scripts for complete paket-groups. The proposal uses this feature to generate the loadDependencies.fsx file. See https://fsprojects.github.io/Paket/paket-generate-include-scripts.html

To put it simple: Yes it will reference everything.
(Which is not entirely correct as you can have multiple paket groups even within the header...)

[brianary]

Wow. I need to think about that, then.

…

On Sun, Jan 8, 2017, 12:34 Matthias Dittrich ***@***.***> wrote: actually this is a feature of Paket itself: Paket can generate loading scripts for complete paket-groups. The proposal uses this feature to generate the loadDependencies.fsx file. See https://fsprojects.github.io/Paket/paket-generate-include-scripts.html To put it simple: Yes it will reference everything. (Which is not entirely correct as you can have multiple paket groups even within the header...) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1759 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH8mJ8lMNwplUOw3VJ2UHaY-VRXgBYRAks5rQUhLgaJpZM4I-eBt> .