Command line parser

[terrajobst]

The purpose of this library is to make command line tools first class by providing a command line parser.

• Designed for cross-platform usage
• Lightweight with minimal configuration
• Optional but built-in support for help, validation, and response files
• Support for multiple commands, like version control tools

Sample code:

using System;
using System.CommandLine;

static class Program
{
    static void Main(string[] args)
    {
        var addressee = "world";

        ArgumentSyntax.Parse(args, syntax =>
        {
            syntax.DefineOption("n|name", ref addressee, "The addressee to greet");
        });

        Console.WriteLine("Hello {0}!", addressee);
    }
}

Usage:

$ ./hello -h
usage: hello [-n <arg>]

    -n, --name    The addressee to greet

$ ./hello
Hello world!
$ ./hello -n Jennifer
Hello Jennifer!
$ ./hello --name Tom
Hello Tom!
$ ./hello -x
error: invalid option -x

More details in the README.md.

Edit: Feedback

• Remove support for slash options. It seems to hinder cross plat more than it helps. (6bea801)
• Allow overriding options, i.e. single options should be allowed to occur more than once with the last one winning. Useful for scripting (dade269).

Considered by decided against:

• Remove the notion of shared options and instead make options defined before commands applicable if, and only if, no command was specified.
• Rename 'command' to 'verb'

[terrajobst]

@ahsonkhan @KrzysztofCwalina @vancem

[terrajobst]

(to be clear that's not a full blown review, just a CR with a quick round of input)

[FransBouma]

Why create a new one when there's an OSS library out there that does everything already and is battle tested? https://github.com/gsscoder/commandline

[jcdickinson]

Brilliant start!

I'm not too hot on this: "n|name". .Net has virtually no suck typing, please don't introduce any. Isn't it easy to change it to this instead:

syntax.DefineOption("n", "name", ref addressee, "The addressee to greet");

That would result in compile-time validation instead of runtime validation if I made a typo, e.g. n\name.

The braces are superfluous, so I'd change it to this:

class ArgumentSyntax
{
    public static void Parse(params Action<ArgumentSyntax>[] p) { }
}

var password = string.Empty;
var strong = false;
var files = new List<string>();
ArgumentSyntax.Parse(
    s => s.DefineOption("p", "password", ref password, "The password to use").Required(),
    s => s.DefineOption("strong", ref strong, "Whether to use strong encryption").Default(false),
    s => s.DefineOption(files, "The files to encrypt").AtLeast(1)
);
if (strong) CaesarStrong(password, files);
else Caesar(password, files);

Also notice that I used a fluent interface there as this could easily end up being overload hell.

I'm wondering if there isn't a more '.Net way' to do this, some inspiration (doesn't compile):

class ArgumentSyntax
{
    public static void Parse(params Expression<Action<Argument>>[] p) { }
}

string pass;
ArgumentSyntax.Parse(
    password => pass = (string)password,
    p => pass = (string)p.AsShortAlternative("password")
);

Edit: caught up a bit on Twitter. Regarding 'commands':

        var command = Commands.None;
        var prune = false;
        var message = string.Empty;
        var amend = false;
        var common = string.Empty;

        ArgumentSyntax.Parse(
            s => s.DefineOption("c", "common", ref common, "Some common/shared option."),

            s => s.DefineCommand("pull", () => command = Commands.Pull, "Pulls down commits"),
            s => s.DefineOption("p", "prune", ref prune, "The password to use").Default(false),

            s => s.DefineCommand("commit", () => command = Commands.Commit, "Commits changes"),
            s => s.DefineOption("m", "message", ref message, "The commit message").Required(),
            s => s.DefineOPtion("amend", ref amend, "Ammend the last commit").Default(false)
        );

Edit: note that with the params approach your first example will still work and could even be used to tidy up commands. E.g.

ArgumentSyntax.Parse(
    s => s.DefineOption("c", "common", ref common, "Some common/shared option."),

    s =>
    {
        s.DefineCommand("pull", () => command = Commands.Pull, "Pulls down commits");
        s.DefineOption("p", "prune", ref prune, "The password to use").Default(false);
    },

    s =>
    {
        s.DefineCommand("commit", () => command = Commands.Commit, "Commits changes");
        s.DefineOption("m", "message", ref message, "The commit message").Required();
        s.DefineOPtion("amend", ref amend, "Ammend the last commit").Default(false);
    }
);

Edit: Another concern, there needs to be an imperative approach. E.g. If I had a plugin system for commands, how would I add them and their args? It might be better to use an instance so that I could pass this to plugins/whatever:

class ArgumentSyntax
{
    public static void Parse(params Action<ArgumentSyntax> p)
    {
        new ArgumentSyntax(p).Parse();
    }

    public ArgumentSyntax(params Action<ArgumentSyntax> p)
    {
        Define(p);
    }

    public void Define(params Action<ArgumentSyntax> p)
    {

    }
}

var syntax = new ArgumentSyntax();
syntax.Define(
    s => s.Option("c", "common", ref common, "Some common/shared option."),
    s =>
    {
        s.Command("pull", ref command, Command.Pull, "Pulls down commits");
        s.Option("p", "prune", ref prune, "Prune branches").Default(false);
    },
    s =>
    {
        s.Command("commit", ref command, Command.Commit, "Commits changes");
        s.Option("m", "message", ref message, "The commit message").Required();
        s.Option("amend", ref amend, "Amend the last commit").Default(false);
    }
);
somePlugin.Define(syntax);
syntax.Parse();

Edit: Just noticed the readme. I much prefer your enum approach, edited the above example.

[dazinator]

I'd like to echo what @FransBouma said - why not iterate on an existing open source command line parser (if need be) like https://github.com/gsscoder/commandline

[terrajobst]

@FransBouma and @dazinator

why not iterate on an existing open source command line parser

The reasons are outlined here. I've discussed it with Frans on Twitter as well.

In general, we prefer to avoid NIH and contribute to existing libraries. There are plenty of examples, like JSON.NET and LibGit2Sharp. However, you can't expect to go to someone's library and fundamentally change how it feels. In the end, that's essentially creating a new library and hence is defeating the purpose of contributing to an existing library.

In particular, gsscoder/commandline is an excellent library and well suited if you want to use the attribute-based parsing. However, if you want to remove the dependency on reflection we'd essentially have to rewrite it. For gsscoder/commandline, the usage of reflection isn't an implementation detail -- it's the key part of the programming model.

[terrajobst]

@jcdickinson

Thanks a bunch for the detailed feedback! I'll take a closer look later.

[dazinator]

@terrajobst thank you for the explanation - that certainly makes sense.

[SimonCropp]

@terrajobst forgive my ignorance. why is the use f reflection in gsscoder/commandline problematic in this case?

[JakeGinnivan]

Just an idea, commands could use nesting rather than ordering to define their options which would be nicer for @jcdickinson's extensibility story.

var command = Commands.None;
var prune = false;
var message = string.Empty;
var amend = false;
var common = string.Empty;

ArgumentSyntax.Parse(
    s => s.DefineOption("c", "common", ref common, "Some common/shared option."),
    s => s.DefineCommand("pull", () => command = Commands.Pull, "Pulls down commits",
        c => c.DefineOption("p", "prune", ref prune, "The password to use").Default(false)),
    s => s.DefineCommand("commit", () => command = Commands.Commit, "Commits changes",
        c => c.DefineOption("m", "message", ref message, "The commit message").Required(),
        c => c.DefineOption("amend", ref amend, "Ammend the last commit").Default(false),
        c => somePlugin.DefineAdditionalCommitOptions(c)),
    s => somePlugin.DefineOptions(s)
);

[jcdickinson]

@JakeGinnivan got me thinking, what about lifting the XLinq pattern? Not sure if this is any better, just adding ideas into the mixing bowl.

class ArgumentSyntax
{
    public static void Parse(params object[] args)
    {
        // Each args can be:
        // - ArgumentOption
        // - ArgumentCommand
        // - An arbitrarily nested IEnumerable containing the above
    }
}

class ArgumentOption
{
    public ArgumentOption(string, string, string, Action<ArgumentValue> v)
    {
    }
}

class ArgumentCommand
{
    public ArgumentCommand(string, string, string, Action<ArgumentValue> v, params object[] args)
    {
        // Each args can be:
        // - ArgumentOption
        // - ArgumentCommand
        // - An arbitrarily nested IEnumerable containing the above
    }
}

class ArgumentValue
{
    public static explicit operator string(ArgumentValue val) { return val._val; }
    public static explicit operator int(ArgumentValue val) { return Int.Parse(val._val); }
    // etc.
}

ArgumentSyntax.Parse(
    new ArgumentOption("c", "common", "Some common/shared option.", v => common = (string)v),
    new ArgumentCommand("pull", "Pulls down commits", v => command = Command.Pull,
        new ArgumentOption("p", "prune", "Prune branches", v => prune = (bool)v).Default(false)),
    new ArgumentCommand("commit", "Commits changes", v => command = Command.Commit,
        new ArgumentOption("m", "message", "The commit message", v => message = (string)v).Required(),
        new ArgumentOption("amend", "Ammend the last commit", v => ammend = (bool)v).Default(false),
        somePlugin.DefineAdditionalCommitOptions()),
    somePlugin.DefineOptions()
);

[terrajobst]

@SimonCropp

forgive my ignorance. why is the use of reflection in gsscoder/commandline problematic in this case?

We want to be able to compile a console application with minimal dependencies. Imagine a world where we statically link the runtime and the used framework, like we do with .NET Native on UWP.

If your code doesn't require reflection, you should be able to produce binary that doesn't have to link in reflection. If something as basic as command line parsing coerces you into taking a dependency on reflection, then you've two options:

• Live with the bloat of your executable
• Roll your own parser

Our goal is to support folks building minimal apps. So we don't want to leverage reflection.

[SimonCropp]

@terrajobst thanks for the clarification.

[dazinator]

I'm for anything that can eliminate those refs from the API - reason is purely aesthetical! :)

[terrajobst]

@jcdickinson

I'm not too hot on this: "n|name" . .Net has virtually no suck typing, please don't introduce any.

I can see that. But both, single name and multiple names is common. So that would require always offering two methods, one with a single name or one with two names. Also, the API currently supports more than two names. In order to support that I'd have to either make it a plain array (yuck!) or make the names the last parameter so it can be marked as params.

The braces are superfluous, so I'd change it to this:

ArgumentSyntax.Parse(
    s => s.DefineOption("p", "password", ref password, "The password to use").Required(),
    s => s.DefineOption("strong", ref strong, "Whether to use strong encryption").Default(false),
    s => s.DefineOption(files, "The files to encrypt").AtLeast(1)
);

That seems over the top to me. There are many examples where the BCL takes a lamda. We don't provide overloads that take multiple lambdas just so that you can eliminate the braces.

Also notice that I used a fluent interface there as this could easily end up being overload hell.

That's an interesting approach. The way I've envisioned is slightly different.

The DefineOption<T> methods takes a Func<string, T> argument that does the parsing. I thought the easiest way for apps would be to provide their own parser. The simplest solution would look like this:

Guid v = Guid.Empty;
s.DefineOption("g", ref g, Guid.Parse, "Some guid");

Of course, it's easy for consumers to define extension methods. For cases where you want to provide validation on top, for example, that the string is valid file name, you could provide an extension method like this:

string fileName = null;
s.DefineOptionInputFile("file", ref fileName, "Some input file");

The benefit of your approach is that it's at the end and doesn't obstruct the mainflow. I like!

As far as validation of mutual exclusivity or occurrences go, it might be easier to allow the caller to specify a pattern like this:

string fileName;
string databaseHost;
string databaseName;

s.DefineOption("f|file", ref fileName, "The data file to use");
s.DefineOption("h|dbHost", ref databaseHost, "The database host to use");
s.DefineOption("n|dbName", ref databaseName, "The database name to use");
s.DefineSyntax("-f | -h [-n]");

This would mean that you can either specify a file or a database. If you specify a database, then you can also specify its name but you don't have to. Extension methods wouldn't quite work for that.

I'm wondering if there isn't a more '.Net way' to do this, some inspiration

I assume the goal would be to avoid magic strings? That's simple with C# 6:

string pass;
ArgumentSyntax.Parse(args, s => 
{
    s.DefineOption(nameof(pass), ref pass, "The password");
});

However, I'm quite certain that we don't want to add a dependency on System.Linq.ExpressionTrees :-)

Regarding 'commands'

It seem the biggest issue folks have with commands is that it shows that this proposal is order sensitive. The point I was trying to make on Twitter that in order to support ref arguments, the parsing has to be incremental (as the definers have to return the value immediately). In other words, there is nothing I can do in the implementation to hide that fact. Once you can't hide that, I think it's a lot easier for the consumers if that becomes an explicit design point of how the API is to be used.

So ignoring commands, this order is invalid:

bool o;
string p;
s.DefineParameter("p", ref p, "parameter p")
s.DefineOption("o", ref o, "option o")

The reason it's invalid is because for this input:

$ tool -o x

it's unclear if DefineParameter should consume x. Depending on how the option -o is defined, it may take a value, in which case x belongs to the option. If -o is defined to be a flag, i.e. of type Boolean, then x is indeed the first parameter. Hence, you must define all options before any parameters.

However, the developer doesn't have to worry too much about getting the order right. The API validates that things are called in the correct order and throws InvalidOperationException in case methods are called out of order. In the past, my team has done extensive case studies and good, actionable exceptions are sometimes much better than a type system that tries to prevent illegal states from occurring in the first place. The counter argument is usually that runtime errors might go undetected for a long time, i.e. until the code actually runs. However, I don't think that argument holds for a command line parser. First of all, that's literally the first code that runs in an application and secondly there are no states -- all calls to Define will happen all the time.

Another concern, there needs to be an imperative approach. E.g. If I had a plugin system for commands, how would I add them and their args? It might be better to use an instance so that I could pass this to plugins/whatever.

There are two options:

1. The plugin system could provide enough metadata so that Main can setup the syntax
2. The plugin gets an instance to ArgumentSyntax and define themselves

The first is supported by the fact that you don't have to use a ref based approach. Each Define* method returns a strongly typed object that provides access to the definition and the parsed value. There is also a base class that provides access to the value as object so that consuming code doesn't have to be generic. So you could use an approach similar to gsscoder/CommandLine and allow attributes on the plugin objects. In that case, the discovery phase in main could use the metadata to discover which APIs to call.

The second approach is actually the easiest and that's probably what I would do. It would look like this:

static class Program
{
    static void Main(string[] args)
    {
        // Getting commands. If you want, this could be call to an IoC
        // container.

        var commands =
        {
            new ImportCommand();
            new ExportCommand();
        };

        // Ask each command to define itself:

        var result = ArgumentSyntax.Parse(args, syntax =>
        {
            foreach (var c in commands)
                c.Define(syntax);
        });

        // Get the command and execute it.

        var command = (Command)result.ActiveCommand.Value;
        command.Execute();
    }

    abstract class Command
    {
        public abstract void Define(ArgumentSyntax syntax);
        public abstract void Execute();
    }

    class ImportCommand : Command
    {
        string _keepExisting;
        string _fileName;

        public override void Define(ArgumentSyntax syntax)
        {
            syntax.DefineCommand("import", this, "Import rows");
            syntax.DefineOption("k|keep-existing", ref _keepExisting, "Keep existing rows");
            syntax.DefineParameter("path", ref _fileName, "File to import");
        }

        public void Execute()
        {
            // ...
        }
    }

    class ExportCommand : Command
    {
        string _fileName;

        public override void Define(ArgumentSyntax syntax)
        {
            syntax.DefineCommand("export", this, "Export rows");
            syntax.DefineParameter("path", ref _fileName, "File to export to");
        }

        public void Execute()
        {
            // ...
        }
   }
}

Thoughts?

[terrajobst]

@dazinator

I'm for anything that can eliminate those ref s from the API - reason is purely aesthetical! :)

What about:

Argument<bool> useForce = null;
Argument<string> fileName = null;

ArgumentSyntax.Parse(args, s =>
{
    useForce = s.DefineOption("f|force", false, "Use force");
    fileName = s.DefineParameter("file", string.Empty, "The file name to delete");

    if (!fileName.IsSpecifed)
        s.Report("need a file name");
});


if (useForce.Value)
    Console.WriteLine("Deleting {0} with force", fileName.Value);
else
    Console.WriteLine("Deleting {0}", fileName.Value);

[KrzysztofCwalina]

At some point we should do an API review of this component and also add solid API docs, but for now it looks good to me.

[whoisj]

I'm having issues loading the changes (size versus browser stability). Does this change support parameter trees? It's very natural to only have options be exposed if others preceded them.

[terrajobst]

@whoisj

Currently that relies on manual validation. Do you have an approach in mind?

[jcdickinson]

@terrajobst thanks for the really detailed response.

the API currently supports more than two names.

I did an 80/20 split here. 80% will only ever need a maximum of two names, but, the remaining 20% could use one of the fluent methods to add more alternatives.

Otherwise, I'm now in complete agreement - looking forward to this hitting the BCL!

[whoisj]

@terrajobst I started (a few months ago) and never finished a param parser based on consuming arg nomenclature.

For example, you could represent git pull [--all (--ff-only|--no-ff|--rebase/-r) --force/-f (--prune/-p) (--tags/-t|--no-tags) --verbose/-v] [<remote> [<refspec>]] as

var options1 = new OrderedOptionSet
{
    new RequiredOptionSet
    {
        new VerbOption("pull")
    },
    new OptionalOptionSet
    {
        new SwitchOption("all"),
        new ExclusiveOptionSet
        {
            new SwitchOption("ff-only"),
            new SwitchOption("no-ff"),
            new SwitchOption("rebase", 'r')
        },
        new SwitchOption("force", 'f'),
        new SwitchOption("prune", 'p'),
        new ExclusiveOptionSet
        {
            new SwitchOption("tags", 't'),
            new SwitchOption("no-tags")
        },
        new SwitchOption("verbose", 'v')
    },
    new OptionalOptionSet
    {
        new LiteralOption("remote"),
        new OptionalOptionSet
        {
            new LiteralOption("refspec")
        }
    },
};

The conversion was bi-directional, and it supported the idea of trees (as you can see above).

The benefit of supporting trees, or exclusive groups of options is that a single parser setup can be used to define the entire input expectations of an application.

A nice side benefit of the way I designed it was that docs <==> code. :-)

All that said, this PR looks very nice - thank you!

[dazinator]

@whoisj - that looks pretty cool - I like that structure!
I had a go at expressing that as a fluent / builder API and it came out like this:

            var syntax = new SyntaxBuilder();

            syntax.HasVerb("pull")
                    .HasOptional((pullOptions) =>
                                  pullOptions.HasSwitch("all")
                                             .HasExclusive((mergeOptions) =>
                                                            mergeOptions.HasSwitch("ff-only")
                                                                        .HasSwitch("no-ff")
                                                                        .HasSwitch("rebase", 'r'))
                                             .HasSwitch("force", 'f')
                                             .HasSwitch("prune", 'p')
                                             .HasExclusive((tagsOptions) =>
                                                            tagsOptions.HasSwitch("tags", 't')
                                                                       .HasSwitch("no-tags"))
                                             .HasSwitch("verbose", 'v'))
                    .HasOptional((remoteOptions) =>
                                  remoteOptions.HasLiteral("remote")
                                               .HasOptional((refSpecOptions) =>
                                                             refSpecOptions.HasLiteral("refspec")));

My fluent API attempt seems a little more complicated thanks to the lambdas than your original and doesn't seem any more concise.. so probably not worth it all things considered haha :)

[pkanavos]

@FransBouma and @terrajobst as a victim of the gsscoder battle, I propose you add a test case that fails in gsscoder - assign the same value to two different options, eg --min:20 --batch:20.