Ideas for improving string literal syntax

[oising]

Taken from discussion at https://twitter.com/alexandair/status/1378100061725466624

Taking @alexandair 's question in hand:

# bicep
output sshCommand string = 'ssh ${adminUsername}@${http://publicIP.properties.dnsSettings.fqdn}'

# psarm
Output 'sshCommand' -Type 'string' -Value (concat 'ssh ' $adminUsername '@' `
   (reference $publicIPAddressName).dnsSettings.fqdn)

Bicep and PSArm. Which one do you prefer?

In short: How can we improve PSArm to allow interpolation of ARM references to keep the clarity of bicep, while allowing the improved expressiveness of powershell?

My suggestion is to use a special powershell arm resolver provider, modeled something like this:

[CmdletProvider("Arm", ProviderCapabilities.None)]
public class PSArmProvider: ItemCmdletProvider, IContentCmdletProvider {
   // ...
}

Problem statement

Regular string interpolation won't work, because as @rjmholt demonstrates:

So if $adminUsername is ‘me’, then “ssh $adminUsername” becomes “ssh me”, which works because it ends up as a string literal
But if $adminUsername is [parameters(‘x’)], then “ssh $adminUsername” will become the string literal “ssh [parameters(‘x’)]”, but it needs to be the ARM expression [concat(‘ssh’, parameters(‘x’))].

So, we can't use direct string interpolation - but what if we use powershell's provider model for our benefit?

Suggested Solution

Let's take advantage of the fact that "ssh $adminUsername" is equivalent to "ssh ${variable:adminUsername}", which is itself equivalent to "ssh $(get-content variable:\adminUsername)"

If we prefix PSArm objects with our own recursive "resolver" (hello graphql!) like so:

$arm:"ssh $arm:adminUsername" then we can apply the same logic:

$arm:adminUsername gets resolved to $(get-content arm:adminUsername) which resolves to [parameters('x')]

$arm:"ssh [parameters('x')]" gets resolved to $(get-content arm:"ssh [parameters('x')]") which resolves to [concat("ssh", [parameters('x'])]

This approach allows regular string interpolation too, example:

$arm:"ssh $arm:adminUsername $hostname" will interpolate $hostname as a regular string taken from the "default" variable provider.

Of course, the devil is in the details but I think the idea seems sound enough, IMHO.

[rjmholt]

I think the big challenges we'd need to overcome here are:

• We'd need to depend on ToString() of an ARM expression having an exact output
• We then take our self-describing object and expand it into a string (no matter what we do, we're forced to expand an object into a string)
• We then have to take a mix of unstructured user input and structured syntax (our ARM expressions) and parse the expressions out of them, which is dangerous

We would also require users to use the $arm: prefix on all strings that used this, and doubly serialise and unserialise every ARM expression object we handled like this.

On the pros side:

• This involves no AST rewriting
• We already have a parser for ARM expressions

[rjmholt]

The fact of the individual parameters being provider qualified in that case doesn't help us much $arm:adminUsername may as well be $adminUsername

[rjmholt]

If that's the case, then we're forced to implement a parser from that path, which has all the information.

Hmmm, unless we could force that path string to be something else because $arm:adminUsername renders differently to $adminUsername.

We'd still be facing a similar problem, but somewhat simpler

[rjmholt]

Errr, no I'm not sure that will work with non-variable expressions, since those can't be provider qualified

[oising]

Ah, gotcha -- I was thinking that you'd split the path string each time against space, and wrap with [concat( ... )] and reduce/tokenize until you're left with a path that matches [ ... ] -- but I'm lacking the context here for want in knowing how PSArm is actually using string literals in the DSL, and also my own forgetfulness about ARM syntax itself (I went full Terraform about 2 years ago)

[rjmholt]

I think we could plausibly try something like that:

• Tokenise the string to split it up the [...] parts
• Parse those parts as fresh ARM expressions (and we already have code for that)
• Recombine it all under a concat call

My worry is that with arbitrary user input, we would plausibly have inputs that conflict with our syntax. I've actually already basically seen meta-templates that have string literals in them like [[parameters('x')] (the extra [ escapes the expression)

[rjmholt]

On the AST rewriting front, my thought is basically that we could attempt something like this:

[Cmdlet(VerbsCommon.Get, "ConcatStrings")]
public class GetConcatStringsCommand : PSCmdlet
{
    [Parameter]
    public ScriptBlock Script { get; set; }

    protected override void EndProcessing()
    {
        var visitor = new StringGatheringVisitor();
        Script.Ast.Visit(visitor);
        WriteObject(visitor.GetRewrittenStrings(), enumerateCollection: true);
    }

    private class StringGatheringVisitor : AstVisitor2
    {
        private List<string> _strings;

        public StringGatheringVisitor()
        {
            _strings = new List<string>();
        }

        public IEnumerable<string> GetRewrittenStrings()
        {
            return _strings;
        }

        public override AstVisitAction VisitExpandableStringExpression(ExpandableStringExpressionAst expandableStringExpressionAst)
        {
            var sb = new StringBuilder(expandableStringExpressionAst.Value.Length).Append("concat ");
            int currIndex = 0;
            foreach (ExpressionAst expression in expandableStringExpressionAst.NestedExpressions)
            {
                int nextIndex = expression.Extent.StartOffset - expandableStringExpressionAst.Extent.StartOffset;
                if (currIndex < nextIndex)
                {
                    sb.Append("'").Append(expandableStringExpressionAst.Value.Substring(currIndex, nextIndex - currIndex - 1).Replace("'", "''")).Append("' ");
                }
                sb.Append(expression.Extent.Text).Append(' ');
                currIndex = expression.Extent.EndOffset - expandableStringExpressionAst.Extent.StartOffset - 1;
            }

            if (currIndex < expandableStringExpressionAst.Value.Length - 1)
            {
                sb.Append("'").Append(expandableStringExpressionAst.Value.Substring(currIndex).Replace("'", "''")).Append("'");
            }

            _strings.Add(sb.ToString());
            return AstVisitAction.Continue;
        }
    }
}

> Get-ConcatStrings -Script {
    "ssh $adminUsername@$((reference $publicIPAddressName).dnsSettings.fqdn)"
}

concat 'ssh ' $adminUsername '@' $((reference $publicIPAddressName).dnsSettings.fqdn)

> Get-ConcatStrings -Script { "hello${there}friends${of}the${woods}" }

concat 'hello' ${there} 'friends' ${of} 'the' ${woods}

[rjmholt]

Actually we could minimise the exposure here by defining a string keyword:

String{"ssh $adminUsername@$(...)"}

Not ideal, but just a suggestion

[oising]

Interesting idea -- I do like explicitness

[oising]

On the dirty arm provider trick, this seems to do what is expected:

# represents list of psarm ref objects
$psarm = @{
    adminUsername = "[parameters('x')]"
}

function arm([string]$expr) {
    # is psarm reference?
    if ($psarm[$expr]) {
        return $psarm[$expr] # emit expression
    }

@"
[concat($(($expr.split()|% { if (([regex]"`^\[.+\]`$").IsMatch($_)) {
@"
$_
"@
} else {
@"
'$_'
"@
}}) -join ","))]
"@
    
}

$hostname = "hostname"
arm("ssh $(arm('adminUsername')) $hostname")

returns [concat('ssh',[parameters('x')],'hostname')]

[oising]

An hour later and the naivety of this script nearly outdoes the count of things it can't handle... lol - not trying to make it sound easy, just scripting out loud!

[rjmholt]

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(BreakUp(args[0]));
        }

        private static string BreakUp(string s)
        {
            if (s.Length < 3
                || s.StartsWith("[[")
                || !s.Contains('['))
            {
                return s;
            }

            return string.Join(Environment.NewLine, ExpandArmString(s));
        }

        private static List<string> ExpandArmString(string s)
        {
            var acc = new List<string>();

            int i = 0;
            for (; i < s.Length; i++)
            {
                int start = s.IndexOf('[', i);

                if (start < 0)
                {
                    acc.Add(s.Substring(i));
                    break;
                }

                if (start > i)
                {
                    acc.Add(s.Substring(i, start - i));
                }

                i = start;
                acc.Add(ReadArmExpression(ref i, s));
            }

            return acc;
        }

        private static string ReadArmExpression(ref int i, string s)
        {
            var bracketStack = new Stack<int>();
            int start = i;
            bracketStack.Push(i);
            i++;

            for (; i < s.Length && bracketStack.Count > 0; i++)
            {
                switch (s[i])
                {
                    case '[':
                        bracketStack.Push(i);
                        continue;

                    case ']':
                        bracketStack.Pop();
                        continue;
                }
            }

            string expression = s.Substring(start, i - start);

            if (bracketStack.Count > 0)
            {
                throw new ArgumentException($"Invalid ARM expression: '{expression}'");
            }

            return expression;
        }
    }

This is also naive to something like [parameters(']')] (no idea if that's legal or not), but can be fixed.

[oising]

Any luck with the approach Rob, or is it starting to smell like a dead end?

[rjmholt]

Haven't put any time toward this yet — have a bunch of other stuff that I need to get to in the backlog. I think we're going to solve some other things before the string syntax, since it's less pressing (more an ergonomics issue). But it is something that's come up a few times, so I'm keen to solve it.

I'm hoping we can get some more voices in this discussion and maybe some other ideas before investing too heavily in an implementation. The AST transformation is possible, but quite dangerous I think, since suddenly other functionalities like +, -join or just methods might be required to work.

[oising]

I'll see if I can drag some other pwsh old men like myself into the convo from Twitter. I think Karl Prosser and Joel Bennett might enjoy this.