From 165f8050cbda1a7c9c23b262f6b8b1e72212059b Mon Sep 17 00:00:00 2001 From: TSlivede Date: Fri, 10 Feb 2017 21:47:33 +0100 Subject: [PATCH 01/10] Create RFC####-Improve-generation-of-argument-string-for-executables.md --- ...tion-of-argument-string-for-executables.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md new file mode 100644 index 00000000..386fef27 --- /dev/null +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -0,0 +1,33 @@ +--- +RFC: RFC +Author: Timo Schwarte +Status: Draft +Version: 1.0 +Area: Calling external executables +Comments Due: 2017-06-31 +--- + +# Improve generation of argument-string for executables + +When Powershell calls an external executable, the +[NativeCommandParameterBinder](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/NativeCommandParameterBinder.cs) +takes all parameters of the command and generates the argument-string (also called "CommandLine") that is sent to the executable (as [ProcessStartInfo.Arguments](https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.arguments.aspx)). +It seems as if the NativeCommandParameterBinder trys to generate an argument-string, such that `argv[]` contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes arround the argument, so it reaches `argv[]` as one argument (not split into parts). +However if the argument itself contains quotes, those are not escaped. Therefor the corrosponding element in `argv[]` has no quotes and (depending on the actual argument) the argument can be splitted into multiple `argv[]` elements and it can even occur that the following arguments are not handled correctly. + +This RFC suggests to make the NativeCommandParameterBinder compatible to +[the typical CommandLine escaping rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). +Additionally it suggests to add a verbose-symbol, to add a custom formated string to the CommandLine (for executables, that don't follow the typical escaping rules.) + +This is basically a bugfix for https://github.com/PowerShell/PowerShell/issues/1995 and https://github.com/PowerShell/PowerShell/issues/3049 and on [stackoverflow](http://stackoverflow.com/a/21334121/2770331) (see `and I think this is a bug` within that answer). +However, as this is a very longstanding bug, there are workarounds for some cases. These will no longer work if this is corrected, therefor this document suggests to add a preference variable to get the old behaviour back if somebody needs it. + +## Motivation + + As a powershell user who wants to call external executables, + I can pass any arguments to those executable, + so that these arguments reach the `argv[]` array unchanged. + +## Specification + +## Alternate Proposals and Considerations From 62cc293df5f5d0e83808aa93812f5a03a4ed0fa2 Mon Sep 17 00:00:00 2001 From: TSlivede Date: Wed, 8 Mar 2017 21:00:16 +0100 Subject: [PATCH 02/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md Add quoting specification and todos --- ...tion-of-argument-string-for-executables.md | 44 +++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index 386fef27..8f29cec0 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -4,7 +4,7 @@ Author: Timo Schwarte Status: Draft Version: 1.0 Area: Calling external executables -Comments Due: 2017-06-31 +Comments Due: 2017-XX-XX --- # Improve generation of argument-string for executables @@ -12,14 +12,15 @@ Comments Due: 2017-06-31 When Powershell calls an external executable, the [NativeCommandParameterBinder](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/NativeCommandParameterBinder.cs) takes all parameters of the command and generates the argument-string (also called "CommandLine") that is sent to the executable (as [ProcessStartInfo.Arguments](https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.arguments.aspx)). -It seems as if the NativeCommandParameterBinder trys to generate an argument-string, such that `argv[]` contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes arround the argument, so it reaches `argv[]` as one argument (not split into parts). +It seems as if the NativeCommandParameterBinder trys to generate an argument-string, such that `argv[]` of the called executable contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes arround the argument, so it reaches `argv[]` as one argument (not split into parts). However if the argument itself contains quotes, those are not escaped. Therefor the corrosponding element in `argv[]` has no quotes and (depending on the actual argument) the argument can be splitted into multiple `argv[]` elements and it can even occur that the following arguments are not handled correctly. This RFC suggests to make the NativeCommandParameterBinder compatible to [the typical CommandLine escaping rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). Additionally it suggests to add a verbose-symbol, to add a custom formated string to the CommandLine (for executables, that don't follow the typical escaping rules.) -This is basically a bugfix for https://github.com/PowerShell/PowerShell/issues/1995 and https://github.com/PowerShell/PowerShell/issues/3049 and on [stackoverflow](http://stackoverflow.com/a/21334121/2770331) (see `and I think this is a bug` within that answer). +This is basically a bugfix for https://github.com/PowerShell/PowerShell/issues/1995 and https://github.com/PowerShell/PowerShell/issues/3049 and on stackoverflow (["This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command."](http://stackoverflow.com/questions/6714165/powershell-stripping-double-quotes-from-command-line-arguments) +and ["... and I think this is a bug Powershell doesn't escape any double quotes that appear inside the arguments."](http://stackoverflow.com/a/21334121/2770331)). However, as this is a very longstanding bug, there are workarounds for some cases. These will no longer work if this is corrected, therefor this document suggests to add a preference variable to get the old behaviour back if somebody needs it. ## Motivation @@ -30,4 +31,41 @@ However, as this is a very longstanding bug, there are workarounds for some case ## Specification +### Quoting + +The decision if an argument needs to be quoted, will be simplified: Any Argument, that contains `"`, `'` or a character that matches `char.IsWhiteSpace` will be quoted. To quote an argument (compatible to [MSVC rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) and `CommandLineToArgvW`): +- Every occurrence of N times `\` followed by `"` will be replaced by (2*N+1) times `\` followed by `"`. (N ∈ {0,1,2,...}) +- N times `\` at the end of the string is replaced by (2*N) times `\`. (N ∈ {0,1,2,...}) +- `"` is added to the beginning and to the end of the string + +### Verbatim Argument + +TODO: +`--=` next argument is copied to cmdline as is, symbol detected in parser + +### Preference Variable + ## Alternate Proposals and Considerations + +### Other shells + +TODO: + +Therefore Powershell should build the Commandline in a way, that matches those rules instead of just sometimes adding quotes around arguments. +Almost every other modern Commandshell on windows does this: +- https://www.hamiltonlabs.com/Cshell.htm (not documented) +- [Cygwin](https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/winf.cc;hb=HEAD) (see `linebuf::fromargv`) +- [Python](https://svn.python.org/projects/python/trunk/Lib/subprocess.py) (see `list2cmdline`) +- [Tcl](https://github.com/tcltk/tcl/blob/master/win/tclWinPipe.c) (see `BuildCommandLine`, Comment: "N backslashes followed a quote -> insert N * 2 + 1 backslashes then a quote.") +### Accepted + +TODO: +msvc, mingw, .net, winapi, .netcore on linux + +### Batch files + +### Linux +TODO: +Not Part of this RFC: +- might depricate `--%` and `--=` on linux +- might add `--$` to only evaluate `$...` and string literals, but treat everything else as verbose text arguments From 169226a8206c9546deb5e8d8e167161544f01abf Mon Sep 17 00:00:00 2001 From: TSlivede Date: Mon, 17 Apr 2017 17:07:00 +0200 Subject: [PATCH 03/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md verbatim argument preference variable prosa (other shells) prosa (rules are accepted) --- ...tion-of-argument-string-for-executables.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index 8f29cec0..27ae6d71 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -40,31 +40,32 @@ The decision if an argument needs to be quoted, will be simplified: Any Argument ### Verbatim Argument -TODO: -`--=` next argument is copied to cmdline as is, symbol detected in parser +When the parser detects the 'Verbatim-Argument-Symbol' (`--=`), the next argument is copied to the argument-string exactly as it is, without escaping or adding quotes around the argument. The symbol is detected in the parser, so the string literal `'--='` or any other object that expands to that string is not interpreted as 'Verbatim-Argument-Symbol'. ### Preference Variable +If the Preference Variable `$PsUseLegacyInconsistentArgumentStringGeneration` is set to true, the quoting is done as it is now. (Only for compatibility with old Scripts). The Variable is false by default. + ## Alternate Proposals and Considerations ### Other shells -TODO: - -Therefore Powershell should build the Commandline in a way, that matches those rules instead of just sometimes adding quotes around arguments. -Almost every other modern Commandshell on windows does this: -- https://www.hamiltonlabs.com/Cshell.htm (not documented) +Maybe this is not the strongest argument ever, but many other modern Commandshells on windows create the argument-string compatible to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx): +- https://www.hamiltonlabs.com/Cshell.htm (not documented, but escapes correctly) - [Cygwin](https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/winf.cc;hb=HEAD) (see `linebuf::fromargv`) - [Python](https://svn.python.org/projects/python/trunk/Lib/subprocess.py) (see `list2cmdline`) - [Tcl](https://github.com/tcltk/tcl/blob/master/win/tclWinPipe.c) (see `BuildCommandLine`, Comment: "N backslashes followed a quote -> insert N * 2 + 1 backslashes then a quote.") -### Accepted -TODO: -msvc, mingw, .net, winapi, .netcore on linux +### [These rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are widely accepted + +Most Compilers on Windows generate executables, that split the CommandLine compatible to [those rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx), for example the compiler shiped with VisualStudio as well as the `mingw` compiler suite. +The .Net runtime also splits the CommandLine in a way that is compatible to those rules. The Windows API function [`CommandLineToArgvW`](https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391.aspx) is (although incorrectly documented) also compatible to those rules. +Therefore Powershell should build the Commandline in a way, that matches those rules instead of just sometimes adding quotes around arguments. ### Batch files ### Linux +.netcore on linux TODO: Not Part of this RFC: - might depricate `--%` and `--=` on linux From f2004e3d414c42e7b56a287232650d985028e2d0 Mon Sep 17 00:00:00 2001 From: TSlivede Date: Sun, 30 Apr 2017 19:50:50 +0200 Subject: [PATCH 04/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md batch considerations prosa: Linux, out of this RFC --- ...ation-of-argument-string-for-executables.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index 27ae6d71..c9977ecf 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -62,11 +62,17 @@ Most Compilers on Windows generate executables, that split the CommandLine compa The .Net runtime also splits the CommandLine in a way that is compatible to those rules. The Windows API function [`CommandLineToArgvW`](https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391.aspx) is (although incorrectly documented) also compatible to those rules. Therefore Powershell should build the Commandline in a way, that matches those rules instead of just sometimes adding quotes around arguments. +### Linux + +While this change is important on windows, it's absolutely necessary on Linux: On Windows the CommandLine is splitted by the next executable and most executables follow the described rules. On Linux the CommandLine is not splitted by the called executable -- the .Net Core runtime splits the string. Therefor on Linux the described rules do not only apply to many calls of external executables, they apply to ALL calls of external executables. When the proposed changes are implemented, the arguments from within powershell always arive -- as expected -- as the `argv[]` array in called executables. + ### Batch files -### Linux -.netcore on linux -TODO: -Not Part of this RFC: -- might depricate `--%` and `--=` on linux -- might add `--$` to only evaluate `$...` and string literals, but treat everything else as verbose text arguments +Sadly, one of the few exeptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of powershell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefor I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirct their arguments to other executable and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are applying. + +### Considerations outside of the scope of this RFC + +Maybe on Linux `--%` and `--=` can be depricated, as they dont make much sense on Linux -- the CommandLine always needs to be escaped acording to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "Linux"). + +As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly but instead a way to disable many special characters in a powershell command, in the future one could possibly add a `--$` token that disables all special characters exept quotes and `$` -- a clean way of what `--%` was originally meant for. + From 16f8649d022c9340d7c21bc9df8491d248a48447 Mon Sep 17 00:00:00 2001 From: TSlivede Date: Sun, 30 Apr 2017 20:58:41 +0200 Subject: [PATCH 05/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md small changes --- ...tion-of-argument-string-for-executables.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index c9977ecf..29c036a6 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -1,10 +1,11 @@ --- -RFC: RFC +RFC: Author: Timo Schwarte Status: Draft Version: 1.0 Area: Calling external executables Comments Due: 2017-XX-XX +Plan to implement: Yes --- # Improve generation of argument-string for executables @@ -12,12 +13,12 @@ Comments Due: 2017-XX-XX When Powershell calls an external executable, the [NativeCommandParameterBinder](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/NativeCommandParameterBinder.cs) takes all parameters of the command and generates the argument-string (also called "CommandLine") that is sent to the executable (as [ProcessStartInfo.Arguments](https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.arguments.aspx)). -It seems as if the NativeCommandParameterBinder trys to generate an argument-string, such that `argv[]` of the called executable contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes arround the argument, so it reaches `argv[]` as one argument (not split into parts). -However if the argument itself contains quotes, those are not escaped. Therefor the corrosponding element in `argv[]` has no quotes and (depending on the actual argument) the argument can be splitted into multiple `argv[]` elements and it can even occur that the following arguments are not handled correctly. +It seems as if the NativeCommandParameterBinder trys to generate an argument-string, such that `argv[]` of the called executable contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes arround the argument, so it reaches `argv[]` as one argument (not splited into parts). +However if the argument itself contains quotes, those are not escaped. Therefor the corrosponding element in `argv[]` has no quotes and (depending on the actual argument) the argument can be splitted into multiple `argv[]` elements. It can even occur that the following arguments are not handled correctly. This RFC suggests to make the NativeCommandParameterBinder compatible to [the typical CommandLine escaping rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). -Additionally it suggests to add a verbose-symbol, to add a custom formated string to the CommandLine (for executables, that don't follow the typical escaping rules.) +Additionally it suggests to add a verbose-symbol to add a custom formated string to the CommandLine (for executables, that don't follow the typical escaping rules.) This is basically a bugfix for https://github.com/PowerShell/PowerShell/issues/1995 and https://github.com/PowerShell/PowerShell/issues/3049 and on stackoverflow (["This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command."](http://stackoverflow.com/questions/6714165/powershell-stripping-double-quotes-from-command-line-arguments) and ["... and I think this is a bug Powershell doesn't escape any double quotes that appear inside the arguments."](http://stackoverflow.com/a/21334121/2770331)). @@ -26,7 +27,7 @@ However, as this is a very longstanding bug, there are workarounds for some case ## Motivation As a powershell user who wants to call external executables, - I can pass any arguments to those executable, + I can pass any arguments to those executables, so that these arguments reach the `argv[]` array unchanged. ## Specification @@ -52,9 +53,9 @@ If the Preference Variable `$PsUseLegacyInconsistentArgumentStringGeneration` is Maybe this is not the strongest argument ever, but many other modern Commandshells on windows create the argument-string compatible to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx): - https://www.hamiltonlabs.com/Cshell.htm (not documented, but escapes correctly) -- [Cygwin](https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/winf.cc;hb=HEAD) (see `linebuf::fromargv`) -- [Python](https://svn.python.org/projects/python/trunk/Lib/subprocess.py) (see `list2cmdline`) -- [Tcl](https://github.com/tcltk/tcl/blob/master/win/tclWinPipe.c) (see `BuildCommandLine`, Comment: "N backslashes followed a quote -> insert N * 2 + 1 backslashes then a quote.") +- [Cygwin](https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/winf.cc;hb=HEAD#l66) (see `linebuf::fromargv`) +- [Python](https://github.com/python/cpython/blob/master/Lib/subprocess.py#L424) (see `list2cmdline`) +- [Tcl](https://github.com/tcltk/tcl/blob/master/win/tclWinPipe.c#L1503) (see `BuildCommandLine`, Comment: "N backslashes followed a quote -> insert N * 2 + 1 backslashes then a quote.") ### [These rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are widely accepted @@ -74,5 +75,5 @@ Sadly, one of the few exeptions to those typical parsing rules is `cmd.exe`. Bec Maybe on Linux `--%` and `--=` can be depricated, as they dont make much sense on Linux -- the CommandLine always needs to be escaped acording to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "Linux"). -As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly but instead a way to disable many special characters in a powershell command, in the future one could possibly add a `--$` token that disables all special characters exept quotes and `$` -- a clean way of what `--%` was originally meant for. +As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly but instead a way to disable many special characters in a powershell command. In the future one could possibly add a `--$` token that disables all special characters exept quotes and `$` -- a clean way of what `--%` was originally meant for. From a016013a306ffb1e16741ce26d7dd41d5c65e020 Mon Sep 17 00:00:00 2001 From: TSlivede Date: Sun, 30 Apr 2017 21:30:12 +0200 Subject: [PATCH 06/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md word spellcheck --- ...tion-of-argument-string-for-executables.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index 29c036a6..d1c725a0 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -10,23 +10,23 @@ Plan to implement: Yes # Improve generation of argument-string for executables -When Powershell calls an external executable, the +When PowerShell calls an external executable, the [NativeCommandParameterBinder](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/NativeCommandParameterBinder.cs) takes all parameters of the command and generates the argument-string (also called "CommandLine") that is sent to the executable (as [ProcessStartInfo.Arguments](https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.arguments.aspx)). -It seems as if the NativeCommandParameterBinder trys to generate an argument-string, such that `argv[]` of the called executable contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes arround the argument, so it reaches `argv[]` as one argument (not splited into parts). -However if the argument itself contains quotes, those are not escaped. Therefor the corrosponding element in `argv[]` has no quotes and (depending on the actual argument) the argument can be splitted into multiple `argv[]` elements. It can even occur that the following arguments are not handled correctly. +It seems as if the NativeCommandParameterBinder tries to generate an argument-string, such that `argv[]` of the called executable contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes around the argument, so it reaches `argv[]` as one argument (not split into parts). +However, if the argument itself contains quotes, those are not escaped. Therefor the corresponding element in `argv[]` has no quotes and (depending on the actual argument) the argument can be split into multiple `argv[]` elements. It can even occur that the following arguments are not handled correctly. This RFC suggests to make the NativeCommandParameterBinder compatible to [the typical CommandLine escaping rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). -Additionally it suggests to add a verbose-symbol to add a custom formated string to the CommandLine (for executables, that don't follow the typical escaping rules.) +Additionally, it suggests to add a verbose-symbol to add a custom formatted string to the CommandLine (for executables, that don't follow the typical escaping rules.) This is basically a bugfix for https://github.com/PowerShell/PowerShell/issues/1995 and https://github.com/PowerShell/PowerShell/issues/3049 and on stackoverflow (["This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command."](http://stackoverflow.com/questions/6714165/powershell-stripping-double-quotes-from-command-line-arguments) and ["... and I think this is a bug Powershell doesn't escape any double quotes that appear inside the arguments."](http://stackoverflow.com/a/21334121/2770331)). -However, as this is a very longstanding bug, there are workarounds for some cases. These will no longer work if this is corrected, therefor this document suggests to add a preference variable to get the old behaviour back if somebody needs it. +However, as this is a very longstanding bug, there are workarounds for some cases. These will no longer work if this is corrected, therefor this document suggests to add a preference variable to get the old behavior back if somebody needs it. ## Motivation - As a powershell user who wants to call external executables, + As a PowerShell user who wants to call external executables, I can pass any arguments to those executables, so that these arguments reach the `argv[]` array unchanged. @@ -59,21 +59,21 @@ Maybe this is not the strongest argument ever, but many other modern Commandshel ### [These rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are widely accepted -Most Compilers on Windows generate executables, that split the CommandLine compatible to [those rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx), for example the compiler shiped with VisualStudio as well as the `mingw` compiler suite. +Most Compilers on Windows generate executables, that split the CommandLine compatible to [those rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx), for example the compiler shipped with VisualStudio as well as the `mingw` compiler suite. The .Net runtime also splits the CommandLine in a way that is compatible to those rules. The Windows API function [`CommandLineToArgvW`](https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391.aspx) is (although incorrectly documented) also compatible to those rules. -Therefore Powershell should build the Commandline in a way, that matches those rules instead of just sometimes adding quotes around arguments. +Therefore, PowerShell should build the CommandLine in a way, that matches those rules instead of just sometimes adding quotes around arguments. ### Linux -While this change is important on windows, it's absolutely necessary on Linux: On Windows the CommandLine is splitted by the next executable and most executables follow the described rules. On Linux the CommandLine is not splitted by the called executable -- the .Net Core runtime splits the string. Therefor on Linux the described rules do not only apply to many calls of external executables, they apply to ALL calls of external executables. When the proposed changes are implemented, the arguments from within powershell always arive -- as expected -- as the `argv[]` array in called executables. +While this change is important on windows, it's absolutely necessary on Linux: On Windows the CommandLine is split by the next executable and most executables follow the described rules. On Linux the CommandLine is not split by the called executable -- the .Net Core runtime splits the string. Therefor on Linux the described rules do not only apply to many calls of external executables, they apply to ALL calls of external executables. When the proposed changes are implemented, the arguments from within PowerShell always arrive -- as expected -- as the `argv[]` array in called executables. ### Batch files -Sadly, one of the few exeptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of powershell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefor I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirct their arguments to other executable and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are applying. +Sadly, one of the few exceptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of PowerShell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefor I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirect their arguments to other executable and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are applying. ### Considerations outside of the scope of this RFC -Maybe on Linux `--%` and `--=` can be depricated, as they dont make much sense on Linux -- the CommandLine always needs to be escaped acording to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "Linux"). +Maybe on Linux `--%` and `--=` can be deprecated, as they don't make much sense on Linux -- the CommandLine always needs to be escaped acording to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "Linux"). -As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly but instead a way to disable many special characters in a powershell command. In the future one could possibly add a `--$` token that disables all special characters exept quotes and `$` -- a clean way of what `--%` was originally meant for. +As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly but instead a way to disable many special characters in a PowerShell command. In the future one could possibly add a `--$` token that disables all special characters except quotes and `$` -- a clean way of what `--%` was originally meant for. From 6fbf9415c3c1734671d74ed18fc03cc228a38d0b Mon Sep 17 00:00:00 2001 From: TSlivede Date: Fri, 5 May 2017 22:07:20 +0200 Subject: [PATCH 07/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md comments-due-date small corrections --- ...tion-of-argument-string-for-executables.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index d1c725a0..78511891 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -4,7 +4,7 @@ Author: Timo Schwarte Status: Draft Version: 1.0 Area: Calling external executables -Comments Due: 2017-XX-XX +Comments Due: 2017-07-01 Plan to implement: Yes --- @@ -12,29 +12,29 @@ Plan to implement: Yes When PowerShell calls an external executable, the [NativeCommandParameterBinder](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/NativeCommandParameterBinder.cs) -takes all parameters of the command and generates the argument-string (also called "CommandLine") that is sent to the executable (as [ProcessStartInfo.Arguments](https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.arguments.aspx)). -It seems as if the NativeCommandParameterBinder tries to generate an argument-string, such that `argv[]` of the called executable contains the arguments, that were given as parameters to the command. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes around the argument, so it reaches `argv[]` as one argument (not split into parts). -However, if the argument itself contains quotes, those are not escaped. Therefor the corresponding element in `argv[]` has no quotes and (depending on the actual argument) the argument can be split into multiple `argv[]` elements. It can even occur that the following arguments are not handled correctly. +takes all parameters of the command and generates the argument-string (also called "CommandLine"), which is sent to the executable (as [ProcessStartInfo.Arguments](https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.arguments.aspx)). +It seems like the NativeCommandParameterBinder tries to generate an argument-string from the given parameters, such that argv[] of the called executable is set accordingly. This works well, if an argument contains spaces. In this case the NativeCommandParameterBinder adds quotes around the argument, so it appears in `argv[]` as one argument (and is not split into parts). +However, if the argument itself contains quotes, these are not escaped. Therefore, the corresponding element in `argv[]` has no quotes and depending on the actual string, the argument might be split into multiple `argv[]` elements. It can even occur, that the following arguments are not handled correctly. -This RFC suggests to make the NativeCommandParameterBinder compatible to +This RFC suggests making the NativeCommandParameterBinder compatible to [the typical CommandLine escaping rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). -Additionally, it suggests to add a verbose-symbol to add a custom formatted string to the CommandLine (for executables, that don't follow the typical escaping rules.) +Additionally, it suggests adding a verbose-symbol to add a custom formatted string to the CommandLine (for executables, that don't follow the typical escaping rules.) This is basically a bugfix for https://github.com/PowerShell/PowerShell/issues/1995 and https://github.com/PowerShell/PowerShell/issues/3049 and on stackoverflow (["This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command."](http://stackoverflow.com/questions/6714165/powershell-stripping-double-quotes-from-command-line-arguments) and ["... and I think this is a bug Powershell doesn't escape any double quotes that appear inside the arguments."](http://stackoverflow.com/a/21334121/2770331)). -However, as this is a very longstanding bug, there are workarounds for some cases. These will no longer work if this is corrected, therefor this document suggests to add a preference variable to get the old behavior back if somebody needs it. +However, as this is a very longstanding bug, there are workarounds for some cases. These will no longer work if the proposed changes are applied, therefore this document suggests to add a preference variable to get the old behavior back if somebody needs it. ## Motivation As a PowerShell user who wants to call external executables, - I can pass any arguments to those executables, - so that these arguments reach the `argv[]` array unchanged. + I can pass any arguments to these executables, + so that they reach the `argv[]` array unchanged. ## Specification ### Quoting -The decision if an argument needs to be quoted, will be simplified: Any Argument, that contains `"`, `'` or a character that matches `char.IsWhiteSpace` will be quoted. To quote an argument (compatible to [MSVC rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) and `CommandLineToArgvW`): +The decision if an argument needs to be quoted, will be simplified: Any Argument that contains `"`, `'` or a character that matches `char.IsWhiteSpace` will be quoted. To quote an argument (compatible to [MSVC rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) and `CommandLineToArgvW`): - Every occurrence of N times `\` followed by `"` will be replaced by (2*N+1) times `\` followed by `"`. (N ∈ {0,1,2,...}) - N times `\` at the end of the string is replaced by (2*N) times `\`. (N ∈ {0,1,2,...}) - `"` is added to the beginning and to the end of the string @@ -45,7 +45,7 @@ When the parser detects the 'Verbatim-Argument-Symbol' (`--=`), the next argumen ### Preference Variable -If the Preference Variable `$PsUseLegacyInconsistentArgumentStringGeneration` is set to true, the quoting is done as it is now. (Only for compatibility with old Scripts). The Variable is false by default. +If the Preference Variable `$PsUseLegacyArgumentStringGeneration` is set to true, the quoting is done as it is now. (Only for compatibility with old Scripts). The Variable is false by default. ## Alternate Proposals and Considerations @@ -59,21 +59,21 @@ Maybe this is not the strongest argument ever, but many other modern Commandshel ### [These rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are widely accepted -Most Compilers on Windows generate executables, that split the CommandLine compatible to [those rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx), for example the compiler shipped with VisualStudio as well as the `mingw` compiler suite. +Most Compilers on Windows generate executables that split the CommandLine compatible to [those rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx), for example the compiler shipped with VisualStudio or the `mingw` compiler suite. The .Net runtime also splits the CommandLine in a way that is compatible to those rules. The Windows API function [`CommandLineToArgvW`](https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391.aspx) is (although incorrectly documented) also compatible to those rules. -Therefore, PowerShell should build the CommandLine in a way, that matches those rules instead of just sometimes adding quotes around arguments. +Therefore, PowerShell should build the CommandLine in a way that matches those rules instead of just sometimes adding quotes around arguments. ### Linux -While this change is important on windows, it's absolutely necessary on Linux: On Windows the CommandLine is split by the next executable and most executables follow the described rules. On Linux the CommandLine is not split by the called executable -- the .Net Core runtime splits the string. Therefor on Linux the described rules do not only apply to many calls of external executables, they apply to ALL calls of external executables. When the proposed changes are implemented, the arguments from within PowerShell always arrive -- as expected -- as the `argv[]` array in called executables. +While this change is important on windows, it's absolutely necessary on Linux: On Windows the CommandLine is split by the next executable and most executables follow the described rules. On Linux the CommandLine is not split by the called executable -- the .Net Core runtime splits the string. Therefore, on Linux the described rules do not only apply to many calls of external executables, they apply to ALL calls of external executables. When the proposed changes are implemented, the arguments from within PowerShell always arrive -- as expected -- as the `argv[]` array in called executables. ### Batch files -Sadly, one of the few exceptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of PowerShell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefor I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirect their arguments to other executable and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are applying. +Sadly, one of the few exceptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of PowerShell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefore I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirect their arguments to other executable and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) apply. ### Considerations outside of the scope of this RFC -Maybe on Linux `--%` and `--=` can be deprecated, as they don't make much sense on Linux -- the CommandLine always needs to be escaped acording to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "Linux"). +Maybe on Linux `--%` and `--=` can be deprecated, as they don't make much sense on Linux -- the CommandLine always needs to be escaped according to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "Linux"). -As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly but instead a way to disable many special characters in a PowerShell command. In the future one could possibly add a `--$` token that disables all special characters except quotes and `$` -- a clean way of what `--%` was originally meant for. +As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly, but instead a way to disable many special characters in a PowerShell command. In the future one could possibly add a `--$` token that disables all special characters except quotes and `$` -- a proper option that does what `--%` was originally meant for. From a1f07b6648f9ab90d03dc5ba0d5a745a8af497f8 Mon Sep 17 00:00:00 2001 From: TSlivede Date: Tue, 6 Jun 2017 22:02:41 +0200 Subject: [PATCH 08/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md Alternative: special rules for batch files --- ...eneration-of-argument-string-for-executables.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index 78511891..8dc551eb 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -4,7 +4,7 @@ Author: Timo Schwarte Status: Draft Version: 1.0 Area: Calling external executables -Comments Due: 2017-07-01 +Comments Due: 2017-07-07 Plan to implement: Yes --- @@ -57,6 +57,8 @@ Maybe this is not the strongest argument ever, but many other modern Commandshel - [Python](https://github.com/python/cpython/blob/master/Lib/subprocess.py#L424) (see `list2cmdline`) - [Tcl](https://github.com/tcltk/tcl/blob/master/win/tclWinPipe.c#L1503) (see `BuildCommandLine`, Comment: "N backslashes followed a quote -> insert N * 2 + 1 backslashes then a quote.") +[Wine](https://source.winehq.org/git/wine.git/blob/refs/heads/master:/dlls/kernel32/process.c#l730) - although not a shell - also faced this problem and also creates the CommandLine compatible to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). + ### [These rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are widely accepted Most Compilers on Windows generate executables that split the CommandLine compatible to [those rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx), for example the compiler shipped with VisualStudio or the `mingw` compiler suite. @@ -69,11 +71,15 @@ While this change is important on windows, it's absolutely necessary on Linux: O ### Batch files -Sadly, one of the few exceptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of PowerShell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefore I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirect their arguments to other executable and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) apply. +Sadly, one of the few exceptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of PowerShell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefore I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirect their arguments to other executables and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) apply. + +**Edit:** +Optionally a special rule for batch files could be added: `"` in arguments of batch files won't be escaped according to the rules described in [Specification->Quoting](#quoting) -- instead, each literal `"` will be replaced by `""`. Many batch files seem to expect this and this way arguments won't be split into multiple `%1`,`%2`,... variables. +**End edit** ### Considerations outside of the scope of this RFC -Maybe on Linux `--%` and `--=` can be deprecated, as they don't make much sense on Linux -- the CommandLine always needs to be escaped according to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "Linux"). +Maybe on Linux `--%` and `--=` can be deprecated, as they don't make much sense on Linux -- the CommandLine always needs to be escaped according to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "[Linux](#linux)"). -As the main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly, but instead a way to disable many special characters in a PowerShell command. In the future one could possibly add a `--$` token that disables all special characters except quotes and `$` -- a proper option that does what `--%` was originally meant for. +The main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly, but instead a way to disable many special characters in a PowerShell command. In the future one could possibly add a `--$` token that disables all special characters except quotes and `$` -- a proper option that does what `--%` was originally meant for. From 707588914b048a8b5c7fc32fa87c39b886ad12ad Mon Sep 17 00:00:00 2001 From: TSlivede Date: Mon, 28 Aug 2017 20:14:51 +0200 Subject: [PATCH 09/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md Update link for MSVC argument parsing rules --- ...eration-of-argument-string-for-executables.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index 8dc551eb..6b1d388f 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -17,7 +17,7 @@ It seems like the NativeCommandParameterBinder tries to generate an argument-str However, if the argument itself contains quotes, these are not escaped. Therefore, the corresponding element in `argv[]` has no quotes and depending on the actual string, the argument might be split into multiple `argv[]` elements. It can even occur, that the following arguments are not handled correctly. This RFC suggests making the NativeCommandParameterBinder compatible to -[the typical CommandLine escaping rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). +[the typical CommandLine escaping rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments). Additionally, it suggests adding a verbose-symbol to add a custom formatted string to the CommandLine (for executables, that don't follow the typical escaping rules.) This is basically a bugfix for https://github.com/PowerShell/PowerShell/issues/1995 and https://github.com/PowerShell/PowerShell/issues/3049 and on stackoverflow (["This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command."](http://stackoverflow.com/questions/6714165/powershell-stripping-double-quotes-from-command-line-arguments) @@ -34,7 +34,7 @@ However, as this is a very longstanding bug, there are workarounds for some case ### Quoting -The decision if an argument needs to be quoted, will be simplified: Any Argument that contains `"`, `'` or a character that matches `char.IsWhiteSpace` will be quoted. To quote an argument (compatible to [MSVC rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) and `CommandLineToArgvW`): +The decision if an argument needs to be quoted, will be simplified: Any Argument that contains `"`, `'` or a character that matches `char.IsWhiteSpace` will be quoted. To quote an argument (compatible to [MSVC rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments) and `CommandLineToArgvW`): - Every occurrence of N times `\` followed by `"` will be replaced by (2*N+1) times `\` followed by `"`. (N ∈ {0,1,2,...}) - N times `\` at the end of the string is replaced by (2*N) times `\`. (N ∈ {0,1,2,...}) - `"` is added to the beginning and to the end of the string @@ -51,17 +51,17 @@ If the Preference Variable `$PsUseLegacyArgumentStringGeneration` is set to true ### Other shells -Maybe this is not the strongest argument ever, but many other modern Commandshells on windows create the argument-string compatible to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx): +Maybe this is not the strongest argument ever, but many other modern Commandshells on windows create the argument-string compatible to [these rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments): - https://www.hamiltonlabs.com/Cshell.htm (not documented, but escapes correctly) - [Cygwin](https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/winf.cc;hb=HEAD#l66) (see `linebuf::fromargv`) - [Python](https://github.com/python/cpython/blob/master/Lib/subprocess.py#L424) (see `list2cmdline`) - [Tcl](https://github.com/tcltk/tcl/blob/master/win/tclWinPipe.c#L1503) (see `BuildCommandLine`, Comment: "N backslashes followed a quote -> insert N * 2 + 1 backslashes then a quote.") -[Wine](https://source.winehq.org/git/wine.git/blob/refs/heads/master:/dlls/kernel32/process.c#l730) - although not a shell - also faced this problem and also creates the CommandLine compatible to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx). +[Wine](https://source.winehq.org/git/wine.git/blob/refs/heads/master:/dlls/kernel32/process.c#l730) - although not a shell - also faced this problem and also creates the CommandLine compatible to [these rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments). -### [These rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) are widely accepted +### [These rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments) are widely accepted -Most Compilers on Windows generate executables that split the CommandLine compatible to [those rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx), for example the compiler shipped with VisualStudio or the `mingw` compiler suite. +Most Compilers on Windows generate executables that split the CommandLine compatible to [those rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments), for example the compiler shipped with VisualStudio or the `mingw` compiler suite. The .Net runtime also splits the CommandLine in a way that is compatible to those rules. The Windows API function [`CommandLineToArgvW`](https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391.aspx) is (although incorrectly documented) also compatible to those rules. Therefore, PowerShell should build the CommandLine in a way that matches those rules instead of just sometimes adding quotes around arguments. @@ -71,7 +71,7 @@ While this change is important on windows, it's absolutely necessary on Linux: O ### Batch files -Sadly, one of the few exceptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of PowerShell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefore I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirect their arguments to other executables and in those cases [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) apply. +Sadly, one of the few exceptions to those typical parsing rules is `cmd.exe`. Because of this, one cannot reliably call batch files with arbitrary arguments. (This is no problem of PowerShell, it's a cmd design problem.) Some arguments are impossible -- an uneven number of double quotes can only be sent in the last argument. To my knowledge there is no clean way to deal with this, therefore I think using the typical escaping rules is still the way to go. In many cases this is the correct way, as many batch files simply redirect their arguments to other executables and in those cases [these rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments) apply. **Edit:** Optionally a special rule for batch files could be added: `"` in arguments of batch files won't be escaped according to the rules described in [Specification->Quoting](#quoting) -- instead, each literal `"` will be replaced by `""`. Many batch files seem to expect this and this way arguments won't be split into multiple `%1`,`%2`,... variables. @@ -79,7 +79,7 @@ Optionally a special rule for batch files could be added: `"` in arguments of ba ### Considerations outside of the scope of this RFC -Maybe on Linux `--%` and `--=` can be deprecated, as they don't make much sense on Linux -- the CommandLine always needs to be escaped according to [these rules](https://msdn.microsoft.com/en-us/library/17w5ykft.aspx) (see paragraph "[Linux](#linux)"). +Maybe on Linux `--%` and `--=` can be deprecated, as they don't make much sense on Linux -- the CommandLine always needs to be escaped according to [these rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments) (see paragraph "[Linux](#linux)"). The main purpose of `--%` was (as far as I know) not the possibility to influence the CommandLine directly, but instead a way to disable many special characters in a PowerShell command. In the future one could possibly add a `--$` token that disables all special characters except quotes and `$` -- a proper option that does what `--%` was originally meant for. From 2a73f2525cd490a28ae40a17402ef0bf8c9f3552 Mon Sep 17 00:00:00 2001 From: TSlivede Date: Mon, 28 Aug 2017 20:53:00 +0200 Subject: [PATCH 10/10] Update RFC####-Improve-generation-of-argument-string-for-executables.md Quote empty arguments --- ...###-Improve-generation-of-argument-string-for-executables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md index 6b1d388f..90dd6fd0 100644 --- a/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md +++ b/1-Draft/RFC####-Improve-generation-of-argument-string-for-executables.md @@ -34,7 +34,7 @@ However, as this is a very longstanding bug, there are workarounds for some case ### Quoting -The decision if an argument needs to be quoted, will be simplified: Any Argument that contains `"`, `'` or a character that matches `char.IsWhiteSpace` will be quoted. To quote an argument (compatible to [MSVC rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments) and `CommandLineToArgvW`): +The decision if an argument needs to be quoted, will be simplified: Empty arguments and any argument that contains `"`, `'` or a character that matches `char.IsWhiteSpace` will be quoted. To quote an argument (compatible to [MSVC rules](https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments) and `CommandLineToArgvW`): - Every occurrence of N times `\` followed by `"` will be replaced by (2*N+1) times `\` followed by `"`. (N ∈ {0,1,2,...}) - N times `\` at the end of the string is replaced by (2*N) times `\`. (N ∈ {0,1,2,...}) - `"` is added to the beginning and to the end of the string