[xabuild.exe] cross-platform form of tools/scripts/xabuild (#910)

Context: https://github.com/jonathanpeppers/xabuild

What do we want? A *usable*, *parallel*, build tree.

To elaborate: we want to be able to have a "system" Xamarin.Android
install, and a "parallel" xamarin-android install, and be able to
*easily* switch between the two. (For various definitions of "easy";
here, we mean *command-line* use, not IDE use.)

On macOS, this (more or less) Just Works™, and is extremely handy
for testing bug fixes:

	$ xbuild /t:Install Project.csproj
	# Verify that some bug is triggered

	$ xbuild /t:Uninstall Project.csproj
	$ tools/scripts/xabuild /t:Install Project.csproj
	# Verify that some bug is *fixed*

There's One Problem™ with this: MSBuild does not make this easy.
(Related: commit aa1db83.) Apps may rely on files located within
`$(MSBuildExtensionsPath)` or `$(TargetFrameworkRootPath)`, files
such as PCL profile assemblies, or 3rd party frameworks.

Meanwhile, on macOS, `xabuild` is *predicated* upon overriding
`$(TargetFrameworkRootPath)` and `$(MSBuildExtensionsPath)` and
`$(XBUILD_FRAMEWORK_FOLDERS_PATH)`, and creating a bunch of symlinks
to "fake out" `msbuild.exe` so that system-installed files such as
PCL assemblies can be found *through* the parallel environment.

It kinda/sorta works on macOS. It completely falls apart when using
Windows. There is no easy "symlink half the world" solution there.
Overriding `$(MSBuildExtensionsPath)` means that
`Microsoft.Common.targets` can't be found. Overriding
`$(TargetFrameworkRootPath)` means PCL files can't be found.

It's a mess.

Fortunately, more recent versions of MSBuild allow for some of these
properties to contain *multiple* directories instead of a single
directory, which means *there is a way* to support our desired
usable parallel install world order. We "just" need to e.g.
force `$(MSBuildExtensionsPath)` to contain *both* the in-tree
directory *and* the system directory:

```xml
<MSBuildExtensionsPath>In-Tree Directory; System Directory</MSBuildExtensionsPath>
```

Unfortunately, *this isn't easy*. Not all of these properties can be
overridden on the `msbuild.exe` command line. Worse, MSBuild doesn't
allow `;` to be part of a property value, as `;` is a property name
[separator char](https://msdn.microsoft.com/en-us/library/ms164311.aspx)

	msbuild.exe /property:WarningLevel=2;OutDir=bin\Debug

The way to force MSBuild to accept multiple paths in a property value
is by providing a `.exe.config` file with the appropriate values.

~~ Enter `xabuild.exe` ~~

`xabuild.exe` is a nice wrapper around MSBuild for compiling
Xamarin.Android projects using a locally built version of
Xamarin.Android on your system. It seems to work on Windows, macOS, and
Linux and doesn’t require elevation or modifications to your system.

`xabuild.exe` works by doing the following:

 1. Reference `MSBuild.exe` or `MSBuild.dll` depending on the platform
 2. Creates symbolic links to `.NETPortable` and `.NETFramework`
    directories inside the Xamarin.Android build output directory
 3. Overrides MSBuild's `app.config` file to set various properties,
    such as `$(MSBuildExtensionsPath)`.
 4. Run MSBuild’s `Main()` method

~~ Usage ~~

On macOS, `tools/scripts/xabuild` has been updated to use
`xabuild.exe` when `$MSBUILD` is `msbuild:

	$ MSBUILD=msbuild tools/scripts/xabuild /t:SignAndroidPackage samples/HelloWorld/HelloWorld.csproj

When `$MSBUILD` is `xbuild` (the current default), the previous
behavior of overriding `$MSBuildExtensionsPath` and
`$XBUILD_FRAMEWORK_FOLDERS_PATH` is still used.

On Windows, MSBuild 15.3 from Visual Studio 2017 is required.
Simply execute `xabuild.exe`:

	> bin\Debug\bin\xabuild.exe Xamarin.Android-Tests.sln /p:XAIntegratedTests=False

Before `xabuild.exe` existed, `setup-windows.exe` would need to be
executed (as Administrator!) in order for `Xamarin.Android-Tests.sln`
to be built using `msbuild.exe`.

Author: jonathanpeppers

Documentation/DevelopmentTips.md

@@ -116,6 +116,17 @@ to hold file locks on output assemblies containing MSBuild tasks. Until there is
 for this, it might be more advisable to use an editor like Visual Studio Code and build via
 the command-line.
 
+Windows also requires `xabuild.exe` in place of the `tools/scripts/xabuild` script used
+on other platforms.
+
+So a command on macOS such as:
+
+	$ tools/scripts/xabuild /t:SignAndroidPackage tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj
+
+Would be run on Windows as:
+
+	> bin\Debug\bin\xabuild.exe /t:SignAndroidPackage tests\locales\Xamarin.Android.Locale-Tests\Xamarin.Android.Locale-Tests.csproj
+
 # Unit Tests
 
 The `xamarin-android` repo contains several unit tests:

Xamarin.Android.sln

@@ -105,6 +105,8 @@ Project("{9344BDBB-3E7F-41FC-A0DD-8665D75EE146}") = "netstandard", "src\netstand
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "setup-windows", "tools\setup-windows\setup-windows.csproj", "{73DF9E10-E933-4222-B8E1-F4536FFF9FAD}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "xabuild", "tools\xabuild\xabuild.csproj", "{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}"
+EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Mono.Android", "src\Mono.Android\Mono.Android.csproj", "{66CF299A-CE95-4131-BCD8-DB66E30C4BF7}"
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Mono.Android.Export", "src\Mono.Android.Export\Mono.Android.Export.csproj", "{B8105878-D423-4159-A3E7-028298281EC6}"
@@ -531,6 +533,18 @@ Global
 		{1E5501E8-49C1-4659-838D-CC9720C5208F}.XAIntegrationDebug|Any CPU.Build.0 = Debug|Any CPU
 		{1E5501E8-49C1-4659-838D-CC9720C5208F}.XAIntegrationRelease|Any CPU.ActiveCfg = Release|Any CPU
 		{1E5501E8-49C1-4659-838D-CC9720C5208F}.XAIntegrationRelease|Any CPU.Build.0 = Release|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.Debug|AnyCPU.ActiveCfg = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.Debug|AnyCPU.Build.0 = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.Release|AnyCPU.ActiveCfg = Release|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.Release|AnyCPU.Build.0 = Release|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationDebug|AnyCPU.ActiveCfg = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationDebug|AnyCPU.Build.0 = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationRelease|AnyCPU.ActiveCfg = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationRelease|AnyCPU.Build.0 = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationDebug|Any CPU.ActiveCfg = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationDebug|Any CPU.Build.0 = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationRelease|Any CPU.ActiveCfg = Debug|Any CPU
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D}.XAIntegrationRelease|Any CPU.Build.0 = Debug|Any CPU
 	EndGlobalSection
 	GlobalSection(NestedProjects) = preSolution
 		{8FF78EB6-6FC8-46A7-8A15-EBBA9045C5FA} = {E351F97D-EA4F-4E7F-AAA0-8EBB1F2A4A62}
@@ -585,6 +599,7 @@ Global
 		{B8105878-D423-4159-A3E7-028298281EC6} = {04E3E11E-B47D-4599-8AFC-50515A95E715}
 		{E34BCFA0-CAA4-412C-AA1C-75DB8D67D157} = {04E3E11E-B47D-4599-8AFC-50515A95E715}
 		{1E5501E8-49C1-4659-838D-CC9720C5208F} = {CAB438D8-B0F5-4AF0-BEBD-9E2ADBD7B483}
+		{B7A457E6-9CB6-43F6-BFD6-14D5397FB98D} = {864062D3-A415-4A6F-9324-5820237BA058}
 	EndGlobalSection
 	GlobalSection(MonoDevelopProperties) = preSolution
 		Policies = $0

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Common/Builder.cs

@@ -10,11 +10,6 @@ namespace Xamarin.ProjectTools
 {
 	public class Builder : IDisposable
 	{
-		const string fixed_osx_xbuild_path = "/Library/Frameworks/Mono.framework/Commands";
-		const string fixed_linux_xbuild_path = "/usr/bin";
-		const string xbuildapp = "xbuild";
-		const string msbuildapp = "msbuild";
-
 		public bool IsUnix { get; set; }
 		public bool RunningMSBuild { get; set; }
 		public LoggerVerbosity Verbosity { get; set; }
@@ -23,18 +18,6 @@ public class Builder : IDisposable
 		public string BuildLogFile { get; set; }
 		public bool ThrowOnBuildFailure { get; set; }
 
-		string GetUnixBuildExe ()
-		{
-			RunningMSBuild = false;
-			var tooldir = Directory.Exists (fixed_osx_xbuild_path) ? fixed_osx_xbuild_path : fixed_linux_xbuild_path;
-			string path = Path.Combine (tooldir, xbuildapp);
-			if (!string.IsNullOrEmpty (Environment.GetEnvironmentVariable ("USE_MSBUILD"))) {
-				path = Path.Combine (tooldir, msbuildapp);
-				RunningMSBuild = true;
-			}
-			return File.Exists (path) ? path : msbuildapp;
-		}
-
 		string GetVisualStudio2017Directory ()
 		{
 			var editions = new [] {
@@ -54,38 +37,20 @@ string GetVisualStudio2017Directory ()
 			return null;
 		}
 
-		string GetWindowsBuildExe ()
-		{
-			RunningMSBuild = true;
-
-			//First try environment variable
-			string msbuildExe = Environment.GetEnvironmentVariable ("XA_MSBUILD_EXE");
-			if (!string.IsNullOrEmpty (msbuildExe) && File.Exists (msbuildExe))
-				return msbuildExe;
-
-			//Next try VS 2017, MSBuild 15.0
-			var visualStudioDirectory = GetVisualStudio2017Directory ();
-			if (!string.IsNullOrEmpty(visualStudioDirectory)) {
-				msbuildExe = Path.Combine (visualStudioDirectory, "MSBuild", "15.0", "Bin", "MSBuild.exe");
-
-				if (File.Exists (msbuildExe))
-					return msbuildExe;
-			}
-
-			//Try older than VS 2017, MSBuild 14.0
-			msbuildExe = Path.Combine (Environment.GetFolderPath (Environment.SpecialFolder.ProgramFilesX86), "MSBuild", "14.0", "Bin", "MSBuild.exe");
-			if (File.Exists (msbuildExe))
-				return msbuildExe;
-			
-			//MSBuild 4.0 last resort
-			return Path.Combine (Environment.GetFolderPath (Environment.SpecialFolder.Windows), "Microsoft.NET", "Framework", "v4.0.30319", "MSBuild.exe");
-		}
-
-		public string MSBuildExe {
+		public string XABuildExe {
 			get {
-				return IsUnix
-					? GetUnixBuildExe ()
-					: GetWindowsBuildExe ();
+				if (IsUnix) {
+					if (!string.IsNullOrEmpty (Environment.GetEnvironmentVariable ("USE_MSBUILD"))) {
+						RunningMSBuild = true;
+					}
+					return Path.GetFullPath (Path.Combine (Root, "..", "..", "tools", "scripts", "xabuild"));
+				}
+
+				#if DEBUG
+				return Path.GetFullPath (Path.Combine (Root, "..", "..", "bin", "Debug", "bin", "xabuild.exe"));
+				#else
+				return Path.GetFullPath (Path.Combine (Root, "..", "..", "bin", "Release", "bin", "xabuild.exe"));
+				#endif
 			}
 		}
 
@@ -200,57 +165,22 @@ protected bool BuildInternal (string projectOrSolution, string target, string []
 					buildLogFullPath, Verbosity.ToString ().ToLower ());
 
 			var start = DateTime.UtcNow;
-			var homeDirectory = Environment.GetFolderPath (Environment.SpecialFolder.Personal);
-			var androidSdkToolPath = Path.Combine (homeDirectory, "android-toolchain");
-			var sdkPath = Environment.GetEnvironmentVariable ("ANDROID_SDK_PATH");
-			if (String.IsNullOrEmpty (sdkPath))
-				sdkPath = GetPathFromRegistry ("AndroidSdkDirectory");
-			if (String.IsNullOrEmpty (sdkPath))
-				sdkPath = Path.GetFullPath (Path.Combine (androidSdkToolPath, "sdk"));
-			var ndkPath = Environment.GetEnvironmentVariable ("ANDROID_NDK_PATH");
-			if (String.IsNullOrEmpty (ndkPath))
-				ndkPath = GetPathFromRegistry ("AndroidNdkDirectory");
-			if (String.IsNullOrEmpty (ndkPath))
-				ndkPath = Path.GetFullPath (Path.Combine (androidSdkToolPath, "ndk"));
-			StringBuilder args = new StringBuilder ();
-			var psi = new ProcessStartInfo (MSBuildExe);
-			if (IsUnix) {
-				if (Directory.Exists (sdkPath)) {
-					args.AppendFormat ("/p:AndroidSdkDirectory=\"{0}\" ", sdkPath);
-				}
-				if (Directory.Exists (ndkPath)) {
-					args.AppendFormat ("/p:AndroidNdkDirectory=\"{0}\" ", ndkPath);
-				}
-				var outdir = Path.GetFullPath (Path.Combine (FrameworkLibDirectory, "..", ".."));
-				var targetsdir = Path.Combine (FrameworkLibDirectory, "xbuild");
-				args.AppendFormat (" {0} ", logger);
+			var args  = new StringBuilder ();
+			var psi   = new ProcessStartInfo (XABuildExe);
+			args.AppendFormat ("{0} /t:{1} {2} /p:UseHostCompilerIfAvailable=false /p:BuildingInsideVisualStudio=true",
+				QuoteFileName(Path.Combine (Root, projectOrSolution)), target, logger);
 
-				if (Directory.Exists (targetsdir)) {
-					psi.EnvironmentVariables ["TARGETS_DIR"] = targetsdir;
-					psi.EnvironmentVariables ["MSBuildExtensionsPath"] = targetsdir;
-				}
-				if (Directory.Exists (outdir)) {
-					var frameworksPath = Path.Combine (outdir, "lib", "xamarin.android", "xbuild-frameworks");
-					psi.EnvironmentVariables ["MONO_ANDROID_PATH"] = outdir;
-					args.AppendFormat ("/p:MonoDroidInstallDirectory=\"{0}\" ", outdir);
-					psi.EnvironmentVariables ["XBUILD_FRAMEWORK_FOLDERS_PATH"] = frameworksPath;
-					if (RunningMSBuild)
-						args.AppendFormat ($"/p:TargetFrameworkRootPath={frameworksPath} ");
-				}
-				args.AppendFormat ("/t:{0} {1} /p:UseHostCompilerIfAvailable=false /p:BuildingInsideVisualStudio=true", target, QuoteFileName (Path.Combine (Root, projectOrSolution)));
-			}
-			else {
-				args.AppendFormat ("{0} /t:{1} {2} /p:UseHostCompilerIfAvailable=false /p:BuildingInsideVisualStudio=true",
-					QuoteFileName(Path.Combine (Root, projectOrSolution)), target, logger);
-			}
 			if (parameters != null) {
 				foreach (var param in parameters) {
 					args.AppendFormat (" /p:{0}", param);
 				}
 			}
+			if (RunningMSBuild) {
+				psi.EnvironmentVariables ["MSBUILD"] = "msbuild";
+			}
 			if (environmentVariables != null) {
 				foreach (var kvp in environmentVariables) {
-					psi.EnvironmentVariables[kvp.Key] = kvp.Value;
+					psi.EnvironmentVariables [kvp.Key] = kvp.Value;
 				}
 			}
 			psi.Arguments = args.ToString ();

tools/scripts/xabuild

@@ -43,7 +43,7 @@ if [ -z "$MSBUILD" ] ; then
 fi
 
 if [ -z "$CONFIGURATION" ]; then
-	for p in "$*"; do
+	for p in "$@"; do
 		case $p in
 			/property:Configuration=*|                                                   \
 			/p:Configuration=*|                                                          \
@@ -82,6 +82,11 @@ else
 	exit 1
 fi
 
+if [[ "$MSBUILD" == "msbuild" ]] ; then
+	exec mono "$prefix/bin/xabuild.exe" "$@"
+	exit $?
+fi
+
 for t in "$TARGETS_DIR" "$prefix/lib/mono/xbuild" "$xa_prefix/xbuild" ; do
 	if [ -z "$t" -o ! -d "$t" ]; then
 		continue

tools/xabuild/App.config

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<configuration>
+  <startup useLegacyV2RuntimeActivationPolicy="true">
+    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6" />
+  </startup>
+  <runtime>
+    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Build.Framework" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />
+        <bindingRedirect oldVersion="0.0.0.0-99.9.9.9" newVersion="15.1.0.0" />
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Build" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />
+        <bindingRedirect oldVersion="0.0.0.0-99.9.9.9" newVersion="15.1.0.0" />
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Build.Tasks.Core" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />
+        <bindingRedirect oldVersion="0.0.0.0-99.9.9.9" newVersion="15.1.0.0" />
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Build.Utilities.Core" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />
+        <bindingRedirect oldVersion="0.0.0.0-99.9.9.9" newVersion="15.1.0.0" />
+      </dependentAssembly>
+    </assemblyBinding>
+  </runtime>
+</configuration>

tools/xabuild/Properties/AssemblyInfo.cs

@@ -0,0 +1,26 @@
+using System.Reflection;
+using System.Runtime.CompilerServices;
+
+// Information about this assembly is defined by the following attributes. 
+// Change them to the values specific to your project.
+
+[assembly: AssemblyTitle ("xabuild")]
+[assembly: AssemblyDescription ("")]
+[assembly: AssemblyConfiguration ("")]
+[assembly: AssemblyCompany ("Microsoft Corporation")]
+[assembly: AssemblyProduct ("")]
+[assembly: AssemblyCopyright ("")]
+[assembly: AssemblyTrademark ("")]
+[assembly: AssemblyCulture ("")]
+
+// The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}".
+// The form "{Major}.{Minor}.*" will automatically update the build and revision,
+// and "{Major}.{Minor}.{Build}.*" will update just the revision.
+
+[assembly: AssemblyVersion ("1.0.*")]
+
+// The following attributes are used to specify the signing key for the assembly, 
+// if desired. See the Mono documentation for more information about signing.
+
+//[assembly: AssemblyDelaySign(false)]
+//[assembly: AssemblyKeyFile("")]

tools/xabuild/SymbolicLink.cs

@@ -0,0 +1,46 @@
+using System;
+using System.ComponentModel;
+using System.IO;
+using System.Runtime.InteropServices;
+using Mono.Unix;
+
+namespace Xamarin.Android.Build
+{
+	static class SymbolicLink
+	{
+		public static bool Create (string source, string target)
+		{
+			if (!Directory.Exists (source)) {
+				if (Environment.OSVersion.Platform == PlatformID.Win32NT) {
+					//NOTE: attempt with and without the AllowUnprivilegedCreate flag, seems to fix Windows Server 2016
+					if (!CreateSymbolicLink (source, target, SymbolLinkFlag.Directory | SymbolLinkFlag.AllowUnprivilegedCreate) &&
+						!CreateSymbolicLink (source, target, SymbolLinkFlag.Directory)) {
+						var error = new Win32Exception ().Message;
+						Console.Error.WriteLine ($"Unable to create symbolic link from `{source}` to `{target}`: {error}");
+						return false;
+					}
+				} else {
+					try {
+						var fileInfo = new UnixFileInfo (target);
+						fileInfo.CreateSymbolicLink (source);
+					} catch (Exception exc) {
+						Console.Error.WriteLine ($"Unable to create symbolic link from `{source}` to `{target}`: {exc.Message}");
+						return false;
+					}
+				}
+			}
+
+			return true;
+		}
+
+		enum SymbolLinkFlag {
+			File = 0,
+			Directory = 1,
+			AllowUnprivilegedCreate = 2,
+		}
+
+		[DllImport ("kernel32.dll")]
+		[return: MarshalAs (UnmanagedType.I1)]
+		static extern bool CreateSymbolicLink (string lpSymlinkFileName, string lpTargetFileName, SymbolLinkFlag dwFlags);
+	}
+}