[Xamarin.Android.Build.Tests] Retry builds on native crashes (#993)

Sometimes when running `Xamarin.Android.Build.Tests`, mono crashes:

	Stacktrace:
	
	  at <unknown> <0xffffffff>
	  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_obj (intptr,intptr) [0x00000] in <4fdc5ed61a074cafb49fa42deb20d521>:0
	  at (wrapper alloc) object.AllocSmall (intptr,intptr) <0x000fa>
	  at System.IO.FileSystemEnumerableIterator`1<TSource_REF>.CreateSearchResult (System.IO.Directory/SearchData,Microsoft.Win32.Win32Native/WIN32_FIND_DATA) [0x00023] in <4fdc5ed61a074cafb49fa42deb20d521>:0
	# Actual stack trace doesn't *really* matter, and is basically random
	...
	Native stacktrace:
	
		0   mono                                0x000000010d9df481 mono_handle_native_crash + 257
	...
	=================================================================
	Got a SIGSEGV while executing native code. This usually indicates
	a fatal error in the mono runtime or one of the native libraries 
	used by your application.
	=================================================================

This is decidedly Not Good™.

However, a significant percent of the time, the crash *also* isn't
reproducible: a re-run of the build+tests will result in all tests
passing. However, a re-run can take upwards of 2-6 *hours* --
depending mostly on whether the PR has a mono bump -- so it is
undesirable to re-run builds just to see if it's a "random" error.

Instead, update `Builder.BuildInternal()` so that if it determines
that mono crashed -- via the `Got a SIGSEGV…` message -- then we'll
re-try the crashing command. (Just once!)

This should hopefully reduce the number of test failures we need
to investigate.

The *downside* is that the #runtime team would dearly love to know
about these crashes, so *they* can investigate. This is *partially*
supported by emitting the following message into the build output
when a native crash is detected:

	Native crash detected! Running the build for … again.

This is less than ideal, and may be improved in the future.

Author: dellis1972

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Common/Builder.cs

@@ -241,37 +241,52 @@ protected bool BuildInternal (string projectOrSolution, string target, string []
 			psi.RedirectStandardError = true;
 			psi.StandardErrorEncoding = Encoding.UTF8;
 			psi.StandardOutputEncoding = Encoding.UTF8;
-			var p = Process.Start (psi);
-			var ranToCompletion = p.WaitForExit ((int)new TimeSpan (0,10,0).TotalMilliseconds);
-			var result = ranToCompletion && p.ExitCode == 0;
 
-			LastBuildTime = DateTime.UtcNow - start;
+			bool nativeCrashDetected = false;
+			bool result = false;
+			int attempts = 1;
+			for (int attempt = 0; attempt < attempts; attempt++) {
+				var p = Process.Start (psi);
+				var ranToCompletion = p.WaitForExit ((int)new TimeSpan (0, 10, 0).TotalMilliseconds);
+				result = ranToCompletion && p.ExitCode == 0;
 
-			var sb = new StringBuilder ();
-			sb.AppendLine ( psi.FileName + " " + args.ToString () + Environment.NewLine);
-			if (!ranToCompletion)
-				sb.AppendLine ("Build Timed Out!");
-			if (buildLogFullPath != null && File.Exists (buildLogFullPath)) {
-				using (var fs = File.OpenRead (buildLogFullPath)) {
-					using (var sr = new StreamReader (fs, Encoding.UTF8, true, 65536)) {
-						string line;
-						while ((line = sr.ReadLine ()) != null) {
-							sb.AppendLine (line);
-							if (line.StartsWith ("Time Elapsed", StringComparison.OrdinalIgnoreCase)) {
-								var match = timeElapsedRegEx.Match (line);
-								if (match.Success) {
-									LastBuildTime = TimeSpan.Parse (match.Groups ["TimeSpan"].Value);
-									Console.WriteLine ($"Found Time Elapsed {LastBuildTime}");
+				LastBuildTime = DateTime.UtcNow - start;
+
+				var sb = new StringBuilder ();
+				sb.AppendLine (psi.FileName + " " + args.ToString () + Environment.NewLine);
+				if (!ranToCompletion)
+					sb.AppendLine ("Build Timed Out!");
+				if (buildLogFullPath != null && File.Exists (buildLogFullPath)) {
+					using (var fs = File.OpenRead (buildLogFullPath)) {
+						using (var sr = new StreamReader (fs, Encoding.UTF8, true, 65536)) {
+							string line;
+							while ((line = sr.ReadLine ()) != null) {
+								sb.AppendLine (line);
+								if (line.StartsWith ("Time Elapsed", StringComparison.OrdinalIgnoreCase)) {
+									var match = timeElapsedRegEx.Match (line);
+									if (match.Success) {
+										LastBuildTime = TimeSpan.Parse (match.Groups ["TimeSpan"].Value);
+										Console.WriteLine ($"Found Time Elapsed {LastBuildTime}");
+									}
+								}
+								if (line.StartsWith ("Got a SIGSEGV while executing native code", StringComparison.OrdinalIgnoreCase)) {
+									nativeCrashDetected = true;
+									break;
 								}
 							}
 						}
 					}
 				}
+				sb.AppendFormat ("\n#stdout begin\n{0}\n#stdout end\n", p.StandardOutput.ReadToEnd ());
+				sb.AppendFormat ("\n#stderr begin\n{0}\n#stderr end\n", p.StandardError.ReadToEnd ());
+
+				LastBuildOutput = sb.ToString ();
+				if (!nativeCrashDetected) {
+					Console.WriteLine ($"Native crash detected! Running the build for {projectOrSolution} again.");
+					continue;
+				}
 			}
-			sb.AppendFormat ("\n#stdout begin\n{0}\n#stdout end\n", p.StandardOutput.ReadToEnd ());
-			sb.AppendFormat ("\n#stderr begin\n{0}\n#stderr end\n", p.StandardError.ReadToEnd ());
 
-			LastBuildOutput = sb.ToString ();
 
 			if (buildLogFullPath != null) {
 				Directory.CreateDirectory (Path.GetDirectoryName (buildLogFullPath));