[Xamarin.Android.Build.Tasks] Path is too long

Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=30147

Windows (still) has a max path limit of 260 characters..
This can cause us a problem if the user creates a project in
a directory which already takes up most of that limit.. Like on
their Desktop!

This is because of the intermediate structure that was introduced
to handle embedding resources into the assemblies. The current
intermediate structure is as follows

	`$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\library_project_imports\`
	`$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\native_library_imports\`

Now consider that `$(AssemblyName)` can sometimes end up with something lile
"Xamarin.Android.Support.v7.AppCompat.21.0.3.0" you can easily see how we
start getting into trouble.

There was an attempt to fix this up a while ago (722dcc05) by introducing the
`$(UseShortFileNames)` property. However while this did introduce a
directory structure which was shorter is broke backwards compatability
with older versions of xamarin-android.

This is because of the structore of the zip files we that are embedded
into the assemblies. The current system just extracts the zip into
the `$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\`
directory and assumes that it contains a `library_project_imports` directory.

All the build tasks also assumed that as well.

So the fix needs to be done on a number of fronts. Firstly we need to
update the `$(_LibraryProjectImportsDirectoryName)` and
`$(_NativeLibraryImportsDirectoryName)` to be something shorter.
Next up we need to shorten "__library_project_imports__" to something
else verbose as well. This should cut down on the amount of the MAX_PATH
we chew up.

The real key to this is to NOT change the structure of the zip files in the
assemblies! So when we generate a zip from "jlibs" we make sure that the
folder in the zip is called "library_project_imports". And when we extract
the zip file we makle sure that "library_project_imports" is replaced by
the shorter name.
This will ensure that we are backward compatbile with older versions BUT
more importantly we get to use the shorter directory structure.
The files for native librarys are not extracted to disk but are extracted
from memory so as long as the structure remains the same i.e "native_library_imports"
that code does not need to change.

The other thing we need to do is to update ResolveLibraryProjectImports Task to
upgrade the system when it runs. So if we already have a "libraryprojectimports.cache"
in place we just use that as is. But if we re-run the ResolveLibraryProjectImports
task (due to a change or a clean build) we detect if we have the older structure in place
and just remove it.. Since we are going to regenerate the entire cache again anyway
we might as well start from scratch.

With this in place it becomes possible we can now enable `$(UseShortFileNames)`
by default!

So the new structure that is created is as follows

	`$(IntermediateOutputPath)\lp\<Hash>\jl`
	`$(IntermediateOutputPath)\lp\<Hash>\nl`

The <Hash> will be the MD2 hash of the assemblyName which produces
a 10 digit value. The old behaviour can be enabled by setting
`$(UseShortFileNames)` to `False`.

Note in order to keep existing bindings generator behaviour consistent,
the BindingsGenerator task will not use the `$(UseShortFileNames)`
property to control how it generates its .cs files. Instead a new
property

	`$(UseShortGeneratorFileNames)`

which can be used to control if the generator produces short names
(e.g 1.cs, 2.cs). This will be `False` by default.

Author: dellis1972

src/Xamarin.Android.Build.Tasks/Tasks/CreateLibraryResourceArchive.cs

@@ -94,7 +94,7 @@ public override bool Execute ()
 
 			if (Files.ArchiveZip (outpath, f => {
 				using (var zip = new ZipArchiveEx (f)) {
-					zip.AddDirectory (OutputDirectory, outDirInfo.Name);
+					zip.AddDirectory (OutputDirectory, "library_project_imports");
 				}
 			})) {
 				Log.LogDebugMessage ("Saving contents to " + outpath);

src/Xamarin.Android.Build.Tasks/Tasks/CreateManagedLibraryResourceArchive.cs

@@ -108,7 +108,7 @@ public override bool Execute ()
 
 			if (Files.ArchiveZip (outpath, f => {
 				using (var zip = new ZipArchiveEx (f)) {
-					zip.AddDirectory (OutputDirectory, outDirInfo.Name);
+					zip.AddDirectory (OutputDirectory, "library_project_imports");
 				}
 			})) {
 				Log.LogDebugMessage ("Saving contents to " + outpath);

src/Xamarin.Android.Build.Tasks/Tasks/CreateNativeLibraryArchive.cs

@@ -66,7 +66,7 @@ public override bool Execute ()
 
 			if (Files.ArchiveZip (outpath, f => {
 				using (var zip = new ZipArchiveEx (f)) {
-					zip.AddDirectory (OutputDirectory, outDirInfo.Name);
+					zip.AddDirectory (OutputDirectory, "native_library_imports");
 				}
 			})) {
 				Log.LogDebugMessage ("Saving contents to " + outpath);

src/Xamarin.Android.Build.Tasks/Tasks/ResolveLibraryProjectImports.cs

@@ -51,13 +51,12 @@ public class ResolveLibraryProjectImports : Task
 		[Output]
 		public ITaskItem [] ResolvedResourceDirectoryStamps { get; set; }
 
-		string        imports_dir = "library_project_imports";
-
 		public ResolveLibraryProjectImports ()
 		{
 		}
 
 		// Extracts library project contents under e.g. obj/Debug/[__library_projects__/*.jar | res/*/*]
+		// Extracts library project contents under e.g. obj/Debug/[lp/*.jar | res/*/*]
 		public override bool Execute ()
 		{
 			Log.LogDebugMessage ("ResolveLibraryProjectImports Task");
@@ -135,13 +134,20 @@ static string GetTargetAssembly (ITaskItem assemblyName)
 		}
 
 		// Extracts library project contents under e.g. obj/Debug/[__library_projects__/*.jar | res/*/*]
+		// Extracts library project contents under e.g. obj/Debug/[lp/*.jar | res/*/*]
 		void Extract (
 				DirectoryAssemblyResolver res,
 				ICollection<string> jars,
 				ICollection<string> resolvedResourceDirectories,
 				ICollection<string> resolvedAssetDirectories,
 				ICollection<string> resolvedEnvironments)
 		{
+			// lets "upgrade" the old directory.
+			string oldPath = Path.GetFullPath (Path.Combine (OutputImportDirectory, "..", "__library_projects__"));
+			if (!OutputImportDirectory.Contains ("__library_projects__") && Directory.Exists (oldPath)) {
+				MonoAndroidHelper.SetDirectoryWriteable (oldPath);
+				Directory.Delete (oldPath, recursive: true);
+			}
 			var outdir = new DirectoryInfo (OutputImportDirectory);
 			if (!outdir.Exists)
 				outdir.Create ();
@@ -154,26 +160,95 @@ void Extract (
 					.Select (a => GetTargetAssembly (a))
 					.Where (a => a != null)
 					.Distinct ()) {
-				foreach (var imp in new string [] {imports_dir, "library_project_imports"}.Distinct ()) {
-					string assemblyIdentName = Path.GetFileNameWithoutExtension (assemblyPath);
-					if (UseShortFileNames) {
-						assemblyIdentName = Xamarin.Android.Tasks.MonoAndroidHelper.GetLibraryImportDirectoryNameForAssembly (assemblyIdentName);
-					}
-					string outDirForDll = Path.Combine (OutputImportDirectory, assemblyIdentName);
-					string importsDir = Path.Combine (outDirForDll, imp);
+				string assemblyIdentName = Path.GetFileNameWithoutExtension (assemblyPath);
+				if (UseShortFileNames) {
+					assemblyIdentName = Xamarin.Android.Tasks.MonoAndroidHelper.GetLibraryImportDirectoryNameForAssembly (assemblyIdentName);
+				}
+				string outDirForDll = Path.Combine (OutputImportDirectory, assemblyIdentName);
+				string importsDir = Path.Combine (outDirForDll, ImportsDirectory);
+#if SEPARATE_CRUNCH
+				// FIXME: review these binResDir thing and enable this. Eclipse does this.
+				// Enabling these blindly causes build failure on ActionBarSherlock.
+				//string binResDir = Path.Combine (importsDir, "bin", "res");
+				//string binAssemblyDir = Path.Combine (importsDir, "bin", "assets");
+#endif
+				string resDir = Path.Combine (importsDir, "res");
+				string assemblyDir = Path.Combine (importsDir, "assets");
+
+				// Skip already-extracted resources.
+				var stamp = new FileInfo (Path.Combine (outdir.FullName, assemblyIdentName + ".stamp"));
+				if (stamp.Exists && stamp.LastWriteTime > new FileInfo (assemblyPath).LastWriteTime) {
+					Log.LogDebugMessage ("Skipped resource lookup for {0}: extracted files are up to date", assemblyPath);
 #if SEPARATE_CRUNCH
-					// FIXME: review these binResDir thing and enable this. Eclipse does this.
+					// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
 					// Enabling these blindly causes build failure on ActionBarSherlock.
-					//string binResDir = Path.Combine (importsDir, "bin", "res");
-					//string binAssemblyDir = Path.Combine (importsDir, "bin", "assets");
+					if (Directory.Exists (binResDir))
+						resolvedResourceDirectories.Add (binResDir);
+					if (Directory.Exists (binAssemblyDir))
+						resolvedAssetDirectories.Add (binAssemblyDir);
 #endif
-					string resDir = Path.Combine (importsDir, "res");
-					string assemblyDir = Path.Combine (importsDir, "assets");
+					if (Directory.Exists (resDir))
+						resolvedResourceDirectories.Add (resDir);
+					if (Directory.Exists (assemblyDir))
+						resolvedAssetDirectories.Add (assemblyDir);
+					continue;
+				}
+
+				if (Directory.Exists (outDirForDll))
+					Directory.Delete (outDirForDll, true);
+
+				var assembly = res.GetAssembly (assemblyPath);
+
+				foreach (var mod in assembly.Modules) {
+					// android environment files
+					foreach (var envtxt in mod.Resources
+							.Where (r => r.Name.StartsWith ("__AndroidEnvironment__", StringComparison.OrdinalIgnoreCase))
+							.Where (r => r is EmbeddedResource)
+							.Cast<EmbeddedResource> ()) {
+						if (!Directory.Exists (outDirForDll))
+							Directory.CreateDirectory (outDirForDll);
+						var finfo = new FileInfo (Path.Combine (outDirForDll, envtxt.Name));
+						using (var fs = finfo.Create ()) {
+							var data = envtxt.GetResourceData ();
+							fs.Write (data, 0, data.Length);
+						}
+						resolvedEnvironments.Add (finfo.FullName);
+					}
+
+					// embedded jars (EmbeddedJar, EmbeddedReferenceJar)
+					var resjars = mod.Resources
+						.Where (r => r.Name.EndsWith (".jar", StringComparison.InvariantCultureIgnoreCase))
+						.Select (r => (EmbeddedResource) r);
+					foreach (var resjar in resjars) {
+						var data = resjar.GetResourceData ();
+						if (!Directory.Exists (importsDir))
+							Directory.CreateDirectory (importsDir);
+						using (var outfs = File.Create (Path.Combine (importsDir, resjar.Name)))
+							outfs.Write (data, 0, data.Length);
+					}
+
+					// embedded AndroidResourceLibrary archive
+					var reszip = mod.Resources.FirstOrDefault (r => r.Name == "__AndroidLibraryProjects__.zip") as EmbeddedResource;
+					if (reszip != null) {
+						if (!Directory.Exists (outDirForDll))
+							Directory.CreateDirectory (outDirForDll);
+						var finfo = new FileInfo (Path.Combine (outDirForDll, reszip.Name));
+						using (var fs = finfo.Create ()) {
+							var data = reszip.GetResourceData ();
+							fs.Write (data, 0, data.Length);
+						}
+
+						// temporarily extracted directory will look like:
+						//    __library_projects__/[dllname]/[library_project_imports | jlibs]/bin
+						using (var zip = MonoAndroidHelper.ReadZipFile (finfo.FullName)) {
+							Files.ExtractAll (zip, outDirForDll, modifyCallback: (entryFullName) => {
+								return entryFullName.Replace ("library_project_imports", ImportsDirectory);
+							});
+						}
 
-					// Skip already-extracted resources.
-					var stamp = new FileInfo (Path.Combine (outdir.FullName, assemblyIdentName + ".stamp"));
-					if (stamp.Exists && stamp.LastWriteTime > new FileInfo (assemblyPath).LastWriteTime) {
-						Log.LogDebugMessage ("Skipped resource lookup for {0}: extracted files are up to date", assemblyPath);
+						// We used to *copy* the resources to overwrite other resources,
+						// which resulted in missing resource issue.
+						// Here we replaced copy with use of '-S' option and made it to work.
 #if SEPARATE_CRUNCH
 						// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
 						// Enabling these blindly causes build failure on ActionBarSherlock.
@@ -186,80 +261,13 @@ void Extract (
 							resolvedResourceDirectories.Add (resDir);
 						if (Directory.Exists (assemblyDir))
 							resolvedAssetDirectories.Add (assemblyDir);
-						continue;
-					}
-
-					if (Directory.Exists (outDirForDll))
-						Directory.Delete (outDirForDll, true);
-
-					Directory.CreateDirectory (importsDir);
-
-					var assembly = res.GetAssembly (assemblyPath);
-
-					foreach (var mod in assembly.Modules) {
-						// android environment files
-						foreach (var envtxt in mod.Resources
-								.Where (r => r.Name.StartsWith ("__AndroidEnvironment__", StringComparison.OrdinalIgnoreCase))
-								.Where (r => r is EmbeddedResource)
-								.Cast<EmbeddedResource> ()) {
-							if (!Directory.Exists (outDirForDll))
-								Directory.CreateDirectory (outDirForDll);
-							var finfo = new FileInfo (Path.Combine (outDirForDll, envtxt.Name));
-							using (var fs = finfo.Create ()) {
-								var data = envtxt.GetResourceData ();
-								fs.Write (data, 0, data.Length);
-							}
-							resolvedEnvironments.Add (finfo.FullName);
-						}
-
-						// embedded jars (EmbeddedJar, EmbeddedReferenceJar)
-						var resjars = mod.Resources
-							.Where (r => r.Name.EndsWith (".jar", StringComparison.InvariantCultureIgnoreCase))
-							.Select (r => (EmbeddedResource) r);
-						foreach (var resjar in resjars) {
-							var data = resjar.GetResourceData ();
-							using (var outfs = File.Create (Path.Combine (importsDir, resjar.Name)))
-								outfs.Write (data, 0, data.Length);
-						}
-
-						// embedded AndroidResourceLibrary archive
-						var reszip = mod.Resources.FirstOrDefault (r => r.Name == "__AndroidLibraryProjects__.zip") as EmbeddedResource;
-						if (reszip != null) {
-							if (!Directory.Exists (outDirForDll))
-								Directory.CreateDirectory (outDirForDll);
-							var finfo = new FileInfo (Path.Combine (outDirForDll, reszip.Name));
-							using (var fs = finfo.Create ()) {
-								var data = reszip.GetResourceData ();
-								fs.Write (data, 0, data.Length);
-							}
-
-							// temporarily extracted directory will look like:
-							//    __library_projects__/[dllname]/[library_project_imports | jlibs]/bin
-							using (var zip = MonoAndroidHelper.ReadZipFile (finfo.FullName))
-								Files.ExtractAll (zip, outDirForDll);
-
-							// We used to *copy* the resources to overwrite other resources,
-							// which resulted in missing resource issue.
-							// Here we replaced copy with use of '-S' option and made it to work.
-#if SEPARATE_CRUNCH
-							// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
-							// Enabling these blindly causes build failure on ActionBarSherlock.
-							if (Directory.Exists (binResDir))
-								resolvedResourceDirectories.Add (binResDir);
-							if (Directory.Exists (binAssemblyDir))
-								resolvedAssetDirectories.Add (binAssemblyDir);
-#endif
-							if (Directory.Exists (resDir))
-								resolvedResourceDirectories.Add (resDir);
-							if (Directory.Exists (assemblyDir))
-								resolvedAssetDirectories.Add (assemblyDir);
 
-							finfo.Delete ();
-						}
+						finfo.Delete ();
 					}
+				}
 
+				if (Directory.Exists (importsDir))
 					stamp.Create ().Close ();
-				}
 			}
 
 			foreach (var f in outdir.GetFiles ("*.jar")

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BindingBuildTest.cs

@@ -280,8 +280,8 @@ public void BindingCheckHiddenFiles ()
 				proj.OtherBuildItems.Add (new BuildItem ("ProjectReference", "..\\Binding\\UnnamedProject.csproj"));
 				using (var b = CreateApkBuilder (Path.Combine ("temp", "BindingCheckHiddenFiles", "App"))) {
 					Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
-					var dsStorePath = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "__library_projects__",
-						"UnnamedProject", "library_project_imports");
+					var dsStorePath = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "lp",
+						"E95D57CD3A", "jl");
 					Assert.IsTrue (Directory.Exists (dsStorePath), "{0} should exist.", dsStorePath);
 					Assert.IsFalse (File.Exists (Path.Combine (dsStorePath, ".DS_Store")), "{0} should NOT exist.",
 						Path.Combine (dsStorePath, ".DS_Store"));

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs

@@ -1740,6 +1740,49 @@ public void BuildInDesignTimeMode ()
 			}
 		}
 
+		[Test]
+		public void CheckLibraryImportsUpgrade ([Values(false, true)] bool useShortFileNames)
+		{
+			var path = Path.Combine ("temp", "CheckLibraryImportsUpgrade");
+			var libproj = new XamarinAndroidLibraryProject () {
+				IsRelease = true,
+				ProjectName = "Library1"
+			};
+			var proj = new XamarinAndroidApplicationProject () {
+				IsRelease = true,
+				ProjectName = "App1",
+			};
+			proj.References.Add (new BuildItem ("ProjectReference", $"..\\Library1\\Library1.csproj"));
+			proj.SetProperty ("_AndroidLibrayProjectIntermediatePath", Path.Combine (proj.IntermediateOutputPath, "__library_projects__"));
+			proj.SetProperty (proj.ActiveConfigurationProperties, "UseShortFileNames", useShortFileNames);
+			using (var libb = CreateDllBuilder (Path.Combine (path, libproj.ProjectName), false, false)) {
+				Assert.IsTrue (libb.Build (libproj), "Build should have succeeded.");
+				using (var builder = CreateApkBuilder (Path.Combine (path, proj.ProjectName), false, false)) {
+					builder.Verbosity = LoggerVerbosity.Diagnostic;
+					Assert.IsTrue (builder.Build (proj), "Build should have succeeded.");
+					Assert.IsTrue (Directory.Exists (Path.Combine (Root, path, proj.ProjectName, proj.IntermediateOutputPath, "__library_projects__")),
+						"The __library_projects__ directory should exist.");
+					proj.RemoveProperty ("_AndroidLibrayProjectIntermediatePath");
+					Assert.IsTrue (builder.Build (proj), "Build should have succeeded.");
+					Assert.IsTrue (Directory.Exists (Path.Combine (Root, path, proj.ProjectName, proj.IntermediateOutputPath, "__library_projects__")),
+						"The __library_projects__ directory should exist.");
+					Assert.IsTrue (libb.Clean (libproj), "Clean should have succeeded.");
+					Assert.IsTrue (libb.Build (libproj), "Build should have succeeded.");
+					Assert.IsTrue (builder.Build (proj), "Build should have succeeded.");
+					if (!useShortFileNames) {
+						Assert.IsTrue (Directory.Exists (Path.Combine (Root, path, proj.ProjectName, proj.IntermediateOutputPath, "__library_projects__")),
+							"The __library_projects__ directory should exist.");
+					} else {
+						Assert.IsFalse (Directory.Exists (Path.Combine (Root, path, proj.ProjectName, proj.IntermediateOutputPath, "__library_projects__")),
+							"The __library_projects__ directory should not exist.");
+						Assert.IsTrue (Directory.Exists (Path.Combine (Root, path, proj.ProjectName, proj.IntermediateOutputPath, "lp")),
+							"The lp directory should exist.");
+					}
+
+				}
+			}
+		}
+
 		[Test]
 		public void BuildAMassiveApp()
 		{

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/IncrementalBuildTest.cs

@@ -171,7 +171,7 @@ public Class2 ()
 						Assert.IsNotNull (libfoo, "libfoo.so should exist in the .apk");
 
 						Assert.AreEqual (so.TextContent ().Length, new FileInfo (Path.Combine (Root, libbuilder.ProjectDirectory, lib.IntermediateOutputPath,
-							"native_library_imports", "armeabi-v7a", "libfoo.so")).Length,
+							"nl", "armeabi-v7a", "libfoo.so")).Length,
 							"intermediate size mismatch");
 						libfoo = ZipHelper.ReadFileFromZip (Path.Combine (Root, builder.ProjectDirectory, app.OutputPath, app.PackageName + ".apk"),
 							"lib/armeabi-v7a/libfoo.so");

src/Xamarin.Android.Build.Tasks/Utilities/Files.cs

@@ -193,7 +193,7 @@ public static ZipArchive ReadZipFile (string filename, bool strictConsistencyChe
 			return ZipArchive.Open (filename, FileMode.Open, strictConsistencyChecks: strictConsistencyChecks);
 		}
 
-		public static void ExtractAll(ZipArchive zip, string destination, Action<int, int> progressCallback = null)
+		public static void ExtractAll(ZipArchive zip, string destination, Action<int, int> progressCallback = null, Func<string, string> modifyCallback = null)
 		{
 			int i = 0;
 			int total = (int)zip.EntryCount;
@@ -202,14 +202,15 @@ public static void ExtractAll(ZipArchive zip, string destination, Action<int, in
 						entry.FullName.EndsWith ("/__MACOSX", StringComparison.OrdinalIgnoreCase) ||
 						entry.FullName.EndsWith ("/.DS_Store", StringComparison.OrdinalIgnoreCase))
 					continue;
+				var fullName = modifyCallback != null ? modifyCallback (entry.FullName) : entry.FullName;
 				if (entry.IsDirectory) {
-					Directory.CreateDirectory (Path.Combine (destination, entry.FullName));
+					Directory.CreateDirectory (Path.Combine (destination, fullName));
 					continue;
 				}
 				if (progressCallback != null)
 					progressCallback (i++, total);
-				Directory.CreateDirectory (Path.Combine (destination, Path.GetDirectoryName (entry.FullName)));
-				entry.Extract (destination, entry.FullName);
+				Directory.CreateDirectory (Path.Combine (destination, Path.GetDirectoryName (fullName)));
+				entry.Extract (destination, fullName);
 			}
 		}