[Xamarin.Android.Build.Tasks] Path is too long

Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=30147

Windows (still) has a max path limit of 260 characters..
This can cause us a problem if the user creates a project in
a directory which already takes up most of that limit.. Like on
their Desktop!

This is because of the intermediate structure that was introduced
to handle embedding resources into the assemblies. The current
intermediate structure is as follows

	`$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\library_project_imports\`
	`$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\native_library_imports\`

Now consider that `$(AssemblyName)` can sometimes end up with something lile
"Xamarin.Android.Support.v7.AppCompat.21.0.3.0" you can easily see how we
start getting into trouble.

There was an attempt to fix this up a while ago (722dcc05) by introducing the
`$(UseShortFileNames)` property. However while this did introduce a
directory structure which was shorter is broke backwards compatability
with older versions of xamarin-android.

This is because of the structore of the zip files we that are embedded
into the assemblies. The current system just extracts the zip into
the `$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\`
directory and assumes that it contains a `library_project_imports` directory.

All the build tasks also assumed that as well.

So the fix needs to be done on a number of fronts. Firstly we need to
update the `$(_LibraryProjectImportsDirectoryName)` and
`$(_NativeLibraryImportsDirectoryName)` to be something shorter.
Next up we need to shorten "__library_project_imports__" to something
else verbose as well. This should cut down on the amount of the MAX_PATH
we chew up.

The real key to this is to NOT change the structure of the zip files in the
assemblies! So when we generate a zip from "jlibs" we make sure that the
folder in the zip is called "library_project_imports". And when we extract
the zip file we makle sure that "library_project_imports" is replaced by
the shorter name.
This will ensure that we are backward compatbile with older versions BUT
more importantly we get to use the shorter directory structure.
The files for native librarys are not extracted to disk but are extracted
from memory so as long as the structure remains the same i.e "native_library_imports"
that code does not need to change.

The other thing we need to do is to update ResolveLibraryProjectImports Task to
upgrade the system when it runs. So if we already have a "libraryprojectimports.cache"
in place we just use that as is. But if we re-run the ResolveLibraryProjectImports
task (due to a change or a clean build) we detect if we have the older structure in place
and just remove it.. Since we are going to regenerate the entire cache again anyway
we might as well start from scratch.

With this in place it becomes possible we can now enable `$(UseShortFileNames)`
by default!

So the new structure that is created is as follows

	`$(IntermediateOutputPath)\lp\<id>\jl`
	`$(IntermediateOutputPath)\lp\<id>\nl`

The <id> will be a single integer value which can be mapped to the source
assembly via a new map.cache file. This .cache file will contain a list of
assemblys. We use the `Index` of the assembly in the list to figure out
which cache directory to use.

	UnnamedProject
	Library1

In this case `Library1` would have an index of `1` and `UnnamedProject` will
be `0`. The old behaviour can be enabled by setting
`$(UseShortFileNames)` to `False`.

Note in order to keep existing bindings generator behaviour consistent,
the BindingsGenerator task will not use the `$(UseShortFileNames)`
property to control how it generates its .cs files. Instead a new
property

	`$(UseShortGeneratorFileNames)`

which can be used to control if the generator produces short names
(e.g 1.cs, 2.cs). This will be `False` by default.

Author: dellis1972

src/Xamarin.Android.Build.Tasks/Tasks/Aapt.cs

@@ -71,12 +71,14 @@ public class Aapt : AsyncTask
 		public string ImportsDirectory { get; set; }
 		public string OutputImportDirectory { get; set; }
 		public bool UseShortFileNames { get; set; }
+		public string AssemblyIdentityMapFile { get; set; }
 
 		public string ResourceNameCaseMap { get; set; }
 
 		public bool ExplicitCrunch { get; set; }
 
 		Dictionary<string,string> resource_name_case_map = new Dictionary<string,string> ();
+		AssemblyIdentityMap assemblyMap = new AssemblyIdentityMap ();
 
 		bool ManifestIsUpToDate (string manifestFile)
 		{
@@ -197,6 +199,8 @@ public override bool Execute ()
 				foreach (var arr in ResourceNameCaseMap.Split (';').Select (l => l.Split ('|')).Where (a => a.Length == 2))
 					resource_name_case_map [arr [1]] = arr [0]; // lowercase -> original
 
+			assemblyMap.Load (AssemblyIdentityMapFile);
+
 			ThreadingTasks.Parallel.ForEach (ManifestFiles, () => 0, DoExecute, (obj) => { Complete (); });
 
 			base.Execute ();
@@ -315,7 +319,7 @@ string ExpandString (string s)
 					return s.Substring (0, st + 1) + ExpandString (s.Substring (st + 1));
 				int ast = st + "${library.imports:".Length;
 				string aname = s.Substring (ast, ed - ast);
-				return s.Substring (0, st) + Path.Combine (OutputImportDirectory, UseShortFileNames ? MonoAndroidHelper.GetLibraryImportDirectoryNameForAssembly (aname) : aname, ImportsDirectory) + Path.DirectorySeparatorChar + ExpandString (s.Substring (ed + 1));
+				return s.Substring (0, st) + Path.Combine (OutputImportDirectory, UseShortFileNames ? assemblyMap.GetLibraryImportDirectoryNameForAssembly (aname) : aname, ImportsDirectory) + Path.DirectorySeparatorChar + ExpandString (s.Substring (ed + 1));
 			}
 			else
 				return s;

src/Xamarin.Android.Build.Tasks/Tasks/CreateLibraryResourceArchive.cs

@@ -94,7 +94,7 @@ public override bool Execute ()
 
 			if (Files.ArchiveZip (outpath, f => {
 				using (var zip = new ZipArchiveEx (f)) {
-					zip.AddDirectory (OutputDirectory, outDirInfo.Name);
+					zip.AddDirectory (OutputDirectory, "library_project_imports");
 				}
 			})) {
 				Log.LogDebugMessage ("Saving contents to " + outpath);

src/Xamarin.Android.Build.Tasks/Tasks/CreateManagedLibraryResourceArchive.cs

@@ -108,7 +108,7 @@ public override bool Execute ()
 
 			if (Files.ArchiveZip (outpath, f => {
 				using (var zip = new ZipArchiveEx (f)) {
-					zip.AddDirectory (OutputDirectory, outDirInfo.Name);
+					zip.AddDirectory (OutputDirectory, "library_project_imports");
 				}
 			})) {
 				Log.LogDebugMessage ("Saving contents to " + outpath);

src/Xamarin.Android.Build.Tasks/Tasks/CreateNativeLibraryArchive.cs

@@ -66,7 +66,7 @@ public override bool Execute ()
 
 			if (Files.ArchiveZip (outpath, f => {
 				using (var zip = new ZipArchiveEx (f)) {
-					zip.AddDirectory (OutputDirectory, outDirInfo.Name);
+					zip.AddDirectory (OutputDirectory, "native_library_imports");
 				}
 			})) {
 				Log.LogDebugMessage ("Saving contents to " + outpath);

src/Xamarin.Android.Build.Tasks/Tasks/ResolveLibraryProjectImports.cs

@@ -34,6 +34,9 @@ public class ResolveLibraryProjectImports : Task
 		[Required]
 		public bool UseShortFileNames { get; set; }
 
+		[Required]
+		public string AssemblyIdentityMapFile { get; set; }
+
 		public string CacheFile { get; set;} 
 
 		[Output]
@@ -51,13 +54,14 @@ public class ResolveLibraryProjectImports : Task
 		[Output]
 		public ITaskItem [] ResolvedResourceDirectoryStamps { get; set; }
 
-		string        imports_dir = "library_project_imports";
+		AssemblyIdentityMap assemblyMap = new AssemblyIdentityMap();
 
 		public ResolveLibraryProjectImports ()
 		{
 		}
 
 		// Extracts library project contents under e.g. obj/Debug/[__library_projects__/*.jar | res/*/*]
+		// Extracts library project contents under e.g. obj/Debug/[lp/*.jar | res/*/*]
 		public override bool Execute ()
 		{
 			Log.LogDebugMessage ("ResolveLibraryProjectImports Task");
@@ -72,6 +76,8 @@ public override bool Execute ()
 			var resolvedAssetDirectories      = new List<string> ();
 			var resolvedEnvironmentFiles      = new List<string> ();
 
+			assemblyMap.Load (AssemblyIdentityMapFile);
+
 			using (var resolver = new DirectoryAssemblyResolver (Log.LogWarning, loadDebugSymbols: false)) {
 				Extract (resolver, jars, resolvedResourceDirectories, resolvedAssetDirectories, resolvedEnvironmentFiles);
 			}
@@ -112,6 +118,8 @@ public override bool Execute ()
 				document.Save (CacheFile);
 			}
 
+			assemblyMap.Save (AssemblyIdentityMapFile);
+
 			Log.LogDebugTaskItems ("  Jars: ", Jars.Select (s => new TaskItem (s)).ToArray ());
 			Log.LogDebugTaskItems ("  ResolvedResourceDirectories: ", ResolvedResourceDirectories.Select (s => new TaskItem (s)).ToArray ());
 			Log.LogDebugTaskItems ("  ResolvedAssetDirectories: ", ResolvedAssetDirectories.Select (s => new TaskItem (s)).ToArray ());
@@ -135,13 +143,20 @@ static string GetTargetAssembly (ITaskItem assemblyName)
 		}
 
 		// Extracts library project contents under e.g. obj/Debug/[__library_projects__/*.jar | res/*/*]
+		// Extracts library project contents under e.g. obj/Debug/[lp/*.jar | res/*/*]
 		void Extract (
 				DirectoryAssemblyResolver res,
 				ICollection<string> jars,
 				ICollection<string> resolvedResourceDirectories,
 				ICollection<string> resolvedAssetDirectories,
 				ICollection<string> resolvedEnvironments)
 		{
+			// lets "upgrade" the old directory.
+			string oldPath = Path.GetFullPath (Path.Combine (OutputImportDirectory, "..", "__library_projects__"));
+			if (!OutputImportDirectory.Contains ("__library_projects__") && Directory.Exists (oldPath)) {
+				MonoAndroidHelper.SetDirectoryWriteable (Path.Combine (oldPath, ".."));
+				Directory.Delete (oldPath, recursive: true);
+			}
 			var outdir = new DirectoryInfo (OutputImportDirectory);
 			if (!outdir.Exists)
 				outdir.Create ();
@@ -154,26 +169,95 @@ void Extract (
 					.Select (a => GetTargetAssembly (a))
 					.Where (a => a != null)
 					.Distinct ()) {
-				foreach (var imp in new string [] {imports_dir, "library_project_imports"}.Distinct ()) {
-					string assemblyIdentName = Path.GetFileNameWithoutExtension (assemblyPath);
-					if (UseShortFileNames) {
-						assemblyIdentName = Xamarin.Android.Tasks.MonoAndroidHelper.GetLibraryImportDirectoryNameForAssembly (assemblyIdentName);
-					}
-					string outDirForDll = Path.Combine (OutputImportDirectory, assemblyIdentName);
-					string importsDir = Path.Combine (outDirForDll, imp);
+				string assemblyIdentName = Path.GetFileNameWithoutExtension (assemblyPath);
+				if (UseShortFileNames) {
+					assemblyIdentName = assemblyMap.GetLibraryImportDirectoryNameForAssembly (assemblyIdentName);
+				}
+				string outDirForDll = Path.Combine (OutputImportDirectory, assemblyIdentName);
+				string importsDir = Path.Combine (outDirForDll, ImportsDirectory);
+#if SEPARATE_CRUNCH
+				// FIXME: review these binResDir thing and enable this. Eclipse does this.
+				// Enabling these blindly causes build failure on ActionBarSherlock.
+				//string binResDir = Path.Combine (importsDir, "bin", "res");
+				//string binAssemblyDir = Path.Combine (importsDir, "bin", "assets");
+#endif
+				string resDir = Path.Combine (importsDir, "res");
+				string assemblyDir = Path.Combine (importsDir, "assets");
+
+				// Skip already-extracted resources.
+				var stamp = new FileInfo (Path.Combine (outdir.FullName, assemblyIdentName + ".stamp"));
+				if (stamp.Exists && stamp.LastWriteTime > new FileInfo (assemblyPath).LastWriteTime) {
+					Log.LogDebugMessage ("Skipped resource lookup for {0}: extracted files are up to date", assemblyPath);
 #if SEPARATE_CRUNCH
-					// FIXME: review these binResDir thing and enable this. Eclipse does this.
+					// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
 					// Enabling these blindly causes build failure on ActionBarSherlock.
-					//string binResDir = Path.Combine (importsDir, "bin", "res");
-					//string binAssemblyDir = Path.Combine (importsDir, "bin", "assets");
+					if (Directory.Exists (binResDir))
+						resolvedResourceDirectories.Add (binResDir);
+					if (Directory.Exists (binAssemblyDir))
+						resolvedAssetDirectories.Add (binAssemblyDir);
 #endif
-					string resDir = Path.Combine (importsDir, "res");
-					string assemblyDir = Path.Combine (importsDir, "assets");
+					if (Directory.Exists (resDir))
+						resolvedResourceDirectories.Add (resDir);
+					if (Directory.Exists (assemblyDir))
+						resolvedAssetDirectories.Add (assemblyDir);
+					continue;
+				}
+
+				if (Directory.Exists (outDirForDll))
+					Directory.Delete (outDirForDll, true);
+
+				var assembly = res.GetAssembly (assemblyPath);
+
+				foreach (var mod in assembly.Modules) {
+					// android environment files
+					foreach (var envtxt in mod.Resources
+							.Where (r => r.Name.StartsWith ("__AndroidEnvironment__", StringComparison.OrdinalIgnoreCase))
+							.Where (r => r is EmbeddedResource)
+							.Cast<EmbeddedResource> ()) {
+						if (!Directory.Exists (outDirForDll))
+							Directory.CreateDirectory (outDirForDll);
+						var finfo = new FileInfo (Path.Combine (outDirForDll, envtxt.Name));
+						using (var fs = finfo.Create ()) {
+							var data = envtxt.GetResourceData ();
+							fs.Write (data, 0, data.Length);
+						}
+						resolvedEnvironments.Add (finfo.FullName);
+					}
+
+					// embedded jars (EmbeddedJar, EmbeddedReferenceJar)
+					var resjars = mod.Resources
+						.Where (r => r.Name.EndsWith (".jar", StringComparison.InvariantCultureIgnoreCase))
+						.Select (r => (EmbeddedResource) r);
+					foreach (var resjar in resjars) {
+						var data = resjar.GetResourceData ();
+						if (!Directory.Exists (importsDir))
+							Directory.CreateDirectory (importsDir);
+						using (var outfs = File.Create (Path.Combine (importsDir, resjar.Name)))
+							outfs.Write (data, 0, data.Length);
+					}
 
-					// Skip already-extracted resources.
-					var stamp = new FileInfo (Path.Combine (outdir.FullName, assemblyIdentName + ".stamp"));
-					if (stamp.Exists && stamp.LastWriteTime > new FileInfo (assemblyPath).LastWriteTime) {
-						Log.LogDebugMessage ("Skipped resource lookup for {0}: extracted files are up to date", assemblyPath);
+					// embedded AndroidResourceLibrary archive
+					var reszip = mod.Resources.FirstOrDefault (r => r.Name == "__AndroidLibraryProjects__.zip") as EmbeddedResource;
+					if (reszip != null) {
+						if (!Directory.Exists (outDirForDll))
+							Directory.CreateDirectory (outDirForDll);
+						var finfo = new FileInfo (Path.Combine (outDirForDll, reszip.Name));
+						using (var fs = finfo.Create ()) {
+							var data = reszip.GetResourceData ();
+							fs.Write (data, 0, data.Length);
+						}
+
+						// temporarily extracted directory will look like:
+						//    __library_projects__/[dllname]/[library_project_imports | jlibs]/bin
+						using (var zip = MonoAndroidHelper.ReadZipFile (finfo.FullName)) {
+							Files.ExtractAll (zip, outDirForDll, modifyCallback: (entryFullName) => {
+								return entryFullName.Replace ("library_project_imports", ImportsDirectory);
+							});
+						}
+
+						// We used to *copy* the resources to overwrite other resources,
+						// which resulted in missing resource issue.
+						// Here we replaced copy with use of '-S' option and made it to work.
 #if SEPARATE_CRUNCH
 						// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
 						// Enabling these blindly causes build failure on ActionBarSherlock.
@@ -186,80 +270,13 @@ void Extract (
 							resolvedResourceDirectories.Add (resDir);
 						if (Directory.Exists (assemblyDir))
 							resolvedAssetDirectories.Add (assemblyDir);
-						continue;
-					}
-
-					if (Directory.Exists (outDirForDll))
-						Directory.Delete (outDirForDll, true);
-
-					Directory.CreateDirectory (importsDir);
-
-					var assembly = res.GetAssembly (assemblyPath);
-
-					foreach (var mod in assembly.Modules) {
-						// android environment files
-						foreach (var envtxt in mod.Resources
-								.Where (r => r.Name.StartsWith ("__AndroidEnvironment__", StringComparison.OrdinalIgnoreCase))
-								.Where (r => r is EmbeddedResource)
-								.Cast<EmbeddedResource> ()) {
-							if (!Directory.Exists (outDirForDll))
-								Directory.CreateDirectory (outDirForDll);
-							var finfo = new FileInfo (Path.Combine (outDirForDll, envtxt.Name));
-							using (var fs = finfo.Create ()) {
-								var data = envtxt.GetResourceData ();
-								fs.Write (data, 0, data.Length);
-							}
-							resolvedEnvironments.Add (finfo.FullName);
-						}
 
-						// embedded jars (EmbeddedJar, EmbeddedReferenceJar)
-						var resjars = mod.Resources
-							.Where (r => r.Name.EndsWith (".jar", StringComparison.InvariantCultureIgnoreCase))
-							.Select (r => (EmbeddedResource) r);
-						foreach (var resjar in resjars) {
-							var data = resjar.GetResourceData ();
-							using (var outfs = File.Create (Path.Combine (importsDir, resjar.Name)))
-								outfs.Write (data, 0, data.Length);
-						}
-
-						// embedded AndroidResourceLibrary archive
-						var reszip = mod.Resources.FirstOrDefault (r => r.Name == "__AndroidLibraryProjects__.zip") as EmbeddedResource;
-						if (reszip != null) {
-							if (!Directory.Exists (outDirForDll))
-								Directory.CreateDirectory (outDirForDll);
-							var finfo = new FileInfo (Path.Combine (outDirForDll, reszip.Name));
-							using (var fs = finfo.Create ()) {
-								var data = reszip.GetResourceData ();
-								fs.Write (data, 0, data.Length);
-							}
-
-							// temporarily extracted directory will look like:
-							//    __library_projects__/[dllname]/[library_project_imports | jlibs]/bin
-							using (var zip = MonoAndroidHelper.ReadZipFile (finfo.FullName))
-								Files.ExtractAll (zip, outDirForDll);
-
-							// We used to *copy* the resources to overwrite other resources,
-							// which resulted in missing resource issue.
-							// Here we replaced copy with use of '-S' option and made it to work.
-#if SEPARATE_CRUNCH
-							// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
-							// Enabling these blindly causes build failure on ActionBarSherlock.
-							if (Directory.Exists (binResDir))
-								resolvedResourceDirectories.Add (binResDir);
-							if (Directory.Exists (binAssemblyDir))
-								resolvedAssetDirectories.Add (binAssemblyDir);
-#endif
-							if (Directory.Exists (resDir))
-								resolvedResourceDirectories.Add (resDir);
-							if (Directory.Exists (assemblyDir))
-								resolvedAssetDirectories.Add (assemblyDir);
-
-							finfo.Delete ();
-						}
+						finfo.Delete ();
 					}
+				}
 
+				if (Directory.Exists (importsDir))
 					stamp.Create ().Close ();
-				}
 			}
 
 			foreach (var f in outdir.GetFiles ("*.jar")

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BindingBuildTest.cs

@@ -3,6 +3,7 @@
 using NUnit.Framework;
 using System.IO;
 using System.Linq;
+using System.Collections.Generic;
 
 namespace Xamarin.Android.Build.Tests
 {
@@ -263,7 +264,7 @@ public void BindngFilterUnsupportedNativeAbiLibraries ()
 		}
 
 		[Test]
-		public void BindingCheckHiddenFiles ()
+		public void BindingCheckHiddenFiles ([Values (true, false)] bool useShortFileNames)
 		{
 			var binding = new XamarinAndroidBindingProject () {
 				UseLatestPlatformSdk = true,
@@ -273,15 +274,31 @@ public void BindingCheckHiddenFiles ()
 			binding.Jars.Add (new AndroidItem.LibraryProjectZip ("Jars\\mylibrary.aar") {
 				WebContent = "https://www.dropbox.com/s/astiqp8jo97x91h/mylibrary.aar?dl=1"
 			});
+			binding.SetProperty (binding.ActiveConfigurationProperties, "UseShortFileNames", useShortFileNames);
 			using (var bindingBuilder = CreateDllBuilder (Path.Combine ("temp", "BindingCheckHiddenFiles", "Binding"))) {
 				bindingBuilder.Verbosity = Microsoft.Build.Framework.LoggerVerbosity.Diagnostic;
 				Assert.IsTrue (bindingBuilder.Build (binding), "binding build should have succeeded");
 				var proj = new XamarinAndroidApplicationProject ();
 				proj.OtherBuildItems.Add (new BuildItem ("ProjectReference", "..\\Binding\\UnnamedProject.csproj"));
+				proj.SetProperty (proj.ActiveConfigurationProperties, "UseShortFileNames", useShortFileNames);
 				using (var b = CreateApkBuilder (Path.Combine ("temp", "BindingCheckHiddenFiles", "App"))) {
 					Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
-					var dsStorePath = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "__library_projects__",
-						"UnnamedProject", "library_project_imports");
+					var assemblyMap = b.Output.GetIntermediaryPath (Path.Combine ("lp", "map.cache"));
+					if (useShortFileNames)
+						Assert.IsTrue (File.Exists (assemblyMap), $"{assemblyMap} should exist.");
+					else
+						Assert.IsFalse (File.Exists (assemblyMap), $"{assemblyMap} should not exist.");
+					var assemblyIdentityMap = new List<string> ();
+					if (useShortFileNames) {
+						foreach (var s in File.ReadLines (assemblyMap)) {
+							assemblyIdentityMap.Add (s);
+						}
+					}
+					var assmeblyIdentity = useShortFileNames ? assemblyIdentityMap.IndexOf ("UnnamedProject").ToString () : "UnnamedProject";
+					var libaryImportsFolder = useShortFileNames ? "lp" : "__library_projects__";
+					var jlibs = useShortFileNames ? "jl" : "library_project_imports";
+					var dsStorePath = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, libaryImportsFolder,
+						assmeblyIdentity, jlibs);
 					Assert.IsTrue (Directory.Exists (dsStorePath), "{0} should exist.", dsStorePath);
 					Assert.IsFalse (File.Exists (Path.Combine (dsStorePath, ".DS_Store")), "{0} should NOT exist.",
 						Path.Combine (dsStorePath, ".DS_Store"));