[Xamarin.Android.Build.Tasks] support Embedded DSOs (#2579)

Fixes: #2415

Context: #2154

Currently, in order to activate the embedded Dynamic Shared Object
(DSO) support, one has to perform the following actions manually
(see also 95ca102):

 1. Add the `android:extractNativeLibs="false"` attribute to the
    `<application>` element within `Properties/AndroidManifest.xml`.

 2. Add the following property to the project file:

        <AndroidStoreUncompressedFileExtensions>.so</AndroidStoreUncompressedFileExtensions>

 3. Add an android environment file to the project with a line
    containing:

        __XA_DSO_IN_APK=1

Instead of requiring these three separate steps, the presence of the
`//application[@android:extractNativeLibs='false']` attribute within
`AndroidManifest.xml` should instead be used as the "feature" toggle;
when `extractNativeLibs` is false, then
`$(AndroidStoreUncompressedFileExtensions)` should be automatically
updated and the `__XA_DSO_IN_APK` environment variable should
automatically exported:

 1. Append the value `.so` to
    `$(AndroidStoreUncompressedFileExtensions)`.
    Both of the `<Aapt/>` and `<BuildApk/>` MSBuild tasks use
    this property.

 2. Create a new file within `$(IntermediateOutputPath)` containing
    the string `__XA_DSO_IN_APK=1`, and add this new file to
    `@(AndroidEnvironment)` so that `__XA_DSO_IN_APK` is set.

The problem here is with incremental builds; if the
`_GenerateJavaStubs` target is skipped, we still need to know if
`//application/@android:extractNativeLibs` is set.  To make this
work, I added a `CacheFile` property on the `<GenerateJavaStubs/>`
MSBuild task.  It writes a simple XML document to
`$(IntermediateOutputPath)javastubs.cache`, such as:

	<Properties>
	  <EmbeddedDSOsEnabled>True</EmbeddedDSOsEnabled>
	</Properties>

The file will not exist at all unless the value is `True`, which will
prevent this feature from impacting build times.

I also added an MSBuild test to verify these changes are happening.
I also updated `tests/EmbeddedDSOs` to rely on the new functionality.

Finally, I updated the `_GenerateJavaStubs` target to follow the
"stamp" file convention of using
`$(_AndroidStampDirectory)_GenerateJavaStubs.stamp`.

Author: jonathanpeppers

Documentation/guides/extract-native-libraries.md

@@ -0,0 +1,49 @@
+# Installation of Native Libraries
+
+An Android `.apk` file can contain native libraries for one or more
+architectures.  Historically, all native libraries for the target
+device are extracted at `.apk` installation time.  This would result
+in *two* copies of native libraries on the target device: a set of
+native libraries which are stored compressed within the `.apk`, and
+a separate set of native libraries on the Android target filesystem.
+
+Starting with Android v6.0 (API-23), Android added an ability for
+native libraries to be stored *uncompressed* within the `.apk` along
+with the ability to load those native libraries *from* the `.apk`
+without requiring a separate filesystem copy of the native libraries.
+
+Android Things *requires* this new mechanism; `.apk` files installed
+on Android Things will no longer have *any* native libraries extracted.
+
+As a result, the `.apk` will be *larger*, because the native
+libraries are stored uncompressed within the `.apk`, but the
+install size will be *smaller*, because there isn't a second "copy"
+of the native libraries (one compressed in the `.apk`, one outside
+of the `.apk`).
+
+On Android versions older than Android v6.0, the native libraries
+will continue to be extracted during `.apk` installation.
+
+In order to indicate to Android v6.0 and later that native libraries
+do not need to be extracted, the
+[`//application/@android:extractNativeLibs`][extractNativeLibs]
+attribute within `AndroidManifest.xml` must be set to `false.`
+
+[extractNativeLibs]: https://developer.android.com/guide/topics/manifest/application-element#extractNativeLibs
+
+## OSS Implementation Details
+
+When `AndroidManifest.xml` contains an XML attribute matching
+`//application[@android:extractNativeLibs='false']`, the
+Xamarin.Android build system will do the following:
+
+ 1. The `$(AndroidStoreUncompressedFileExtensions)` MSBuild property
+    will be automatically updated to contain the `.so` file
+    extension, causing native libraries to be stored uncompressed
+    within the `.apk`.
+
+ 2. The `__XA_DSO_IN_APK` environment variable will be set within the
+    created `.apk` file with the value of `1`, indicating to
+    the app that native libraries should be loaded from the `.apk`
+    itself instead of from the filesystem.
+

src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs

@@ -2,9 +2,9 @@
 
 using System;
 using System.Collections.Generic;
-using System.Diagnostics;
 using System.IO;
 using System.Linq;
+using System.Xml.Linq;
 using Microsoft.Build.Framework;
 using Microsoft.Build.Utilities;
 using MonoDroid.Utils;
@@ -68,6 +68,11 @@ public class GenerateJavaStubs : Task
 
 		public string ApplicationJavaClass { get; set; }
 
+		/// <summary>
+		/// If specified, we need to cache the value of EmbeddedDSOsEnabled=True for incremental builds
+		/// </summary>
+		public string CacheFile { get; set; }
+
 		public override bool Execute ()
 		{
 			try {
@@ -223,6 +228,22 @@ void Run (DirectoryAssemblyResolver res)
 				MonoAndroidHelper.CopyIfStreamChanged (stream, MergedAndroidManifestOutput);
 			}
 
+			// Create the CacheFile if needed
+			if (!string.IsNullOrEmpty (CacheFile)) {
+				bool extractNativeLibraries = manifest.ExtractNativeLibraries ();
+				if (!extractNativeLibraries) {
+					//We need to write the value to a file, if _GenerateJavaStubs is skipped on incremental builds
+					var document = new XDocument (
+						new XDeclaration ("1.0", "UTF-8", null),
+						new XElement ("Properties", new XElement (nameof (ReadJavaStubsCache.EmbeddedDSOsEnabled), "True"))
+					);
+					document.SaveIfChanged (CacheFile);
+				} else {
+					//Delete the file otherwise, since we only need to specify when EmbeddedDSOsEnabled=True
+					File.Delete (CacheFile);
+				}
+			}
+
 			// Create additional runtime provider java sources.
 			string providerTemplateFile = UseSharedRuntime ? "MonoRuntimeProvider.Shared.java" : "MonoRuntimeProvider.Bundled.java";
 			string providerTemplate = GetResource<JavaCallableWrapperGenerator> (providerTemplateFile);

src/Xamarin.Android.Build.Tasks/Tasks/ReadJavaStubsCache.cs

@@ -0,0 +1,28 @@
+using Microsoft.Build.Framework;
+using Microsoft.Build.Utilities;
+using System.IO;
+using System.Xml.Linq;
+
+namespace Xamarin.Android.Tasks
+{
+	public class ReadJavaStubsCache : Task
+	{
+		[Required]
+		public string CacheFile { get; set; }
+
+		[Output]
+		public bool EmbeddedDSOsEnabled { get; set; }
+
+		public override bool Execute ()
+		{
+			if (File.Exists (CacheFile)) {
+				var doc = XDocument.Load (CacheFile);
+				string text = doc.Element ("Properties")?.Element (nameof (EmbeddedDSOsEnabled))?.Value;
+				if (bool.TryParse (text, out bool value))
+					EmbeddedDSOsEnabled = value;
+			}
+
+			return !Log.HasLoggedErrors;
+		}
+	}
+}

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs

@@ -202,6 +202,53 @@ public void CheckIncludedNativeLibraries ([Values (true, false)] bool compressNa
 			}
 		}
 
+		[Test]
+		public void EmbeddedDSOs ()
+		{
+			var proj = new XamarinAndroidApplicationProject ();
+			proj.AndroidManifest = $@"<?xml version=""1.0"" encoding=""utf-8""?>
+<manifest xmlns:android=""http://schemas.android.com/apk/res/android"" android:versionCode=""1"" android:versionName=""1.0"" package=""{proj.PackageName}"">
+	<uses-sdk />
+	<application android:label=""{proj.ProjectName}"" android:extractNativeLibs=""false"">
+	</application>
+</manifest>";
+
+			using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) {
+				Assert.IsTrue (b.Build (proj), "first build should have succeeded");
+
+				var apk = Path.Combine (Root, b.ProjectDirectory,
+						proj.IntermediateOutputPath, "android", "bin", "UnnamedProject.UnnamedProject.apk");
+				AssertEmbeddedDSOs (apk);
+
+				//Delete the apk & build again
+				File.Delete (apk);
+				Assert.IsTrue (b.Build (proj), "second build should have succeeded");
+				AssertEmbeddedDSOs (apk);
+			}
+		}
+
+		void AssertEmbeddedDSOs (string apk)
+		{
+			FileAssert.Exists (apk);
+
+			using (var zip = ZipHelper.OpenZip (apk)) {
+				foreach (var entry in zip) {
+					if (entry.FullName.EndsWith (".so")) {
+						Assert.AreEqual (entry.Size, entry.CompressedSize, $"`{entry.FullName}` should be uncompressed!");
+					} else if (entry.FullName == "environment") {
+						using (var stream = new MemoryStream ()) {
+							entry.Extract (stream);
+							stream.Position = 0;
+							using (var reader = new StreamReader (stream)) {
+								string environment = reader.ReadToEnd ();
+								StringAssert.Contains ("__XA_DSO_IN_APK=1", environment, "`__XA_DSO_IN_APK=1` should be set via @(AndroidEnvironment)");
+							}
+						}
+					}
+				}
+			}
+		}
+
 		[Test]
 		public void ExplicitPackageNamingPolicy ()
 		{

src/Xamarin.Android.Build.Tasks/Utilities/ManifestDocument.cs

@@ -644,6 +644,20 @@ bool IsMainLauncher (XElement intentFilter)
 					intentFilter.Elements (entry.Key).Any (e => ((string) e.Attribute (attName) == entry.Value)));
 		}
 
+		/// <summary>
+		/// Returns the value of //application/@android:extractNativeLibs.
+		/// </summary>
+		public bool ExtractNativeLibraries ()
+		{
+			string text = app?.Attribute (androidNs + "extractNativeLibs")?.Value;
+			if (bool.TryParse (text, out bool value)) {
+				return value;
+			}
+
+			// If android:extractNativeLibs is omitted, returns true.
+			return true;
+		}
+
 		XElement ActivityFromTypeDefinition (TypeDefinition type, string name, int targetSdkVersion)
 		{
 			if (name.StartsWith ("_"))

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj

@@ -122,6 +122,7 @@
     <Compile Include="Tasks\JarToXml.cs" />
     <Compile Include="Tasks\GetJavaPlatformJar.cs" />
     <Compile Include="Tasks\GetFilesThatExist.cs" />
+    <Compile Include="Tasks\ReadJavaStubsCache.cs" />
     <Compile Include="Tasks\RemoveRegisterAttribute.cs" />
     <Compile Include="Tasks\GetMonoPlatformJar.cs" />
     <Compile Include="Tasks\GeneratePackageManagerJava.cs" />

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

@@ -91,6 +91,7 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
 <UsingTask TaskName="Xamarin.Android.Tasks.GetExtraPackages" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="Xamarin.Android.Tasks.CopyGeneratedJavaResourceClasses" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="Xamarin.Android.Tasks.ResolveLibraryProjectImports" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
+<UsingTask TaskName="Xamarin.Android.Tasks.ReadJavaStubsCache" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="Xamarin.Android.Tasks.ReadLibraryProjectImportsCache" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="Xamarin.Android.Tasks.ScanAssemblies" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="Xamarin.Android.Tasks.CheckProjectItems" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
@@ -2284,7 +2285,7 @@ because xbuild doesn't support framework reference assemblies.
 <Target Name="_GenerateJavaStubs"
   DependsOnTargets="_SetLatestTargetFrameworkVersion;_PrepareAssemblies;$(_AfterPrepareAssemblies)"
   Inputs="$(MSBuildAllProjects);@(_ResolvedAssemblies);$(_AndroidManifestAbs);$(_AndroidBuildPropertiesCache);@(_AndroidResourceDest)"
-  Outputs="$(IntermediateOutputPath)_javastubs.stamp">
+  Outputs="$(_AndroidStampDirectory)_GenerateJavaStubs.stamp">
   <GenerateJavaStubs
     ResolvedAssemblies="@(_ResolvedAssemblies)"
     ResolvedUserAssemblies="@(_ResolvedUserAssemblies)"
@@ -2308,7 +2309,8 @@ because xbuild doesn't support framework reference assemblies.
 	PackageNamingPolicy="$(AndroidPackageNamingPolicy)"
 	ApplicationJavaClass="$(AndroidApplicationJavaClass)"
 	FrameworkDirectories="$(_XATargetFrameworkDirectories);$(_XATargetFrameworkDirectories)Facades"
-	AcwMapFile="$(_AcwMapFile)">
+	AcwMapFile="$(_AcwMapFile)"
+	CacheFile="$(IntermediateOutputPath)javastubs.cache">
   </GenerateJavaStubs>
   <ConvertCustomView
 	Condition="Exists('$(_CustomViewMapFile)')"
@@ -2317,13 +2319,45 @@ because xbuild doesn't support framework reference assemblies.
 	ResourceDirectories="$(MonoAndroidResDirIntermediate);@(LibraryResourceDirectories)"
 	ResourceNameCaseMap="$(_AndroidResourceNameCaseMap)"
   />
-  <Touch Files="$(IntermediateOutputPath)_javastubs.stamp" AlwaysCreate="True" />
+  <Touch Files="$(_AndroidStampDirectory)_GenerateJavaStubs.stamp" AlwaysCreate="True" />
+  <ItemGroup>
+    <FileWrites Include="$(IntermediateOutputPath)javastubs.cache" Condition="Exists('$(IntermediateOutputPath)javastubs.cache')" />
+  </ItemGroup>
+</Target>
+
+<Target Name="_ReadJavaStubsCache">
+  <ReadJavaStubsCache CacheFile="$(IntermediateOutputPath)javastubs.cache">
+    <Output TaskParameter="EmbeddedDSOsEnabled" PropertyName="_EmbeddedDSOsEnabled" />
+  </ReadJavaStubsCache>
+</Target>
+
+<Target Name="_SetupEmbeddedDSOs"
+    Condition=" '$(_EmbeddedDSOsEnabled)' == 'True' "
+    Inputs="$(IntermediateOutputPath)javastubs.cache"
+    Outputs="$(IntermediateOutputPath)dsoenvironment.txt">
+  <WriteLinesToFile
+      File="$(IntermediateOutputPath)dsoenvironment.txt"
+      Lines="__XA_DSO_IN_APK=1"
+      Overwrite="True"
+  />
+  <PropertyGroup>
+    <AndroidStoreUncompressedFileExtensions>.so;$(AndroidStoreUncompressedFileExtensions)</AndroidStoreUncompressedFileExtensions>
+  </PropertyGroup>
   <ItemGroup>
-	<FileWrites Include="$(IntermediateOutputPath)_javastubs.stamp" />
+    <AndroidEnvironment Include="$(IntermediateOutputPath)dsoenvironment.txt" />
+    <FileWrites         Include="$(IntermediateOutputPath)dsoenvironment.txt" />
   </ItemGroup>
 </Target>
 
-<Target Name="_GetAddOnPlatformLibraries" DependsOnTargets="_GenerateJavaStubs">
+<PropertyGroup>
+  <_GetAddOnPlatformLibrariesDependsOn>
+    _GenerateJavaStubs;
+    _ReadJavaStubsCache;
+    _SetupEmbeddedDSOs;
+  </_GetAddOnPlatformLibrariesDependsOn>
+</PropertyGroup>
+
+<Target Name="_GetAddOnPlatformLibraries" DependsOnTargets="$(_GetAddOnPlatformLibrariesDependsOn)">
   <GetAddOnPlatformLibraries
 	AndroidSdkPlatform="$(_AndroidApiLevel)"
 	AndroidSdkDir="$(_AndroidSdkDirectory)"
@@ -3195,7 +3229,6 @@ because xbuild doesn't support framework reference assemblies.
 	<Delete Files="$(MonoAndroidIntermediate)__AndroidNativeLibraries__.zip" />
 	<Delete Files="$(MonoAndroidIntermediate)stub_application_data.txt" />
 	<Delete Files="$(IntermediateOutputPath)_javac.stamp" />
-	<Delete Files="$(IntermediateOutputPath)_javastubs.stamp" />
 	<Delete Files="$(_AndroidResFlagFile)" />
 	<Delete Files="$(_AndroidLinkFlag)" />
 	<Delete Files="$(_AndroidComponentResgenFlagFile)" />

tests/CodeGen-Binding/Xamarin.Android.McwGen-Tests/Jars/.gitignore

@@ -0,0 +1 @@
+*.jar

tests/EmbeddedDSOs/EmbeddedDSO/EmbeddedDSO.csproj

@@ -18,11 +18,6 @@
     <AndroidUseLatestPlatformSdk>true</AndroidUseLatestPlatformSdk>
     <AndroidUseSharedRuntime>false</AndroidUseSharedRuntime>
     <AndroidSupportedAbis>armeabi-v7a;x86</AndroidSupportedAbis>
-
-    <!-- This should be removed once XA build tasks are updated to support embedded DSOs.
-	 The same applies to the `__XA_DSO_IN_APK=1` line in Environment.txt
-    -->
-    <AndroidStoreUncompressedFileExtensions>.so</AndroidStoreUncompressedFileExtensions>
   </PropertyGroup>
 
   <PropertyGroup Condition=" '$(UnitTestsMode)' == 'true' ">

tests/EmbeddedDSOs/EmbeddedDSO/Environment.txt

@@ -2,4 +2,3 @@ debug.mono.debug=1
 MONO_LOG_LEVEL=debug
 MONO_LOG_MASK=asm
 MONO_XDEBUG=1
-__XA_DSO_IN_APK=1