From d02910f15f9230d2b99d95ed66ee817d556dd732 Mon Sep 17 00:00:00 2001 From: Jonathan Peppers Date: Wed, 2 Jan 2019 13:31:44 -0600 Subject: [PATCH 1/3] [Xamarin.Android.Build.Tasks] support Embedded DSOs Context: https://github.com/xamarin/xamarin-android/pull/2154 Fixes: https://github.com/xamarin/xamarin-android/issues/2415 Currently, in order to activate the embedded DSO support, one has to perform the following actions manually: 1. Add the `android:extractNativeLibs="false"` attribute to the `` element in the `Properties/AndroidManifest.xml` file. 2. Add the following property to the project file: .so 3. Add an android environment file to the project with a line which says: __XA_DSO_IN_APK=1 ~~ Changes ~~ The presence of `android:extractNativeLibs="false"` in `AndroidManifest.xml` should setup all the extra "stuff" so developers don't have to manually do it. `android:extractNativeLibs="false"` would automatically do the following: 1. `.so` will be appended to `$(AndroidStoreUncompressedFileExtensions)`. Both the `` and `` MSBuild tasks use this property. 3. A new generated file in `$(IntermediateOutputPath)` will add `__XA_DSO_IN_APK=1` as an `@(AndroidEnvironment)` build item. The problem here is with incremental builds... If `_GenerateJavaStubs` is skipped, we still need to know if `android:extractNativeLibs` is set. To make this work, I added a `CacheFile` property on the `GenerateJavaStubs` MSBuild task. It writes a simple XML document in `$(IntermediateOutputPath)javastubs.cache` such as: True The file will not exist at all unless the value is `True`, which will prevent this feature from impacting build times. I also added an MSBuild test to verify these changes are happening. I was also able to update `tests/EmbeddedDSOs` to rely on the new functionality. ~~ Other Changes ~~ I updated `_GenerateJavaStubs` to follow the "stamp" file convention of using `$(_AndroidStampDirectory)_GenerateJavaStubs.stamp`. --- Documentation/guides/embedded-dsos.md | 58 +++++++++++++++++++ .../Tasks/GenerateJavaStubs.cs | 23 +++++++- .../Tasks/ReadJavaStubsCache.cs | 28 +++++++++ .../PackagingTest.cs | 47 +++++++++++++++ .../Utilities/ManifestDocument.cs | 14 +++++ .../Xamarin.Android.Build.Tasks.csproj | 1 + .../Xamarin.Android.Common.targets | 46 +++++++++++++-- .../Jars/.gitignore | 1 + .../EmbeddedDSO/EmbeddedDSO.csproj | 5 -- .../EmbeddedDSOs/EmbeddedDSO/Environment.txt | 1 - 10 files changed, 211 insertions(+), 13 deletions(-) create mode 100644 Documentation/guides/embedded-dsos.md create mode 100644 src/Xamarin.Android.Build.Tasks/Tasks/ReadJavaStubsCache.cs create mode 100644 tests/CodeGen-Binding/Xamarin.Android.McwGen-Tests/Jars/.gitignore diff --git a/Documentation/guides/embedded-dsos.md b/Documentation/guides/embedded-dsos.md new file mode 100644 index 00000000000..19d96007e65 --- /dev/null +++ b/Documentation/guides/embedded-dsos.md @@ -0,0 +1,58 @@ +# Embedded DSOs + +Android v6.0 (API-23) introduced a new way of dealing with the native +shared libraries shipped in the `.apk`. Before API-23, the libraries +would be always extracted and placed in the application data +directory, thus occupying more space than necessary. API-23 added a +new `AndroidManifest.xml` attribute, +`//application/@android:extractNativeLibs`, which if set to `false` +makes Android *not* extract the libraries to the filesystem. API-23 +added a way to instead load those libraries directly from the `.apk`. +In order to support that there are a few requirements which this +commit implements: + + * DSO (`.so`) files must be *stored uncompressed* in the `.apk`. + * `` must be set + * DSOs in the `.apk` must be aligned on the memory page boundary; + `zipalign -p` takes care of this. + +This commit also implements `libmonodroid.so` suport for loading our +DSOs directly from the `.apk`. This operation mode is enabled by the +presence of the `$__XA_DSO_IN_APK` environment variable. This +environment variable is inserted into the application's environment +by way of placing it in the environment file (a file part of the +Xamarin.Android App project that has the `@(AndroidEnvironment)` +build action). In this mode, the DSOs are *no longer* looked up in +the application data directory but only in the override directories +(if the APK is built in Debug configuration) and the `.apk` itself. + +When `/manifest/application/@android:extractNativeLibs` is set to +`false` in `AndroidManifest.xml`, Xamarin.Android should automatically +setup the proper uncompressed file extension settings and environment +variables. Adding the value to the manifest should "just work". + +See the official Android documentation for details about +[extractNativeLibs][extractNativeLibs] and its usage. + +[extractNativeLibs]: https://developer.android.com/guide/topics/manifest/application-element#extractNativeLibs + +# MSBuild Implementation Details + +When `/manifest/application/@android:extractNativeLibs` is set to +`false` in `AndroidManifest.xml`, Xamarin.Android's build system would +automatically do the following: + +1. `.so` will be automatically added to + `$(AndroidStoreUncompressedFileExtensions)`. Both the `` and + `` MSBuild tasks use this property. + +3. A new `dsoenvironment.txt` generated file in + `$(IntermediateOutputPath)` will add `__XA_DSO_IN_APK=1` as an + `@(AndroidEnvironment)` build item. + +To make this happen, Xamarin.Android's MSBuild targets will set the +`$(_EmbeddedDSOsEnabled)` property. A new `_SetupEmbeddedDSOs` MSBuild +target creates a `$(IntermediateOutputPath)dsoenvironment.txt` file +containing `__XA_DSO_IN_APK=1` when `$(_EmbeddedDSOsEnabled)` is +`True`. `_SetupEmbeddedDSOs` also prepends `.so;` to +`$(AndroidStoreUncompressedFileExtensions)`. diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs b/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs index 8ab08c7d297..fd5bcb37901 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs @@ -2,9 +2,9 @@ using System; using System.Collections.Generic; -using System.Diagnostics; using System.IO; using System.Linq; +using System.Xml.Linq; using Microsoft.Build.Framework; using Microsoft.Build.Utilities; using MonoDroid.Utils; @@ -68,6 +68,11 @@ public class GenerateJavaStubs : Task public string ApplicationJavaClass { get; set; } + /// + /// If specified, we need to cache the value of EmbeddedDSOsEnabled=True for incremental builds + /// + public string CacheFile { get; set; } + public override bool Execute () { try { @@ -223,6 +228,22 @@ void Run (DirectoryAssemblyResolver res) MonoAndroidHelper.CopyIfStreamChanged (stream, MergedAndroidManifestOutput); } + // Create the CacheFile if needed + if (!string.IsNullOrEmpty (CacheFile)) { + bool extractNativeLibraries = manifest.ExtractNativeLibraries (); + if (!extractNativeLibraries) { + //We need to write the value to a file, if _GenerateJavaStubs is skipped on incremental builds + var document = new XDocument ( + new XDeclaration ("1.0", "UTF-8", null), + new XElement ("Properties", new XElement (nameof (ReadJavaStubsCache.EmbeddedDSOsEnabled), "True")) + ); + document.SaveIfChanged (CacheFile); + } else { + //Delete the file otherwise, since we only need to specify when EmbeddedDSOsEnabled=True + File.Delete (CacheFile); + } + } + // Create additional runtime provider java sources. string providerTemplateFile = UseSharedRuntime ? "MonoRuntimeProvider.Shared.java" : "MonoRuntimeProvider.Bundled.java"; string providerTemplate = GetResource (providerTemplateFile); diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/ReadJavaStubsCache.cs b/src/Xamarin.Android.Build.Tasks/Tasks/ReadJavaStubsCache.cs new file mode 100644 index 00000000000..e5b1e120db6 --- /dev/null +++ b/src/Xamarin.Android.Build.Tasks/Tasks/ReadJavaStubsCache.cs @@ -0,0 +1,28 @@ +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; +using System.IO; +using System.Xml.Linq; + +namespace Xamarin.Android.Tasks +{ + public class ReadJavaStubsCache : Task + { + [Required] + public string CacheFile { get; set; } + + [Output] + public bool EmbeddedDSOsEnabled { get; set; } + + public override bool Execute () + { + if (File.Exists (CacheFile)) { + var doc = XDocument.Load (CacheFile); + string text = doc.Element ("Properties")?.Element (nameof (EmbeddedDSOsEnabled))?.Value; + if (bool.TryParse (text, out bool value)) + EmbeddedDSOsEnabled = value; + } + + return !Log.HasLoggedErrors; + } + } +} diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs index 873884083f0..80ace65b602 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs @@ -195,6 +195,53 @@ public void CheckIncludedNativeLibraries ([Values (true, false)] bool compressNa } } + [Test] + public void EmbeddedDSOs () + { + var proj = new XamarinAndroidApplicationProject (); + proj.AndroidManifest = $@" + + + + +"; + + using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { + Assert.IsTrue (b.Build (proj), "first build should have succeeded"); + + var apk = Path.Combine (Root, b.ProjectDirectory, + proj.IntermediateOutputPath, "android", "bin", "UnnamedProject.UnnamedProject.apk"); + AssertEmbeddedDSOs (apk); + + //Delete the apk & build again + File.Delete (apk); + Assert.IsTrue (b.Build (proj), "second build should have succeeded"); + AssertEmbeddedDSOs (apk); + } + } + + void AssertEmbeddedDSOs (string apk) + { + FileAssert.Exists (apk); + + using (var zip = ZipHelper.OpenZip (apk)) { + foreach (var entry in zip) { + if (entry.FullName.EndsWith (".so")) { + Assert.AreEqual (entry.Size, entry.CompressedSize, $"`{entry.FullName}` should be uncompressed!"); + } else if (entry.FullName == "environment") { + using (var stream = new MemoryStream ()) { + entry.Extract (stream); + stream.Position = 0; + using (var reader = new StreamReader (stream)) { + string environment = reader.ReadToEnd (); + StringAssert.Contains ("__XA_DSO_IN_APK=1", environment, "`__XA_DSO_IN_APK=1` should be set via @(AndroidEnvironment)"); + } + } + } + } + } + } + [Test] public void ExplicitPackageNamingPolicy () { diff --git a/src/Xamarin.Android.Build.Tasks/Utilities/ManifestDocument.cs b/src/Xamarin.Android.Build.Tasks/Utilities/ManifestDocument.cs index aab7e8720d4..8524dccb6ed 100644 --- a/src/Xamarin.Android.Build.Tasks/Utilities/ManifestDocument.cs +++ b/src/Xamarin.Android.Build.Tasks/Utilities/ManifestDocument.cs @@ -644,6 +644,20 @@ bool IsMainLauncher (XElement intentFilter) intentFilter.Elements (entry.Key).Any (e => ((string) e.Attribute (attName) == entry.Value))); } + /// + /// Returns the value of //application/@android:extractNativeLibs. + /// + public bool ExtractNativeLibraries () + { + string text = app?.Attribute (androidNs + "extractNativeLibs")?.Value; + if (bool.TryParse (text, out bool value)) { + return value; + } + + // If android:extractNativeLibs is omitted, returns true. + return true; + } + XElement ActivityFromTypeDefinition (TypeDefinition type, string name, int targetSdkVersion) { if (name.StartsWith ("_")) diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj index f16e8a78db9..3218846fe5b 100644 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj @@ -122,6 +122,7 @@ + diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets index bf4afabcdc2..57c3f186b80 100755 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets @@ -91,6 +91,7 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. + @@ -2285,7 +2286,7 @@ because xbuild doesn't support framework reference assemblies. + Outputs="$(_AndroidStampDirectory)_GenerateJavaStubs.stamp"> + AcwMapFile="$(_AcwMapFile)" + CacheFile="$(IntermediateOutputPath)javastubs.cache"> - + + + + + + + + + + + + + + + + .so;$(AndroidStoreUncompressedFileExtensions) + - + + - + + <_GetAddOnPlatformLibrariesDependsOn> + _GenerateJavaStubs; + _ReadJavaStubsCache; + _SetupEmbeddedDSOs; + + + + - diff --git a/tests/CodeGen-Binding/Xamarin.Android.McwGen-Tests/Jars/.gitignore b/tests/CodeGen-Binding/Xamarin.Android.McwGen-Tests/Jars/.gitignore new file mode 100644 index 00000000000..d392f0e82c4 --- /dev/null +++ b/tests/CodeGen-Binding/Xamarin.Android.McwGen-Tests/Jars/.gitignore @@ -0,0 +1 @@ +*.jar diff --git a/tests/EmbeddedDSOs/EmbeddedDSO/EmbeddedDSO.csproj b/tests/EmbeddedDSOs/EmbeddedDSO/EmbeddedDSO.csproj index 81d42105437..6c456feb415 100644 --- a/tests/EmbeddedDSOs/EmbeddedDSO/EmbeddedDSO.csproj +++ b/tests/EmbeddedDSOs/EmbeddedDSO/EmbeddedDSO.csproj @@ -18,11 +18,6 @@ true false armeabi-v7a;x86 - - - .so diff --git a/tests/EmbeddedDSOs/EmbeddedDSO/Environment.txt b/tests/EmbeddedDSOs/EmbeddedDSO/Environment.txt index 966b2ab27fb..ba117b60829 100644 --- a/tests/EmbeddedDSOs/EmbeddedDSO/Environment.txt +++ b/tests/EmbeddedDSOs/EmbeddedDSO/Environment.txt @@ -2,4 +2,3 @@ debug.mono.debug=1 MONO_LOG_LEVEL=debug MONO_LOG_MASK=asm MONO_XDEBUG=1 -__XA_DSO_IN_APK=1 From 58d9aa8d96476f7d2ba10105ceaa8d4f70e0c65b Mon Sep 17 00:00:00 2001 From: Jonathan Peppers Date: Thu, 3 Jan 2019 21:16:26 -0600 Subject: [PATCH 2/3] Fix for incremental builds in _SetupEmbeddedDSOs `` is wrong --- src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets | 1 - 1 file changed, 1 deletion(-) diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets index 57c3f186b80..89a4183e6bb 100755 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets @@ -2340,7 +2340,6 @@ because xbuild doesn't support framework reference assemblies. File="$(IntermediateOutputPath)dsoenvironment.txt" Lines="__XA_DSO_IN_APK=1" Overwrite="True" - WriteOnlyWhenDifferent="True" /> .so;$(AndroidStoreUncompressedFileExtensions) From f5e80a04c7bde9c7a90374b9e746b916fa522bf3 Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Thu, 3 Jan 2019 21:19:24 -0600 Subject: [PATCH 3/3] Documentation rewrite From: https://gist.github.com/jonpryor/fc46ef8f7933cede130eadeeafc736b9 --- Documentation/guides/embedded-dsos.md | 58 ------------------- .../guides/extract-native-libraries.md | 49 ++++++++++++++++ 2 files changed, 49 insertions(+), 58 deletions(-) delete mode 100644 Documentation/guides/embedded-dsos.md create mode 100644 Documentation/guides/extract-native-libraries.md diff --git a/Documentation/guides/embedded-dsos.md b/Documentation/guides/embedded-dsos.md deleted file mode 100644 index 19d96007e65..00000000000 --- a/Documentation/guides/embedded-dsos.md +++ /dev/null @@ -1,58 +0,0 @@ -# Embedded DSOs - -Android v6.0 (API-23) introduced a new way of dealing with the native -shared libraries shipped in the `.apk`. Before API-23, the libraries -would be always extracted and placed in the application data -directory, thus occupying more space than necessary. API-23 added a -new `AndroidManifest.xml` attribute, -`//application/@android:extractNativeLibs`, which if set to `false` -makes Android *not* extract the libraries to the filesystem. API-23 -added a way to instead load those libraries directly from the `.apk`. -In order to support that there are a few requirements which this -commit implements: - - * DSO (`.so`) files must be *stored uncompressed* in the `.apk`. - * `` must be set - * DSOs in the `.apk` must be aligned on the memory page boundary; - `zipalign -p` takes care of this. - -This commit also implements `libmonodroid.so` suport for loading our -DSOs directly from the `.apk`. This operation mode is enabled by the -presence of the `$__XA_DSO_IN_APK` environment variable. This -environment variable is inserted into the application's environment -by way of placing it in the environment file (a file part of the -Xamarin.Android App project that has the `@(AndroidEnvironment)` -build action). In this mode, the DSOs are *no longer* looked up in -the application data directory but only in the override directories -(if the APK is built in Debug configuration) and the `.apk` itself. - -When `/manifest/application/@android:extractNativeLibs` is set to -`false` in `AndroidManifest.xml`, Xamarin.Android should automatically -setup the proper uncompressed file extension settings and environment -variables. Adding the value to the manifest should "just work". - -See the official Android documentation for details about -[extractNativeLibs][extractNativeLibs] and its usage. - -[extractNativeLibs]: https://developer.android.com/guide/topics/manifest/application-element#extractNativeLibs - -# MSBuild Implementation Details - -When `/manifest/application/@android:extractNativeLibs` is set to -`false` in `AndroidManifest.xml`, Xamarin.Android's build system would -automatically do the following: - -1. `.so` will be automatically added to - `$(AndroidStoreUncompressedFileExtensions)`. Both the `` and - `` MSBuild tasks use this property. - -3. A new `dsoenvironment.txt` generated file in - `$(IntermediateOutputPath)` will add `__XA_DSO_IN_APK=1` as an - `@(AndroidEnvironment)` build item. - -To make this happen, Xamarin.Android's MSBuild targets will set the -`$(_EmbeddedDSOsEnabled)` property. A new `_SetupEmbeddedDSOs` MSBuild -target creates a `$(IntermediateOutputPath)dsoenvironment.txt` file -containing `__XA_DSO_IN_APK=1` when `$(_EmbeddedDSOsEnabled)` is -`True`. `_SetupEmbeddedDSOs` also prepends `.so;` to -`$(AndroidStoreUncompressedFileExtensions)`. diff --git a/Documentation/guides/extract-native-libraries.md b/Documentation/guides/extract-native-libraries.md new file mode 100644 index 00000000000..b5e7de01b7f --- /dev/null +++ b/Documentation/guides/extract-native-libraries.md @@ -0,0 +1,49 @@ +# Installation of Native Libraries + +An Android `.apk` file can contain native libraries for one or more +architectures. Historically, all native libraries for the target +device are extracted at `.apk` installation time. This would result +in *two* copies of native libraries on the target device: a set of +native libraries which are stored compressed within the `.apk`, and +a separate set of native libraries on the Android target filesystem. + +Starting with Android v6.0 (API-23), Android added an ability for +native libraries to be stored *uncompressed* within the `.apk` along +with the ability to load those native libraries *from* the `.apk` +without requiring a separate filesystem copy of the native libraries. + +Android Things *requires* this new mechanism; `.apk` files installed +on Android Things will no longer have *any* native libraries extracted. + +As a result, the `.apk` will be *larger*, because the native +libraries are stored uncompressed within the `.apk`, but the +install size will be *smaller*, because there isn't a second "copy" +of the native libraries (one compressed in the `.apk`, one outside +of the `.apk`). + +On Android versions older than Android v6.0, the native libraries +will continue to be extracted during `.apk` installation. + +In order to indicate to Android v6.0 and later that native libraries +do not need to be extracted, the +[`//application/@android:extractNativeLibs`][extractNativeLibs] +attribute within `AndroidManifest.xml` must be set to `false.` + +[extractNativeLibs]: https://developer.android.com/guide/topics/manifest/application-element#extractNativeLibs + +## OSS Implementation Details + +When `AndroidManifest.xml` contains an XML attribute matching +`//application[@android:extractNativeLibs='false']`, the +Xamarin.Android build system will do the following: + + 1. The `$(AndroidStoreUncompressedFileExtensions)` MSBuild property + will be automatically updated to contain the `.so` file + extension, causing native libraries to be stored uncompressed + within the `.apk`. + + 2. The `__XA_DSO_IN_APK` environment variable will be set within the + created `.apk` file with the value of `1`, indicating to + the app that native libraries should be loaded from the `.apk` + itself instead of from the filesystem. +