Switch to the new Mono.Unix nuget

grendello · grendello · commit 82a007078e0f · 2021-06-10T14:24:13.000+02:00
`Xamarin.Android` uses the Mono's Mono.Posix assembly on Unix machines
to perform tasks not possible with BCL classes, provided so far by the
`Mono.Posix.NETStandard` nuget.  Unfortunately, that nuget doesn't work
with `dotnet` and so the `Mono.Posix` code has recently been extracted
from the Mono repository and placed in its own, from which the new
`Mono.Unix` nuget is built and published.

This commit switches to `Mono.Unix` in Xamarin.Android and also updates
a number of submodules which likewise migrated from
`Mono.Posix.NETStandard` to `Mono.Unix`

Additionally, LibZipSharp is updated as well, since it now uses
`Mono.Unix`

`Mono.Unix` no longer uses the older `libMonoPosixHelper` dynamic
library, replaced by a new `libMono.Unix` one.  This change, however,
broke a number of tests since the `Mono.Unix.dll` assembly was no longer
able to find its companion native shared library.  The reason is that
`Mono.Posix.NETStandard` used the "standard" helper library which was
also part of the Mono distribution that we use on all the Unix machines.
However, that meant the helper library from `Mono.Posix.NETStandard` was
never used, instead the Mono copy was loaded and everything worked as
expected.

`Mono.Unix`'s new native helper library, however, must be taken from the
nuget and both Mono and dotnet runtimes must be told where from to load
the library once a p/invoke into it is encountered in managed code.  The
native library is copied from the nuget to the referencing application's
output directory and it should be loaded from there.  This proved to be
easy for the "legacy" Mono - a simple `dllmap` configuration is provided
and everything works as it should.

With `dotnet` however, dllmap doesn't work.  `dotnet` has instead a
number (5 I think) mechanisms to configure where the native libraries
can be found.  Unfortunately, the mechanisms either require that a main
executable of the application calls the APIs on entry (e.g. in the
`Main`) method or that a JSON configuration file is provided for the
application, telling the runtime where the native libraries reside.  In
case of `Xamarin.Android.Build.Tasks` there is no main executable we can
configure, since it works in the msbuild context, providing tasks and
utilities to build Xamarin.Android apps.  In such instance, `dotnet`
could be persuaded to find the libraries by calling one of the 5 APIs.
The problem with this approach, however, is that this action would have
to be performed at **every** possible entry point to the
`Xamarin.Android.Build.Tasks` assembly, since any of them could be used
as the first one.  While certainly possible, it would be both fragile
and an unnecessary maintenance burden.  Instead, a simpler (albeit a bit
kludgy) solution was chosen.  The `Xamarin.Android.Build.Tasks` build
process now takes care of generating a fat (multi-architecture) binary
for macOS hosts (including `x86-64` and `arm64` architectures) using the
`lipo` utility, then it copies the resulting binary to the same
directory where `Xamarin.Android.Build.Tasks.dll` and `Mono.Unix.dll`
live.  The Linux shared library is also copied to the same location.
The `dotnet` runtime is able to find and load shared libraries that are
in the same directory as the assembly that needs them and everything
works as expected.
diff --git a/.external b/.external
@@ -1,2 +1,2 @@
-xamarin/monodroid:main@76c04cd15eca7afc269a6d26296e9d2db6f79be2
+xamarin/monodroid:main@1f2ce156245ef1bf63ec8014882d283b3224216b
 mono/mono:2020-02@c633fe923832f0c3db3c4e6aa98e5592bf5a06e7
diff --git a/Directory.Build.props b/Directory.Build.props
@@ -26,7 +26,7 @@
 
   <!-- Common <PackageReference/> versions -->
   <PropertyGroup>
-    <LibZipSharpVersion>1.0.24</LibZipSharpVersion>
+    <LibZipSharpVersion>2.0.0-alpha6</LibZipSharpVersion>
     <MicroBuildCoreVersion>0.4.1</MicroBuildCoreVersion>
     <MonoCecilVersion>0.11.2</MonoCecilVersion>
     <NuGetApiPackageVersion>5.4.0</NuGetApiPackageVersion>
diff --git a/build-tools/create-packs/SignList.xml b/build-tools/create-packs/SignList.xml
@@ -1,7 +1,6 @@
 <Project>
   <ItemGroup>
     <!-- Do not sign files that already have a signature -->
-    <Skip Include="libzip*" />
     <Skip Include="libZipSharp*" />
   </ItemGroup>
 
diff --git a/build-tools/debian-metadata/rules b/build-tools/debian-metadata/rules
@@ -21,7 +21,7 @@ override_dh_install:
 	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/opt.exe
 	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/aapt2.exe
 	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/libwinpthread-1.dll
-	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/libzip.dll
+	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/libZipSharpNative.dll
 
 	dh_install
 
diff --git a/build-tools/installers/create-installers.targets b/build-tools/installers/create-installers.targets
@@ -193,10 +193,12 @@
     <_MSBuildFiles Include="@(AndroidSupportedTargetJitAbi->'$(MSBuildSrcDir)\lib\interpreter-%(Identity)\libmonosgen-2.0.so')"               ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\libZipSharp.dll" />
     <_MSBuildFiles Include="@(_LocalizationLanguages->'$(MSBuildSrcDir)\%(Identity)\libZipSharp.resources.dll')" />
-    <_MSBuildFiles Include="$(MSBuildSrcDir)\libZipSharp.dll.config" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\libZipSharp.pdb" />
-    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Posix.NETStandard.dll" />
-    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Posix.NETStandard.pdb" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Unix.dll" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Unix.dll.config" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Unix.pdb" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\libMono.Unix.so" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\libMono.Unix.dylib" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Profiler.Log.dll" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Profiler.Log.pdb" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\logcat-parse.exe"      ExcludeFromAndroidNETSdk="true" />
@@ -304,10 +306,10 @@
     <_MSBuildTargetsSrcFiles Include="$(MSBuildTargetsSrcDir)\Xamarin.Android.DefaultOutputPaths.targets" />
   </ItemGroup>
   <ItemGroup>
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libzip.dll" />
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libzip.pdb" />
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libzip.dll" />
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libzip.pdb" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libZipSharpNative.dll" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libZipSharpNative.pdb" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libZipSharpNative.dll" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libZipSharpNative.pdb" />
     <_MSBuildFilesWin Include="$(MSBuildSrcDir)\proguard\bin\proguard.bat" />
     <_MSBuildFilesWin Include="$(MSBuildSrcDir)\aapt2.exe" />
     <_MSBuildFilesWin Include="$(MSBuildSrcDir)\ndk\aarch64-linux-android-as.exe" />
@@ -368,9 +370,10 @@
     <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\lib\host-$(HostOS)\libmonosgen-2.0.$(LibExtension)"          ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\lib\host-$(HostOS)\libxamarin-app.$(LibExtension)"           ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\lib\host-$(HostOS)\libxa-internal-api.$(LibExtension)"       ExcludeFromAndroidNETSdk="true" />
-    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libzip.$(LibExtension)" />
-    <!-- A second libMonoPosixHelper.dylib is needed for libZipSharp.dll to load -->
-    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libMonoPosixHelper.$(LibExtension)" />
+    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libZipSharpNative.$(LibExtension)" />
+    <!-- A second libMono.Unix.{dylib,so} is needed for libZipSharp.dll to load -->
+    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libMono.Unix.so" />
+    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libMono.Unix.dylib" />
     <_MSBuildFilesUnix Include="$(MSBuildSrcDir)\proguard\bin\proguard.sh" />
   </ItemGroup>
   <!-- Allow us to exclude mono bundle files for PR builds -->
diff --git a/build-tools/xaprepare/xaprepare/xaprepare.csproj b/build-tools/xaprepare/xaprepare/xaprepare.csproj
@@ -52,7 +52,7 @@
     <PackageReference Include="7-Zip.CommandLine" Version="18.1.0" GeneratePathProperty="true" />
     <PackageReference Include="Kajabity.Tools.Java" Version="0.2.6862.30334" />
     <PackageReference Include="Mono.Options" Version="$(MonoOptionsVersion)" />
-    <PackageReference Include="Mono.Posix.NetStandard" Version="1.0.0" />
+    <PackageReference Include="Mono.Unix" Version="7.0.0-alpha8.21302.6" />
     <PackageReference Include="System.Net.Http" Version="4.3.4" />
     <PackageReference Include="Xamarin.LibZipSharp" Version="$(LibZipSharpVersion)" GeneratePathProperty="true" />
     <PackageReference Include="System.Text.Encoding.CodePages" Version="5.0.0" />
diff --git a/external/xamarin-android-tools b/external/xamarin-android-tools
@@ -1 +1 @@
-Subproject commit 683f37508b56c76c24b3287a5687743438625341
+Subproject commit e3d708c94cd6f29e05f36d87132022bfb40006be
diff --git a/src/Xamarin.Android.Build.Tasks/Resources/Mono.Unix.dll.config b/src/Xamarin.Android.Build.Tasks/Resources/Mono.Unix.dll.config
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="utf-8"?>
+<configuration>
+  <dllmap dll="Mono.Unix" os="linux" cpu="x86-64" wordsize="64" target="libMono.Unix.so" />
+  <dllmap dll="Mono.Unix" os="osx" target="libMono.Unix.dylib" />
+</configuration>
diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs
@@ -451,7 +451,7 @@ public void CheckAapt2WarningsDoNotGenerateErrors ()
 			//https://github.com/xamarin/xamarin-android/issues/3083
 			var proj = new XamarinAndroidApplicationProject () {
 				IsRelease = true,
-				TargetFrameworkVersion = Versions.Oreo_27,
+				TargetFrameworkVersion = Xamarin.ProjectTools.Versions.Oreo_27,
 				UseLatestPlatformSdk = false,
 			};
 			proj.PackageReferences.Add (KnownPackages.XamarinForms_2_3_4_231);
diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj
@@ -366,12 +366,6 @@
     <None Include="startup-xf.aotprofile">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </None>
-    <None Include="$(PkgMono_Posix_NETStandard)\runtimes\osx\native\libMonoPosixHelper.dylib">
-      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
-    </None>
-    <None Include="$(PkgMono_Posix_NETStandard)\runtimes\linux-x64\native\libMonoPosixHelper.so">
-      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
-    </None>
   </ItemGroup>
 
   <ItemGroup>
diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets
@@ -351,24 +351,57 @@
     <_ExtraPackageTarget Include="$(OutputPath)\libZipSharp.pdb" />
     <_ExtraPackageSource Include="$(PkgXamarin_Build_AsyncTask)\lib\$(TargetFrameworkNETStandard)\Xamarin.Build.AsyncTask.pdb" />
     <_ExtraPackageTarget Include="$(OutputPath)\Xamarin.Build.AsyncTask.pdb" />
-    <_ExtraPackageSource Include="$(PkgMono_Posix_NETStandard)\ref\$(TargetFrameworkNETStandard)\Mono.Posix.NETStandard.pdb" />
-    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Posix.NETStandard.pdb" />
-    <_ExtraPackageSource Include="$(PkgMono_Posix_NETStandard)\ref\$(TargetFrameworkNETStandard)\Mono.Posix.NETStandard.dll" />
-    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Posix.NETStandard.dll" />
+    <_ExtraPackageSource Include="$(PkgMono_Unix)\lib\$(TargetFrameworkNETStandard)\Mono.Unix.pdb" />
+    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Unix.pdb" />
+
+    <_ExtraPackageSource Include="$(MSBuildThisFileDirectory)\Resources\Mono.Unix.dll.config" />
+    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Unix.dll.config" />
+
+    <_ExtraPackageSource Include="$(PkgMono_Unix)\lib\$(TargetFrameworkNETStandard)\Mono.Unix.dll" />
+    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Unix.dll" />
+
+    <_ExtraPackageSource Include="$(PkgMono_Unix)\runtimes\linux-x64\native\libMono.Unix.so" />
+    <_ExtraPackageTarget Include="$(OutputPath)\libMono.Unix.so" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-arm64\native\libMono.Unix.dylib" Condition=" '$(HostOS)' == 'Darwin' " />
+    <_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-x64\native\libMono.Unix.dylib" />
   </ItemGroup>
 
+  <PropertyGroup>
+    <_MonoUnixOutputDylib>$(OutputPath)\libMono.Unix.dylib</_MonoUnixOutputDylib>
+  </PropertyGroup>
+
+  <!--
+      On operating systems other than macOS we just need to copy a single .dylib, so that pack creation works
+  -->
+  <Target Name="_MakeMonoUnixFatBinariesOSX"
+      Inputs="@(_MonoUnixDylib)"
+      Outputs="$(_MonoUnixOutputDylib)">
+    <Exec
+        Command="lipo @(_MonoUnixDylib->'&quot;%(FullPath)&quot;', ' ') -create -output &quot;$(_MonoUnixOutputDylib)&quot;"
+        Condition=" '$(HostOS)' == 'Darwin' "
+    />
+    <Copy
+        SourceFiles="@(_MonoUnixDylib)"
+        DestinationFolder="$(OutputPath)"
+    />
+  </Target>
+
   <!-- Copy package ref symbols for our installers. '$(Pkg*)' props are set during NuGet restore when GeneratePathProperty="true" -->
   <Target Name="_CopyExtraPackageContent"
       Inputs="@(_ExtraPackageSource)"
-      Outputs="@(_ExtraPackageTarget)" >
+      Outputs="@(_ExtraPackageTarget)"
+      DependsOnTargets="_MakeMonoUnixFatBinariesOSX">
     <Copy
         SourceFiles="@(_ExtraPackageSource)"
         DestinationFolder="$(OutputPath)"
         SkipUnchangedFiles="True"
     />
   </Target>
 
-  <Target Name="_Remove32BitLinux" Condition="Exists ('$(OutputPath)lib32\libzip.so')" AfterTargets="CopyFilesToOutputDirectory">
-    <Delete Files="$(OutputPath)lib32\libzip.so" />
+  <Target Name="_Remove32BitLinux" Condition="Exists ('$(OutputPath)lib32\libZipSharpNative.so')" AfterTargets="CopyFilesToOutputDirectory">
+    <Delete Files="$(OutputPath)lib32\libZipSharpNative.so" />
   </Target>
 </Project>
diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets
@@ -91,6 +91,7 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
 <UsingTask TaskName="Xamarin.Android.Tasks.PrepareAbiItems" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="Xamarin.Android.Tasks.WriteLockFile" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="GenerateCompressedAssembliesNativeSourceFiles" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
+
 <!--
 *******************************************
   Extensibility hook that allows VS to
@@ -102,7 +103,6 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
         Condition="Exists('$(MSBuildThisFileDirectory)$(MSBuildThisFileName).Before.targets')"/>
 
 
-
 <!--
 *******************************************
           Code Analysis Setup
diff --git a/tests/MSBuildDeviceIntegration/Tests/InstantRunTest.cs b/tests/MSBuildDeviceIntegration/Tests/InstantRunTest.cs
@@ -297,6 +297,10 @@ public void InstantRunNativeLibrary ([Values ("dx", "d8")] string dexTool)
 					nativeLib,
 				},
 			};
+			if (Builder.UseDotNet) {
+				//NOTE: in .NET 6 by default an x86_64 emulator would fall back to x86 if we don't set this.
+				proj.SetAndroidSupportedAbis (DeviceAbi);
+			}
 			proj.SetDefaultTargetDevice ();
 			using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) {
 				Assert.IsTrue (b.Install (proj), "install should have succeeded. 0");

Original file line number	Diff line number	Diff line change
`@@ -1,2 +1,2 @@`
`1`		`-xamarin/monodroid:main@76c04cd15eca7afc269a6d26296e9d2db6f79be2`
	`1`	`+xamarin/monodroid:main@1f2ce156245ef1bf63ec8014882d283b3224216b`
`2`	`2`	`mono/mono:2020-02@c633fe923832f0c3db3c4e6aa98e5592bf5a06e7`