Skip to content

Conversation

@jonpryor
Copy link
Contributor

@jonpryor jonpryor commented Aug 12, 2016

DO NOT MERGE

Before merging the bundle filename needs to be changed (see the TODO in the commit). I don't want to do this yet so that I can see if the current bundle is used by the PR bot. :-)


Context: fbfd676

Stage 3 of the cunning plan is to (attempt to) use the mono bundle
introduced in commit fbfd676.

This "simple" desire (ha!) re-raises the architectural project
dependency issue "solved" in fbfd676, but first, a simple question:

What should download the mono bundle?

There are two plausible answers:

  1. make prepare can (somehow) handle it.
  2. MSBuild can (somehow) handle it.

Both are plausible. The problem with using make targets (1) is there
is increased potential for "duplication" -- duplication of the bundle
filename, downloading it, and extracting it. Plus, make isn't
"Windows friendly", in that GNU make isn't (normally) present with
Visual Studio. (NMAKE is, but the Makefiles in this project are not
compatible with NMAKE.)

Which brings us to MSBuild (2): can it handle the task?

To tackle that, we need to be able to have an MSBuild task project
which has no dependencies, so that it can download and extract the
mono bundle before anything else runs, as it may be downloading
contents which mean that other projects don't need to run.

The need for a "pre-bootstrap" task assembly -- called xa-prep-tasks
-- thus "undoes" some of the logic regarding libzip-windows.mdproj
and the <Zip/> task from fbfd676: it isn't possible to rely on
libzip from a "pre-build" state, as libzip is one of the things in
the mono bundle, so now we need two "bootstrap" task assemblies:
one without a libzip dependency -- xa-prep-tasks.dll -- and one
with a libzip dependency --
Xamarin.Android.Tools.BootstrapTasks.dll

Move tasks which don't currently require libzip -- or won't in the
future, or laziness -- from Xamarin.Android.Tools.BootstrapTasks.dll
and move them into xa-prep-tasks.dll.

With that architectural compromise in place, add xa-prep-tasks as a
@(ProjectReference) to various projects to help ensure it's built
first, and rearchitect bundle.mdproj so that
xa-prep-tasks.targets and bundle.targets can use the same targets
to compute the bundle filename, now in
build-tools/bundle/bundle-path.targets.

Add a post-build step to xa-prep-tasks.csproj which downloads and
extracts the expected mono bundle.

One "problem" (feature?) is that the new <SystemUnzip/> task doesn't
report errors as errors when unzip'ing the file. This turns out to be
fine here because when downloading the mono bundle from Azure we don't
get a 404 anyway -- Azure instead returns an XML document containing
an error message (wat?!). We can thus ignore most error handling
entirely...though we're also ignoring any checking for invalid
downloads, which is something we should address in the future.

Update the varioous project files so that they won't attempt to
rebuild binaries that were present in the mono bundle.

Finally, the mono bundle isn't quite right, as it installs llc.exe
and opt.exe into bin/, not lib/mandroid. This was overlooked in
commit fbfd676.

Add a version number to the bundle filename so that a unique filename
is used, thus ensuring that we attempt to download an updated mono
bundle in the future, and correct the CreateBundle target so that it
packages required files.

@jonpryor jonpryor force-pushed the jonp-use-bundle branch 2 times, most recently from 7528870 to 58ba31a Compare August 12, 2016 21:11
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Aug 14, 2016
Context: dotnet#162

PR dotnet#162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR dotnet#162 currently fails][0] when building `src/monodroid`:

	Executing: /Users/builder/android-toolchain/ndk/ndk-build CONFIGURATION=Debug NDK_LIBS_OUT=./libs/Debug NDK_OUT=./obj/Debug V=1
	...
	[armeabi-v7a] Compile thumb  : monodroid <= nl.c
	/Users/builder/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -MMD -MP -MF ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -Os -g -DNDEBUG -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib/src -Ijni/../../../external/mono/eglib/src -Ijni/zip -Ijni -DANDROID -ggdb3 -O0 -fno-omit-frame-pointer -std=c99 -DHAVE_LINUX_NETLINK_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -D_REENTRANT -DPLATFORM_ANDROID -DANDROID -DLINUX -Dlinux -D__linux_ -DHAVE_CONFIG_H -DJI_DLL_EXPORT -DMONO_DLL_EXPORT -I/libmonodroid/zip -I/include -I/include/eglib -mandroid -fno-strict-aliasing -ffunction-sections -fomit-frame-pointer -funswitch-loops -finline-limit=300 -fvisibility=hidden -fstack-protector -Wa,--noexecstack -Wformat -Werror=format-security -DDEBUG=1 -Wa,--noexecstack -Wformat -Werror=format-security    -isystem /Users/builder/android-toolchain/ndk/platforms/android-9/arch-arm/usr/include -c  jni/../../../external/mono/support/nl.c -o ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o
	In file included from jni/../../../external/mono/support/nl.h:3:0,
	                 from jni/../../../external/mono/support/nl.c:14:
	jni/../../../external/mono/eglib/src/glib.h:18:26: fatal error: eglib-config.h: No such file or directory
	 #include <eglib-config.h>
	                          ^
	compilation terminated.

`src/monodroid` fails to build because it requires build artifacts
located in
`build-tools/mono-runtimes/obj/$(Configuration)/%(Identity)/eglib`,
which doesn't exist when the mono bundle is being used (because we
never built mono! by design!)

Fix this issue by updating the `_InstallRuntimes` target in
`mono-runtimes.targets` to copy the generated eglib header files into
`bin/$(Configuration)/include/%(Identity)/eglib`, and update
`bundle.targets` so that the `include` directory is also included in
the mono bundle.

This should allow a future PR dotnet#162 fix to build, as all required
generated files will be present within the bundle.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/61/
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Aug 14, 2016
Context: dotnet#162

PR dotnet#162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR dotnet#162 currently fails][0] when building `src/monodroid`:

	Executing: /Users/builder/android-toolchain/ndk/ndk-build CONFIGURATION=Debug NDK_LIBS_OUT=./libs/Debug NDK_OUT=./obj/Debug V=1
	...
	[armeabi-v7a] Compile thumb  : monodroid <= nl.c
	/Users/builder/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -MMD -MP -MF ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -Os -g -DNDEBUG -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib/src -Ijni/../../../external/mono/eglib/src -Ijni/zip -Ijni -DANDROID -ggdb3 -O0 -fno-omit-frame-pointer -std=c99 -DHAVE_LINUX_NETLINK_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -D_REENTRANT -DPLATFORM_ANDROID -DANDROID -DLINUX -Dlinux -D__linux_ -DHAVE_CONFIG_H -DJI_DLL_EXPORT -DMONO_DLL_EXPORT -I/libmonodroid/zip -I/include -I/include/eglib -mandroid -fno-strict-aliasing -ffunction-sections -fomit-frame-pointer -funswitch-loops -finline-limit=300 -fvisibility=hidden -fstack-protector -Wa,--noexecstack -Wformat -Werror=format-security -DDEBUG=1 -Wa,--noexecstack -Wformat -Werror=format-security    -isystem /Users/builder/android-toolchain/ndk/platforms/android-9/arch-arm/usr/include -c  jni/../../../external/mono/support/nl.c -o ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o
	In file included from jni/../../../external/mono/support/nl.h:3:0,
	                 from jni/../../../external/mono/support/nl.c:14:
	jni/../../../external/mono/eglib/src/glib.h:18:26: fatal error: eglib-config.h: No such file or directory
	 #include <eglib-config.h>
	                          ^
	compilation terminated.

`src/monodroid` fails to build because it requires build artifacts
located in
`build-tools/mono-runtimes/obj/$(Configuration)/%(Identity)/eglib`,
which doesn't exist when the mono bundle is being used (because we
never built mono!)

Fix this issue by updating the `_InstallRuntimes` target in
`mono-runtimes.targets` to copy the generated eglib header files into
`bin/$(Configuration)/include/%(Identity)/eglib`, and update
`bundle.targets` so that the `include` directory is also included in
the mono bundle.

This should allow a future PR dotnet#162 fix to build, as all required
generated files will be present within the bundle.

Additionally, add a version number to the bundle filename, now `-v1`,
to note that the structure of the bundle has changed, *without* a
corresponding mono, llvm, or libzip commit bump.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/61/
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Aug 14, 2016
Context: dotnet#162

PR dotnet#162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR dotnet#162 currently fails][0] when building `src/monodroid`:

	Executing: /Users/builder/android-toolchain/ndk/ndk-build CONFIGURATION=Debug NDK_LIBS_OUT=./libs/Debug NDK_OUT=./obj/Debug V=1
	...
	[armeabi-v7a] Compile thumb  : monodroid <= nl.c
	/Users/builder/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -MMD -MP -MF ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -Os -g -DNDEBUG -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib/src -Ijni/../../../external/mono/eglib/src -Ijni/zip -Ijni -DANDROID -ggdb3 -O0 -fno-omit-frame-pointer -std=c99 -DHAVE_LINUX_NETLINK_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -D_REENTRANT -DPLATFORM_ANDROID -DANDROID -DLINUX -Dlinux -D__linux_ -DHAVE_CONFIG_H -DJI_DLL_EXPORT -DMONO_DLL_EXPORT -I/libmonodroid/zip -I/include -I/include/eglib -mandroid -fno-strict-aliasing -ffunction-sections -fomit-frame-pointer -funswitch-loops -finline-limit=300 -fvisibility=hidden -fstack-protector -Wa,--noexecstack -Wformat -Werror=format-security -DDEBUG=1 -Wa,--noexecstack -Wformat -Werror=format-security    -isystem /Users/builder/android-toolchain/ndk/platforms/android-9/arch-arm/usr/include -c  jni/../../../external/mono/support/nl.c -o ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o
	In file included from jni/../../../external/mono/support/nl.h:3:0,
	                 from jni/../../../external/mono/support/nl.c:14:
	jni/../../../external/mono/eglib/src/glib.h:18:26: fatal error: eglib-config.h: No such file or directory
	 #include <eglib-config.h>
	                          ^
	compilation terminated.

`src/monodroid` fails to build because it requires build artifacts
located in
`build-tools/mono-runtimes/obj/$(Configuration)/%(Identity)/eglib`,
which doesn't exist when the mono bundle is being used (because we
never built mono!)

Fix this issue by updating the `_InstallRuntimes` target in
`mono-runtimes.targets` to copy the generated eglib header files into
`bin/$(Configuration)/include/%(Identity)/eglib`, and update
`bundle.targets` so that the `include` directory is also included in
the mono bundle.

This should allow a future PR dotnet#162 fix to build, as all required
generated files will be present within the bundle.

Additionally, add the `$(HostOS)` and a version number to the bundle
filename, now `-v1`, to note that the structure of the bundle has
changed, *without* a corresponding mono, llvm, or libzip commit bump.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/61/
jonpryor added a commit that referenced this pull request Aug 14, 2016
Context: #162

PR #162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR #162 currently fails][0] when building `src/monodroid`:

	Executing: /Users/builder/android-toolchain/ndk/ndk-build CONFIGURATION=Debug NDK_LIBS_OUT=./libs/Debug NDK_OUT=./obj/Debug V=1
	...
	[armeabi-v7a] Compile thumb  : monodroid <= nl.c
	/Users/builder/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -MMD -MP -MF ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -Os -g -DNDEBUG -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib/src -Ijni/../../../external/mono/eglib/src -Ijni/zip -Ijni -DANDROID -ggdb3 -O0 -fno-omit-frame-pointer -std=c99 -DHAVE_LINUX_NETLINK_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -D_REENTRANT -DPLATFORM_ANDROID -DANDROID -DLINUX -Dlinux -D__linux_ -DHAVE_CONFIG_H -DJI_DLL_EXPORT -DMONO_DLL_EXPORT -I/libmonodroid/zip -I/include -I/include/eglib -mandroid -fno-strict-aliasing -ffunction-sections -fomit-frame-pointer -funswitch-loops -finline-limit=300 -fvisibility=hidden -fstack-protector -Wa,--noexecstack -Wformat -Werror=format-security -DDEBUG=1 -Wa,--noexecstack -Wformat -Werror=format-security    -isystem /Users/builder/android-toolchain/ndk/platforms/android-9/arch-arm/usr/include -c  jni/../../../external/mono/support/nl.c -o ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o
	In file included from jni/../../../external/mono/support/nl.h:3:0,
	                 from jni/../../../external/mono/support/nl.c:14:
	jni/../../../external/mono/eglib/src/glib.h:18:26: fatal error: eglib-config.h: No such file or directory
	 #include <eglib-config.h>
	                          ^
	compilation terminated.

`src/monodroid` fails to build because it requires build artifacts
located in
`build-tools/mono-runtimes/obj/$(Configuration)/%(Identity)/eglib`,
which doesn't exist when the mono bundle is being used (because we
never built mono!)

Fix this issue by updating the `_InstallRuntimes` target in
`mono-runtimes.targets` to copy the generated eglib header files into
`bin/$(Configuration)/include/%(Identity)/eglib`, and update
`bundle.targets` so that the `include` directory is also included in
the mono bundle.

This should allow a future PR #162 fix to build, as all required
generated files will be present within the bundle.

Additionally, add the `$(HostOS)` and a version number to the bundle
filename, now `-v1`, to note that the structure of the bundle has
changed, *without* a corresponding mono, llvm, or libzip commit bump.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/61/
@jonpryor jonpryor force-pushed the jonp-use-bundle branch 2 times, most recently from 58972f1 to 6e07fb8 Compare August 14, 2016 19:20
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Aug 14, 2016
Context: dotnet#162

PR dotnet#162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR dotnet#162 currently fails][0] when building `src/Mono.Data.Sqlite`:

	CSC: error CS2001: Source file `/Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/external/mono/mcs/build/common/Consts.cs' could not be found

Fix this error by copying `external/mono/mcs/build/common/Consts.cs`
into `bin/$(Configuration)/include`, and updating the `Consts.cs`
references within `Mono.Data.Sqlite.csproj` and `Mono.Posix.csproj` so
that they'll use the `include` copy.

This should allow us to use the bundle to build these projects.

Additionally, `llc.exe` and `opt.exe` are installed into
`$prefix/bin`, which isn't correct; `$prefix/bin` as a construct
doesn't exist on Windows. Install these into `$prefix/lib/mandroid`.

Update the bundle version to note these bundle changes.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/67/console
jonpryor added a commit that referenced this pull request Aug 14, 2016
#165)

Context: #162

PR #162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR #162 currently fails][0] when building `src/Mono.Data.Sqlite`:

	CSC: error CS2001: Source file `/Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/external/mono/mcs/build/common/Consts.cs' could not be found

Fix this error by copying `external/mono/mcs/build/common/Consts.cs`
into `bin/$(Configuration)/include`, and updating the `Consts.cs`
references within `Mono.Data.Sqlite.csproj` and `Mono.Posix.csproj` so
that they'll use the `include` copy.

This should allow us to use the bundle to build these projects.

Additionally, `llc.exe` and `opt.exe` are installed into
`$prefix/bin`, which isn't correct; `$prefix/bin` as a construct
doesn't exist on Windows. Install these into `$prefix/lib/mandroid`.

Update the bundle version to note these bundle changes.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/67/console
@jonpryor jonpryor force-pushed the jonp-use-bundle branch 2 times, most recently from 5f4138b to d16225b Compare August 15, 2016 23:46
@jonpryor
Copy link
Contributor Author

  1. This PR builds!
  2. This PR takes ~1/3 the time, taking ~4.5 minutes to build, compared to the historical average of 12 minutes.

prepare:: prepare-external prepare-props

# $(call GetPath,path)
GetPath = $(shell $(MSBUILD) $(MSBUILD_FLAGS) /p:DoNotLoadOSProperties=True /nologo /v:minimal /t:Get$(1)FullPath build-tools/scripts/Paths.targets | tr -d '[[:space:]]' )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of this kind of {x,ms}build invocation, to be honest. It's convenient if it works, but if it breaks... The error output of msbuild is mangled and next to impossible to read, also it gets interpreted by shell as a set of commands. Maybe (not as part of this PR) we should have a wrapper script for such one-liners that will properly check the exit status of the command as well as log and process its output.

@grendello
Copy link
Contributor

LGTM

<Target Name="GetBundleFileName"
DependsOnTargets="_GetHashes">
<PropertyGroup>
<XABundleFileName>bundle-v2-$(Configuration)-$(HostOS)-libzip=$(_LibZipHash),llvm=$(_LlvmHash),mono=$(_MonoHash).zip</XABundleFileName>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the v2 be a property? It is being used in more than one place and sounds like something that might be updated in future too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where else is -v2 being used? It should only be used in this property.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@radical
Copy link
Member

radical commented Aug 16, 2016

Other than what I commented on, LGTM!

@jonpryor jonpryor force-pushed the jonp-use-bundle branch 2 times, most recently from eccabe3 to 766d869 Compare August 16, 2016 17:19
Context: dotnet@fbfd676

Stage 3 of the cunning plan is to (attempt to) use the mono bundle
introduced in commit fbfd676.

This "simple" desire (ha!) re-raises the architectural project
dependency issue "solved" in fbfd676, but first, a simple question:

What should download the mono bundle?

There are two plausible answers:

1. `make prepare` can (somehow) handle it.
2. MSBuild can (somehow) handle it.

Both are plausible. The problem with using `make` targets (1) is there
is increased potential for "duplication" -- duplication of the bundle
filename, downloading it, and extracting it. Plus, `make` isn't
"Windows friendly", in that GNU make isn't (normally) present with
Visual Studio. (`NMAKE` is, but the Makefiles in this project are not
compatible with `NMAKE`.)

Which brings us to MSBuild (2): can it handle the task?

To tackle that, we need to be able to have an MSBuild task project
which has *no dependencies*, so that it can download and extract the
mono bundle *before anything else runs*, as it may be downloading
contents which mean that other projects don't *need* to run.

The need for a "pre-bootstrap" task assembly -- called `xa-prep-tasks`
-- thus "undoes" *some* of the logic regarding `libzip-windows.mdproj`
and the `<Zip/>` task from fbfd676: it isn't *possible* to rely on
`libzip` from a "pre-build" state, as `libzip` is one of the things in
the mono bundle, so now we need *two* "bootstrap" task assemblies:
one without a `libzip` dependency -- `xa-prep-tasks.dll` -- and one
*with* a `libzip` dependency --
`Xamarin.Android.Tools.BootstrapTasks.dll`

Move tasks which don't currently require `libzip` -- or won't in the
future, or laziness -- from `Xamarin.Android.Tools.BootstrapTasks.dll`
and move them into `xa-prep-tasks.dll`.

With that architectural compromise in place, add `xa-prep-tasks` as a
`@(ProjectReference)` to various projects to help ensure it's built
*first*, and rearchitect `bundle.mdproj` so that
`xa-prep-tasks.targets` and `bundle.targets` can use the same targets
to compute the bundle filename, now in
`build-tools/bundle/bundle-path.targets`.

Add a post-build step to `xa-prep-tasks.csproj` which downloads and
extracts the expected mono bundle.

One "problem" (feature?) is that the new `<SystemUnzip/>` task doesn't
report errors as errors when unzip'ing the file. This turns out to be
fine here because when downloading the mono bundle from Azure we don't
get a 404 *anyway* -- Azure instead returns an XML document containing
an error message (wat?!). We can thus ignore most error handling
entirely...though we're *also* ignoring any checking for invalid
downloads, which is something we should address in the future.

Update the varioous project files so that they won't attempt to
rebuild binaries that were present in the mono bundle.
@jonpryor jonpryor merged commit 2956811 into dotnet:master Aug 16, 2016
radekdoulik pushed a commit to radekdoulik/xamarin-android that referenced this pull request Aug 18, 2016
)

Context: dotnet#162

PR dotnet#162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR dotnet#162 currently fails][0] when building `src/monodroid`:

	Executing: /Users/builder/android-toolchain/ndk/ndk-build CONFIGURATION=Debug NDK_LIBS_OUT=./libs/Debug NDK_OUT=./obj/Debug V=1
	...
	[armeabi-v7a] Compile thumb  : monodroid <= nl.c
	/Users/builder/android-toolchain/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -MMD -MP -MF ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -Os -g -DNDEBUG -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib -Ijni/../../../build-tools/mono-runtimes/obj/Debug/armeabi-v7a/eglib/src -Ijni/../../../external/mono/eglib/src -Ijni/zip -Ijni -DANDROID -ggdb3 -O0 -fno-omit-frame-pointer -std=c99 -DHAVE_LINUX_NETLINK_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -D_REENTRANT -DPLATFORM_ANDROID -DANDROID -DLINUX -Dlinux -D__linux_ -DHAVE_CONFIG_H -DJI_DLL_EXPORT -DMONO_DLL_EXPORT -I/libmonodroid/zip -I/include -I/include/eglib -mandroid -fno-strict-aliasing -ffunction-sections -fomit-frame-pointer -funswitch-loops -finline-limit=300 -fvisibility=hidden -fstack-protector -Wa,--noexecstack -Wformat -Werror=format-security -DDEBUG=1 -Wa,--noexecstack -Wformat -Werror=format-security    -isystem /Users/builder/android-toolchain/ndk/platforms/android-9/arch-arm/usr/include -c  jni/../../../external/mono/support/nl.c -o ./obj/Debug/local/armeabi-v7a/objs/monodroid/__/__/__/external/mono/support/nl.o
	In file included from jni/../../../external/mono/support/nl.h:3:0,
	                 from jni/../../../external/mono/support/nl.c:14:
	jni/../../../external/mono/eglib/src/glib.h:18:26: fatal error: eglib-config.h: No such file or directory
	 #include <eglib-config.h>
	                          ^
	compilation terminated.

`src/monodroid` fails to build because it requires build artifacts
located in
`build-tools/mono-runtimes/obj/$(Configuration)/%(Identity)/eglib`,
which doesn't exist when the mono bundle is being used (because we
never built mono!)

Fix this issue by updating the `_InstallRuntimes` target in
`mono-runtimes.targets` to copy the generated eglib header files into
`bin/$(Configuration)/include/%(Identity)/eglib`, and update
`bundle.targets` so that the `include` directory is also included in
the mono bundle.

This should allow a future PR dotnet#162 fix to build, as all required
generated files will be present within the bundle.

Additionally, add the `$(HostOS)` and a version number to the bundle
filename, now `-v1`, to note that the structure of the bundle has
changed, *without* a corresponding mono, llvm, or libzip commit bump.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/61/
radekdoulik pushed a commit to radekdoulik/xamarin-android that referenced this pull request Aug 18, 2016
dotnet#165)

Context: dotnet#162

PR dotnet#162 tries to use the mono bundle (fbfd676) to avoid rebuilding
mono on subsequent builds (when the mono commit hasn't changed), in
order to speed up the Jenkins build process (currently tracking at
~3.5 *hours* for a non-PR build...)

[PR dotnet#162 currently fails][0] when building `src/Mono.Data.Sqlite`:

	CSC: error CS2001: Source file `/Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/external/mono/mcs/build/common/Consts.cs' could not be found

Fix this error by copying `external/mono/mcs/build/common/Consts.cs`
into `bin/$(Configuration)/include`, and updating the `Consts.cs`
references within `Mono.Data.Sqlite.csproj` and `Mono.Posix.csproj` so
that they'll use the `include` copy.

This should allow us to use the bundle to build these projects.

Additionally, `llc.exe` and `opt.exe` are installed into
`$prefix/bin`, which isn't correct; `$prefix/bin` as a construct
doesn't exist on Windows. Install these into `$prefix/lib/mandroid`.

Update the bundle version to note these bundle changes.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/67/console
radekdoulik pushed a commit to radekdoulik/xamarin-android that referenced this pull request Aug 18, 2016
Context: dotnet@fbfd676

Stage 3 of the cunning plan is to (attempt to) use the mono bundle
introduced in commit fbfd676.

This "simple" desire (ha!) re-raises the architectural project
dependency issue "solved" in fbfd676, but first, a simple question:

What should download the mono bundle?

There are two plausible answers:

1. `make prepare` can (somehow) handle it.
2. MSBuild can (somehow) handle it.

Both are plausible. The problem with using `make` targets (1) is there
is increased potential for "duplication" -- duplication of the bundle
filename, downloading it, and extracting it. Plus, `make` isn't
"Windows friendly", in that GNU make isn't (normally) present with
Visual Studio. (`NMAKE` is, but the Makefiles in this project are not
compatible with `NMAKE`.)

Which brings us to MSBuild (2): can it handle the task?

To tackle that, we need to be able to have an MSBuild task project
which has *no dependencies*, so that it can download and extract the
mono bundle *before anything else runs*, as it may be downloading
contents which mean that other projects don't *need* to run.

The need for a "pre-bootstrap" task assembly -- called `xa-prep-tasks`
-- thus "undoes" *some* of the logic regarding `libzip-windows.mdproj`
and the `<Zip/>` task from fbfd676: it isn't *possible* to rely on
`libzip` from a "pre-build" state, as `libzip` is one of the things in
the mono bundle, so now we need *two* "bootstrap" task assemblies:
one without a `libzip` dependency -- `xa-prep-tasks.dll` -- and one
*with* a `libzip` dependency --
`Xamarin.Android.Tools.BootstrapTasks.dll`

Move tasks which don't currently require `libzip` -- or won't in the
future, or laziness -- from `Xamarin.Android.Tools.BootstrapTasks.dll`
and move them into `xa-prep-tasks.dll`.

With that architectural compromise in place, add `xa-prep-tasks` as a
`@(ProjectReference)` to various projects to help ensure it's built
*first*, and rearchitect `bundle.mdproj` so that
`xa-prep-tasks.targets` and `bundle.targets` can use the same targets
to compute the bundle filename, now in
`build-tools/bundle/bundle-path.targets`.

Add a post-build step to `xa-prep-tasks.csproj` which downloads and
extracts the expected mono bundle.

One "problem" (feature?) is that the new `<SystemUnzip/>` task doesn't
report errors as errors when unzip'ing the file. This turns out to be
fine here because when downloading the mono bundle from Azure we don't
get a 404 *anyway* -- Azure instead returns an XML document containing
an error message (wat?!). We can thus ignore most error handling
entirely...though we're *also* ignoring any checking for invalid
downloads, which is something we should address in the future.

Update the varioous project files so that they won't attempt to
rebuild binaries that were present in the mono bundle.
jonpryor pushed a commit that referenced this pull request May 26, 2022
Changes: dotnet/android-tools@fc3c2ac...20f6112

  * dotnet/android-tools@20f6112: Bump LibZipSharp to 2.0.4 (#166)
  * dotnet/android-tools@e4f0d59: Add support for writing android:roundIcon to Android manifest (#162)

Changes: dotnet/android-libzipsharp@2.0.3...2.0.4

  * dotnet/android-libzipsharp@8ac30f3: Bump Version to 2.0.4 for Release (#109)
  * dotnet/android-libzipsharp@8e22fe1: Use RuntimeInformation to determine the Processor architecture. (#108)
  * dotnet/android-libzipsharp@33f5b01: Bump vcpkg to latest version (#106)
  * dotnet/android-libzipsharp@6fd0a14: [ci] Use new EO compliant build pools (#107)
  * dotnet/android-libzipsharp@d339358: Add a new more complex unit test (#105)

Other changes:

  * 32-bit `libZipSharpNative.dll` is now in
    `x86\libZipSharpNative.dll`.

  * 64-bit `lib64\libZipSharpNative.dll` is now in
    `x64\libZipSharpNative.dll`.
simonrozsival pushed a commit to simonrozsival/xamarin-android that referenced this pull request May 31, 2022
Changes: dotnet/android-tools@fc3c2ac...20f6112

  * dotnet/android-tools@20f6112: Bump LibZipSharp to 2.0.4 (dotnet#166)
  * dotnet/android-tools@e4f0d59: Add support for writing android:roundIcon to Android manifest (dotnet#162)

Changes: dotnet/android-libzipsharp@2.0.3...2.0.4

  * dotnet/android-libzipsharp@8ac30f3: Bump Version to 2.0.4 for Release (dotnet#109)
  * dotnet/android-libzipsharp@8e22fe1: Use RuntimeInformation to determine the Processor architecture. (dotnet#108)
  * dotnet/android-libzipsharp@33f5b01: Bump vcpkg to latest version (dotnet#106)
  * dotnet/android-libzipsharp@6fd0a14: [ci] Use new EO compliant build pools (dotnet#107)
  * dotnet/android-libzipsharp@d339358: Add a new more complex unit test (dotnet#105)

Other changes:

  * 32-bit `libZipSharpNative.dll` is now in
    `x86\libZipSharpNative.dll`.

  * 64-bit `lib64\libZipSharpNative.dll` is now in
    `x64\libZipSharpNative.dll`.
jonathanpeppers added a commit to jonathanpeppers/xamarin-android that referenced this pull request May 31, 2022
Changes: dotnet/android-tools@fc3c2ac...20f6112

  * dotnet/android-tools@20f6112: Bump LibZipSharp to 2.0.4 (dotnet#166)
  * dotnet/android-tools@e4f0d59: Add support for writing android:roundIcon to Android manifest (dotnet#162)

Changes: dotnet/android-libzipsharp@2.0.3...2.0.4

  * dotnet/android-libzipsharp@8ac30f3: Bump Version to 2.0.4 for Release (#109)
  * dotnet/android-libzipsharp@8e22fe1: Use RuntimeInformation to determine the Processor architecture. (#108)
  * dotnet/android-libzipsharp@33f5b01: Bump vcpkg to latest version (#106)
  * dotnet/android-libzipsharp@6fd0a14: [ci] Use new EO compliant build pools (#107)
  * dotnet/android-libzipsharp@d339358: Add a new more complex unit test (#105)

Other changes:

  * 32-bit `libZipSharpNative.dll` is now in
    `x86\libZipSharpNative.dll`.

  * 64-bit `lib64\libZipSharpNative.dll` is now in
    `x64\libZipSharpNative.dll`.
@github-actions github-actions bot locked and limited conversation to collaborators Feb 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants