Skip to content

[build] Extend dotnet build support on unix #6585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 5, 2022
Merged

[build] Extend dotnet build support on unix #6585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2cef9a1
[WIP] More dotnet build support for unix
pjcollins Dec 20, 2021
31395e4
Feedback
pjcollins Dec 22, 2021
d603046
Formatting
pjcollins Dec 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ _PREPARE_CI_MODE_ARGS = $(_PREPARE_CI_MODE_PR_ARGS) -a
_PREPARE_ARGS =

all:
$(call MSBUILD_BINLOG,all,$(_SLN_BUILD)) /restore $(MSBUILD_FLAGS) $(SOLUTION)
$(call DOTNET_BINLOG,all) $(MSBUILD_FLAGS) $(SOLUTION) -m:1
Copy link
Member

@jonathanpeppers jonathanpeppers Dec 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should DOTNET_BINLOG add the -m:1, so this is set in one place?

Copy link
Member Author

@pjcollins pjcollins Dec 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if we want to set it for every dotnet invocation as we should only need it when building Xamarin.Android.sln.

I haven't found great documentation on this parameter though. Neither the command line docs (below) or this guide mention whether or not this is a "solution only" flag, or if it also works on projects with multiple project references.

  -maxCpuCount[:n]   Specifies the maximum number of concurrent processes to
                     build with. If the switch is not used, the default
                     value used is 1. If the switch is used without a value
                     MSBuild will use up to the number of processors on the
                     computer. (Short form: -m[:n])

The command line docs aren't consistent with what I see locally either. The Xamarin.Android.sln build fails for me in lots of weird ways if -m:1 isn't set (even though the docs suggest the default value should be 1, which I assume would be "no parallelization" ). I know we had to set this explicitly on the Windows .sln build in the past for similar reasons as well.

The overall mac build job does seem faster in general in this PR, but I haven't done any digging into how much time is saved by 1) skipping the OSS zip creation, 2) using dotnet build more, or even the potential of 3) having more parallelization in some build commands by default.

My inclination is to keep it mostly as is, and only specify the -m flag when we know something breaks without it.

Copy link
Member

@jonathanpeppers jonathanpeppers Dec 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, we can leave -m:1 for now.

$(call MSBUILD_BINLOG,all,$(_SLN_BUILD)) /restore $(MSBUILD_FLAGS) tools/xabuild/xabuild.csproj

-include bin/Build$(CONFIGURATION)/rules.mk
Expand Down Expand Up @@ -97,7 +97,7 @@ all-tests::
MSBUILD="$(MSBUILD)" $(call MSBUILD_BINLOG,all-tests,tools/scripts/xabuild) /restore $(MSBUILD_FLAGS) Xamarin.Android-Tests.sln

pack-dotnet::
$(call MSBUILD_BINLOG,pack-dotnet,$(_SLN_BUILD)) $(MSBUILD_FLAGS) Xamarin.Android.sln /t:PackDotNet
$(call DOTNET_BINLOG,pack-dotnet) $(MSBUILD_FLAGS) $(SOLUTION) -t:PackDotNet

install::
@if [ ! -d "bin/$(CONFIGURATION)" ]; then \
Expand Down Expand Up @@ -141,7 +141,7 @@ run-all-tests:
exit $$_r

clean:
$(call MSBUILD_BINLOG,clean) /t:Clean Xamarin.Android.sln
$(call DOTNET_BINLOG,clean) -t:Clean $(SOLUTION) -m:1
tools/scripts/xabuild $(MSBUILD_FLAGS) /t:Clean Xamarin.Android-Tests.sln

distclean:
Expand Down
14 changes: 13 additions & 1 deletion build-tools/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project>

<Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk" />

<PropertyGroup>
<TargetFramework>$(TargetFrameworkNETStandard)</TargetFramework>
Expand All @@ -19,4 +21,14 @@
<ProjectReference Include="..\..\external\xamarin-android-tools\src\Xamarin.Android.Tools.AndroidSdk\Xamarin.Android.Tools.AndroidSdk.csproj" />
</ItemGroup>

<Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk" />
<Import Project="$(XamarinAndroidSourcePath)build-tools\scripts\MonoUnixNative.targets" />

<PropertyGroup>
<BuildDependsOn>
$(BuildDependsOn);
_MakeMonoUnixFatBinariesOSX;
</BuildDependsOn>
</PropertyGroup>

</Project>
4 changes: 2 additions & 2 deletions build-tools/scripts/BuildEverything.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ ifneq ("$(wildcard $(topdir)/external/monodroid/Makefile)","")
cd $(topdir)/external/monodroid && ./configure --with-xamarin-android='$(topdir)'
cd $(topdir)/external/monodroid && $(MAKE) build-monodroid CONFIGURATION=$(CONFIGURATION) XAMARIN_ANDROID_PATH=$(topdir)
endif
$(MAKE) leeroy $(ZIP_OUTPUT)
$(MAKE) leeroy

leeroy: leeroy-all framework-assemblies

leeroy-all:
$(call MSBUILD_BINLOG,leeroy-all,$(_SLN_BUILD)) /restore $(SOLUTION) /p:Configuration=$(CONFIGURATION) $(_MSBUILD_ARGS)
$(call DOTNET_BINLOG,leeroy-all) $(SOLUTION) -m:1 $(_MSBUILD_ARGS)
$(call MSBUILD_BINLOG,leeroy-all,$(_SLN_BUILD)) /restore tools/xabuild/xabuild.csproj /p:Configuration=$(CONFIGURATION) $(_MSBUILD_ARGS)
30 changes: 30 additions & 0 deletions build-tools/scripts/MonoUnixNative.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<!--
See https://github.com/xamarin/xamarin-android/commit/ceca993068c0401f664676db85edec1d772bf5ff
-->
<Project>

<ItemGroup>
<None Include="$(PkgMono_Unix)\runtimes\linux-x64\native\libMono.Unix.so" CopyToOutputDirectory="PreserveNewest" />
<_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-arm64\native\libMono.Unix.dylib" Condition=" '$(HostOS)' == 'Darwin' " />
<_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-x64\native\libMono.Unix.dylib" />
</ItemGroup>

<PropertyGroup>
<_MonoUnixOutputDylib>$(OutputPath)\libMono.Unix.dylib</_MonoUnixOutputDylib>
</PropertyGroup>

<Target Name="_MakeMonoUnixFatBinariesOSX"
Inputs="@(_MonoUnixDylib)"
Outputs="$(_MonoUnixOutputDylib)">
<Exec
Command="lipo @(_MonoUnixDylib->'&quot;%(FullPath)&quot;', ' ') -create -output &quot;$(_MonoUnixOutputDylib)&quot;"
Condition=" '$(HostOS)' == 'Darwin' "
/>
<Copy
SourceFiles="@(_MonoUnixDylib)"
DestinationFolder="$(OutputPath)"
Condition=" '$(HostOS)' != 'Darwin' "
/>
</Target>

</Project>
6 changes: 4 additions & 2 deletions build-tools/scripts/Packaging.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ create-nupkgs:
$(call DOTNET_BINLOG,create-all-packs) -t:CreateAllPacks $(topdir)/build-tools/create-packs/Microsoft.Android.Sdk.proj

create-pkg:
MONO_IOMAP=all MONO_OPTIONS="$(MONO_OPTIONS)" $(call MSBUILD_BINLOG,create-pkg) /p:Configuration=$(CONFIGURATION) /t:CreatePkg \
$(call DOTNET_BINLOG,create-pkg) /t:CreatePkg \
build-tools/create-pkg/create-pkg.csproj \
$(if $(PACKAGE_VERSION),/p:ProductVersion="$(PACKAGE_VERSION)") \
$(if $(PACKAGE_VERSION_REV),/p:XAVersionCommitCount="$(PACKAGE_VERSION_REV)") \
Expand All @@ -21,10 +21,12 @@ create-pkg:
$(if $(_MSBUILD_ARGS),"$(_MSBUILD_ARGS)")

create-workload-installers:
MONO_IOMAP=all MONO_OPTIONS="$(MONO_OPTIONS)" $(call MSBUILD_BINLOG,create-workload-installers) /p:Configuration=$(CONFIGURATION) /t:CreateWorkloadInstallers \
$(call DOTNET_BINLOG,create-workload-installers) /t:CreateWorkloadInstallers \
Xamarin.Android.sln \
$(if $(_MSBUILD_ARGS),"$(_MSBUILD_ARGS)")

# create-vsix.csproj dependencies do not yet support `dotnet build`:
# .nuget/packages/microsoft.vssdk.buildtools/17.0.4207-preview4/build/Microsoft.VSSDK.BuildTools.targets(16,5): error MSB4801: The task factory "CodeTaskFactory" is not supported on the .NET Core version of MSBuild.
create-vsix:
MONO_IOMAP=all MONO_OPTIONS="$(MONO_OPTIONS)" $(call MSBUILD_BINLOG,create-vsix) /p:Configuration=$(CONFIGURATION) /p:CreateVsixContainer=True \
build-tools/create-vsix/create-vsix.csproj \
Expand Down
8 changes: 4 additions & 4 deletions build-tools/xaprepare/xaprepare/Application/GeneratedMakeRulesFile.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,8 @@ void Generate (Context context, StreamWriter sw)
WriteRuleLine ( "if [ $$? -ne 0 ] ; then \\");
WriteRuleLine ($"\trm -f {redistFile}; \\");
WriteRuleLine ( "fi; \\");
WriteRuleLine ( "$(call MSBUILD_BINLOG,Mono.Android,$(_SLN_BUILD)) src/Mono.Android/Mono.Android.csproj \\");
WriteRuleLine ( "\t/p:Configuration=$(CONFIGURATION) $(_MSBUILD_ARGS) \\");
WriteRuleLine ( "$(call DOTNET_BINLOG,Mono.Android) src/Mono.Android/Mono.Android.csproj \\");
WriteRuleLine ( "\t$(_MSBUILD_ARGS) \\");
WriteRuleLine ($"\t/p:AndroidApiLevel={apiLevel} /p:AndroidPlatformId={platformId} /p:AndroidFrameworkVersion={curVersion} \\");
WriteRuleLine ($"\t/p:AndroidPreviousFrameworkVersion={prevVersion} || exit 1;");

Expand Down Expand Up @@ -170,8 +170,8 @@ void Generate (Context context, StreamWriter sw)
void WriteMSBuildCall (string fileToRemovePath, string projectPath)
{
WriteRuleLine ($"rm -f {fileToRemovePath}");
WriteRuleLine ($"$(call MSBUILD_BINLOG,NUnitLite,$(_SLN_BUILD)) $(MSBUILD_FLAGS) {projectPath} \\");
WriteRuleLine ( "\t/p:Configuration=$(CONFIGURATION) $(_MSBUILD_ARGS) \\");
WriteRuleLine ($"$(call DOTNET_BINLOG,NUnitLite) $(MSBUILD_FLAGS) {projectPath} \\");
WriteRuleLine ( "\t$(_MSBUILD_ARGS) \\");
WriteRuleLine ($"\t/p:AndroidApiLevel={firstApiLevel} /p:AndroidPlatformId={firstPlatformId} \\");
WriteRuleLine ($"\t/p:AndroidFrameworkVersion={firstFramework} || exit 1;");
}
Expand Down
2 changes: 1 addition & 1 deletion external/debugger-libs
Open in desktop
Submodule debugger-libs updated 26 files
+1 −1 Mono.Debugger.Soft/Mono.Debugger.Soft.csproj
+9 −2 Mono.Debugger.Soft/Mono.Debugger.Soft/Connection.cs
+20 −0 Mono.Debugger.Soft/Mono.Debugger.Soft/EncEvents.cs
+13 −0 Mono.Debugger.Soft/Mono.Debugger.Soft/EventRequest.cs
+4 −2 Mono.Debugger.Soft/Mono.Debugger.Soft/EventType.cs
+11 −0 Mono.Debugger.Soft/Mono.Debugger.Soft/MethodMirror.cs
+2 −0 Mono.Debugger.Soft/Mono.Debugger.Soft/ModuleMirror.cs
+3 −4 Mono.Debugger.Soft/Mono.Debugger.Soft/ObjectMirror.cs
+3 −0 Mono.Debugger.Soft/Mono.Debugger.Soft/VirtualMachine.cs
+56 −45 Mono.Debugger.Soft/Mono.Debugger.Soft/VirtualMachineManager.cs
+2 −6 Mono.Debugging.Soft/Mono.Debugging.Soft.csproj
+1 −1 Mono.Debugging.Soft/SoftDebuggerAdaptor.cs
+4 −2 Mono.Debugging.Soft/SoftDebuggerBacktrace.cs
+33 −0 Mono.Debugging.Soft/SoftDebuggerSession.cs
+11 −0 Mono.Debugging/Mono.Debugging.Client/Catchpoint.cs
+2 −0 Mono.Debugging/Mono.Debugging.Client/ObjectValue.cs
+10 −1 Mono.Debugging/Mono.Debugging.Client/StackFrame.cs
+2 −0 Mono.Debugging/Mono.Debugging.Evaluation/RemoteFrameObject.cs
+1 −1 Mono.Debugging/Mono.Debugging.csproj
+1 −1 UnitTests/Mono.Debugging.Tests/Shared/AdvancedEvaluationTests.cs
+3 −3 UnitTests/Mono.Debugging.Tests/Shared/BreakpointsAndSteppingTests.cs
+1 −1 UnitTests/Mono.Debugging.Tests/Shared/CodeCompletionTests.cs
+2 −2 UnitTests/Mono.Debugging.Tests/Shared/DebugTests.cs
+2 −2 UnitTests/Mono.Debugging.Tests/Shared/DebuggerSessionTests.cs
+1 −1 UnitTests/Mono.Debugging.Tests/Shared/EvaluationTests.cs
+1 −1 UnitTests/Mono.Debugging.Tests/Shared/StackFrameTests.cs
32 changes: 2 additions & 30 deletions src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,8 +316,7 @@
<Target Name="ILRepacker"
BeforeTargets="CopyFilesToOutputDirectory"
Inputs="$(MSBuildAllProjects);@(IntermediateAssembly);@(InputAssemblies)"
Outputs="$(IntermediateOutputPath)ILRepacker.stamp"
Condition=" '$(HostOS)' != 'Linux' ">
Copy link
Contributor

@dellis1972 dellis1972 Jan 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can anyone remember the reason why we disabled this in the first place?

Copy link
Contributor

@grendello grendello Jan 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It used to crash on Linux under Mono

Outputs="$(IntermediateOutputPath)ILRepacker.stamp" >
<ItemGroup>
<_InputAssembliesThatExist Include="@(InputAssemblies)" Condition="Exists('%(Identity)')" />
<_NetstandardPath Include="@(ReferencePath->'%(RootDir)%(Directory)')" Condition="'%(FileName)%(Extension)' == 'netstandard.dll'" />
Expand Down Expand Up @@ -356,36 +355,9 @@

<_ExtraPackageSource Include="$(PkgMono_Unix)\lib\$(TargetFrameworkNETStandard)\Mono.Unix.dll" />
<_ExtraPackageTarget Include="$(OutputPath)\Mono.Unix.dll" />

<_ExtraPackageSource Include="$(PkgMono_Unix)\runtimes\linux-x64\native\libMono.Unix.so" />
<_ExtraPackageTarget Include="$(OutputPath)\libMono.Unix.so" />
</ItemGroup>

<ItemGroup>
<_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-arm64\native\libMono.Unix.dylib" Condition=" '$(HostOS)' == 'Darwin' " />
<_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-x64\native\libMono.Unix.dylib" />
</ItemGroup>

<PropertyGroup>
<_MonoUnixOutputDylib>$(OutputPath)\libMono.Unix.dylib</_MonoUnixOutputDylib>
</PropertyGroup>

<!--
On operating systems other than macOS we just need to copy a single .dylib, so that pack creation works
-->
<Target Name="_MakeMonoUnixFatBinariesOSX"
Inputs="@(_MonoUnixDylib)"
Outputs="$(_MonoUnixOutputDylib)">
<Exec
Command="lipo @(_MonoUnixDylib->'&quot;%(FullPath)&quot;', ' ') -create -output &quot;$(_MonoUnixOutputDylib)&quot;"
Condition=" '$(HostOS)' == 'Darwin' "
/>
<Copy
SourceFiles="@(_MonoUnixDylib)"
DestinationFolder="$(OutputPath)"
Condition=" '$(HostOS)' != 'Darwin' "
/>
</Target>
<Import Project="$(XamarinAndroidSourcePath)build-tools\scripts\MonoUnixNative.targets" />

<!-- Copy package ref symbols for our installers. '$(Pkg*)' props are set during NuGet restore when GeneratePathProperty="true" -->
<Target Name="_CopyExtraPackageContent"
Expand Down
2 changes: 1 addition & 1 deletion tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Target Name="_RemapAssemblies"
AfterTargets="Build"
Condition=" '$(TargetFramework)' != '' ">
Condition=" '$(TargetFramework)' == 'net472' ">
<ItemGroup>
<_Source Include="$(OutputPath)Mono.Debugging.Soft.dll">
<From>Mono.Cecil</From>
Expand Down