[v7.0.0-rc.1.22426.10] Extremely slow build of non-portable runtime on `armv7`

[ayakael]

Non-portable build of runtime v7.0.0-rc.1.22426.10 extremly slow in alpine.3.17-arm environment. Portable build w/ RID linux-musl-arm has no issue, and v6.0.9 also builds without issue.

Environment:

• armv8l environment (i.e. 32-bit userspace in 64-bit aarch64 environment)
• Alpine Edge (3.17)
• clang / llvm version 14

Any pointers on what might be the problem with this?

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jkotas]

How did you build the runtime that is very slow? Is it release build?

build.sh produces Debug builds by default. Debug builds are have asserts on and compiler optimizations off that makes them multiple times slower than Release builds. Pass -c Release to build.sh to produce Release build.

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Non-portable build of runtime v7.0.0-rc.1.22426.10 extremly slow in alpine.3.17-arm environment. Portable build w/ RID linux-musl-arm has no issue, and v6.0.9 also builds without issue.

Environment:

• armv8l environment (i.e. 32-bit userspace in 64-bit aarch64 environment)
• Alpine Edge (3.17)
• clang / llvm version 14

Any pointers on what might be the problem with this?

Author:	ayakael
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[ayakael]

My apologies for not specifying - this is as built in source-build tarball. Here are the build settings:

/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/src/runtime/build.sh  --ci --configuration Release --restore --build --pack --publish -bl /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath="/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/artifacts/obj/arm/Release/PackageVersions.props" /p:AdditionalSourceBuiltNupkgCacheDir="/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/artifacts/obj/arm/Release/blob-feed/packages/" /p:ReferencePackageNupkgCacheDir="/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/packages/reference/" /p:PreviouslySourceBuiltNupkgCacheDir="/usr/lib/dotnet/artifacts/7.0.100_rc1/" /p:TargetRid=alpine.3.17-arm /p:SourceBuildNonPortable=true  /p:DotNetPackageVersionPropsPath=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/artifacts/obj/arm/Release/PackageVersions.props /p:DotNetRestoreSourcePropsPath=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/artifacts/obj/arm/Release/RestoreSources.props /p:DotNetBuildOffline=true
    Log: /builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/artifacts/logs/runtime.log
    With Environment Variables:
      DotNetBuildFromSource=true
      DotNetBuildFromSourceFlavor=Product
      DotNetPackageVersionPropsPath=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/artifacts/obj/arm/Release/PackageVersions.props
      DotNetRestorePackagesPath=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/packages/restored/
      DotNetBuildOffline=true
      AddDotnetfeedProjectSource=false
      DOTNET_INSTALL_DIR=/builds/ayakael/aports/testing/dotnet7-build/src/bootstrap/
      DOTNET_PATH=/builds/ayakael/aports/testing/dotnet7-build/src/bootstrap/
      DOTNET_HOST_PATH=/builds/ayakael/aports/testing/dotnet7-build/src/bootstrap/dotnet
      _InitializeDotNetCli=/builds/ayakael/aports/testing/dotnet7-build/src/bootstrap/
      _DotNetInstallDir=/builds/ayakael/aports/testing/dotnet7-build/src/bootstrap/
      _InitializeToolset=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
      _OverrideArcadeInitializeBuildToolFramework=net7.0
      DotNetUseShippingVersions=true
      PreReleaseVersionLabel=rc.1
      PackageVersionStamp=rc.1
      PB_VersionStamp=rc.1
      ContinuousIntegrationBuild=true
      MSBUILDDISABLENODEREUSE=1
      OfficialBuildId=20220826.10
      BUILD_BUILDNUMBER=20220826.10
      GitCommitCount=
      GitCommitHash=06aceb7015f3bd2ff019ef5920d2354eb2ea2c92
      GitInfoCommitHash=06aceb7015f3bd2ff019ef5920d2354eb2ea2c92
      SourceRevisionId=06aceb7015f3bd2ff019ef5920d2354eb2ea2c92
      RepositoryCommit=06aceb7015f3bd2ff019ef5920d2354eb2ea2c92
      COMMIT_SHA=06aceb7015f3bd2ff019ef5920d2354eb2ea2c92
      GIT_COMMIT=06aceb7015f3bd2ff019ef5920d2354eb2ea2c92
      RepositoryType=Git
      DeterministicSourcePaths=true
      SourceRoot=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/src/runtime/
      BuildInParallel=false
      LatestCommit=06aceb7015f3bd2ff019ef5920d2354eb2ea2c92
      SOURCE_BUILT_SDK_ID_ARCADE=Microsoft.DotNet.Arcade.Sdk
      SOURCE_BUILT_SDK_ID_ARCADE_PACKAGING=Microsoft.DotNet.Build.Tasks.Packaging
      SOURCE_BUILT_SDK_ID_ARCADE_TGT_FX=Microsoft.DotNet.Build.Tasks.TargetFramework
      SOURCE_BUILT_SDK_ID_ARCADE_SHARED_FX_SDK=Microsoft.DotNet.SharedFramework.Sdk
      SOURCE_BUILT_SDK_VERSION_ARCADE=7.0.0-beta.22418.4
      SOURCE_BUILT_SDK_VERSION_ARCADE_PACKAGING=7.0.0-beta.22418.4
      SOURCE_BUILT_SDK_VERSION_ARCADE_TGT_FX=7.0.0-beta.22418.4
      SOURCE_BUILT_SDK_VERSION_ARCADE_SHARED_FX_SDK=7.0.0-beta.22418.4
      SOURCE_BUILT_SDK_DIR_ARCADE=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/
      SOURCE_BUILT_SDK_DIR_ARCADE_PACKAGING=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/Tools/source-built/Microsoft.DotNet.Build.Tasks.Packaging/
      SOURCE_BUILT_SDK_DIR_ARCADE_TGT_FX=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/Tools/source-built/Microsoft.DotNet.Build.Tasks.TargetFramework/
      SOURCE_BUILT_SDK_DIR_ARCADE_SHARED_FX_SDK=/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-v7.0.100-rc.1.22431.12/Tools/source-built/Microsoft.DotNet.SharedFramework.Sdk/

Indeed, configuration is set to Release

[ayakael]

For references, build on aarch64 of source-build takes 1h48min. Build on that same machine in armv7 user-space takes 7h50 minutes, most of it (6hours) is runtime build in non-portable.

[jkotas]

It means that some build commands take a lot longer in non-portable build. Are you able to identify these from the logs?

[ayakael]

Interesting - the extreme slowness only occurs with gcc. In clang mode (with -clang), both aarch64 and armv7 build about the same speed. I don't know if this is reproducible on other (non-musl) platforms. As a mitigation, I'll use clang to build dotnet on armv7.

[ayakael]

The above comment is a lie - the slowness on gcc was just a fluke. Basically back to zero. Building just runtime in non-portable mode outside of source-build finishes in 15 minutes, just like its portable counterpart. Moving to sourcebuild inflates that to hours. Essentially everything gets slower, there's not just one step that slows down. Of course, having timestamps on everything would be great to really measure out how long each steps take, but unfortunately I do not know of any switch that would do that

[hoyosjs]

I wonder if using -bl would help us check here for some high level broad timestamps.

[ayakael]

I'm trying now a build with /p:LogVerbosity:diag, if memory serves that activated log dumps of info. Right now I'm testing against the possibility that my bootstrap is to blame. The latter is a product of a linux-musl-x64 to linux-musl-armv7 crossbuild using:

ROOTFS_DIR="$CBUILDROOT" ./build.sh 
		-c Release
		-arch $_dotnet_target
		-clang
		/p:NoPgoOptimize=true
		/p:EnableNgenOptimization=false
		/p:VersionSuffix=${_runtimever##*-}
		/p:GitCommitHash=$(cat ./.git/HEAD)

Full script here also crossbuild llvm, sdk, aspnetcore, installer and roslyn.

[ayakael]

Could the disabling of NgenOptimization and NoPgoOptimize be the cause for this? I had issues building them due to lack of pgo packages, thus disabled.