Skip to content

Conversation

@jonathanpeppers
Copy link
Member

Context: #5040

In trying to enable our F# MSBuild tests, one fails with:

GenerateCompressedAssembliesNativeSourceFiles
    Xamarin.Android.Common.targets(1933,3): error XAGCANSF7004: System.ArgumentException: An item with the same key has already been added. Key: [FSharp.Core.resources.dll, Xamarin.Android.Tasks.CompressedAssemblyInfo]
    at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
    at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
    at Xamarin.Android.Tasks.GenerateCompressedAssembliesNativeSourceFiles.GenerateCompressedAssemblySources()
    at Xamarin.Android.Tasks.GenerateCompressedAssembliesNativeSourceFiles.RunTask()
    at Xamarin.Android.Tasks.AndroidTask.Execute()

Looking at some of the item groups earlier:

C:\src\xamarin-android\packages\xamarin.android.fsharp.resourceprovider\1.0.1\lib\monoandroid81\Xamarin.Android.FSharp.ResourceProvider.Runtime.dll
    ...
    DestinationSubPath = Xamarin.Android.FSharp.ResourceProvider.Runtime.dll
    ...
C:\src\xamarin-android\packages\fsharp.core\4.7.2\lib\netstandard2.0\cs\FSharp.Core.resources.dll
    ...
    DestinationSubDirectory = cs\
    DestinationSubPath = cs\FSharp.Core.resources.dll
    ...
C:\src\xamarin-android\packages\fsharp.core\4.7.2\lib\netstandard2.0\de\FSharp.Core.resources.dll
    ...
    DestinationSubDirectory = de\
    DestinationSubPath = de\FSharp.Core.resources.dll
    ...

Two instances of FSharp.Core.resources.dll is causing the above
exception. Should we be using the %(DestinationSubDirectory) and
%(DestinationSubPath) item metadata?

Currently, we have some behavior for architecture-specific .NET
assemblies:

  • %(AbiDirectory) is set for x86, armeabi-v7a, etc.
  • %(IntermediateLinkerOutput) is set to a full path taking
    %(AbiDirectory) into account.

To clean things up here, we should just use
%(DestinationSubDirectory) coming from dotnet/sdk and
%(DestinationSubPath) and remove the item data that we invented.

Other changes:

  • Usage of %(AbiDirectory) can use %(DestinationSubDirectory) or
    %(DestinationSubPath) where appropriate.
  • Any usage of %(IntermediateLinkerOutput) can use
    $(MonoAndroidIntermediateAssemblyDir)%(DestinationSubPath)
    instead.
  • The <ResolveAssemblies/> (legacy) and <ProcessAssemblies/>
    MSBuild tasks no longer need the IntermediateAssemblyDirectory
    property.

@jonathanpeppers jonathanpeppers force-pushed the destinationsubdirectory branch 2 times, most recently from 18f77c5 to 33adda2 Compare August 25, 2020 14:38
Context: dotnet#5040

In trying to enable our F# MSBuild tests, one fails with:

    GenerateCompressedAssembliesNativeSourceFiles
        Xamarin.Android.Common.targets(1933,3): error XAGCANSF7004: System.ArgumentException: An item with the same key has already been added. Key: [FSharp.Core.resources.dll, Xamarin.Android.Tasks.CompressedAssemblyInfo]
        at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
        at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
        at Xamarin.Android.Tasks.GenerateCompressedAssembliesNativeSourceFiles.GenerateCompressedAssemblySources()
        at Xamarin.Android.Tasks.GenerateCompressedAssembliesNativeSourceFiles.RunTask()
        at Xamarin.Android.Tasks.AndroidTask.Execute()

Looking at some of the item groups earlier:

    C:\src\xamarin-android\packages\xamarin.android.fsharp.resourceprovider\1.0.1\lib\monoandroid81\Xamarin.Android.FSharp.ResourceProvider.Runtime.dll
        ...
        DestinationSubPath = Xamarin.Android.FSharp.ResourceProvider.Runtime.dll
        ...
    C:\src\xamarin-android\packages\fsharp.core\4.7.2\lib\netstandard2.0\cs\FSharp.Core.resources.dll
        ...
        DestinationSubDirectory = cs\
        DestinationSubPath = cs\FSharp.Core.resources.dll
        ...
    C:\src\xamarin-android\packages\fsharp.core\4.7.2\lib\netstandard2.0\de\FSharp.Core.resources.dll
        ...
        DestinationSubDirectory = de\
        DestinationSubPath = de\FSharp.Core.resources.dll
        ...

Two instances of `FSharp.Core.resources.dll` is causing the above
exception. Should we be using the `%(DestinationSubDirectory)` and
`%(DestinationSubPath)` item metadata?

Currently, we have some behavior for architecture-specific .NET
assemblies:

* `%(AbiDirectory)` is set for x86, armeabi-v7a, etc.
* `%(IntermediateLinkerOutput)` is set to a full path taking
  `%(AbiDirectory)` into account.

To clean things up here, we should just use
`%(DestinationSubDirectory)` coming from dotnet/sdk and
`%(DestinationSubPath)` and remove the item data that we invented.

Other changes:

* Usage of `%(AbiDirectory)` can use `%(DestinationSubDirectory)` or
  `%(DestinationSubPath)` where appropriate.
* Any usage of `%(IntermediateLinkerOutput)` can use
  `$(MonoAndroidIntermediateAssemblyDir)%(DestinationSubPath)`
  instead.
* The `<ResolveAssemblies/>` (legacy) and `<ProcessAssemblies/>`
  MSBuild tasks no longer need the `IntermediateAssemblyDirectory`
  property.
* A new check was needed in `<ProcessAssemblies/>` to make sure we
  don't append `%(DestinationSubDirectory)` for the same assembly
  multiple times.
@jonathanpeppers jonathanpeppers force-pushed the destinationsubdirectory branch from 33adda2 to fb9e5d5 Compare August 27, 2020 16:47
@jonathanpeppers jonathanpeppers marked this pull request as ready for review August 27, 2020 21:38
@jonathanpeppers
Copy link
Member Author

There is a problem here, let me look into it:

(_BuildJava target) -> 
  Xamarin.Android.Common.targets(1933,3): error XAGCANSF7004: System.ArgumentException: An item with the same key has already been added. Key: [FSharp.Core.resources.dll, Xamarin.Android.Tasks.CompressedAssemblyInfo] [C:\a\1\s\external\Java.Interop\tools\java-source-utils\java-source-utils.csproj]
  C:\a\1\s\external\Java.Interop\tools\java-source-utils\Directory.Build.targets(10,5): error MSB3073: The command ""C:\a\1\s\external\Java.Interop\build-tools\gradle\gradlew" -d --stacktrace --no-daemon -PjavaSourceVer=1.8 -PjavaTargetVer=1.8 jar" exited with code -1. [C:\a\1\s\external\Java.Interop\tools\java-source-utils\java-source-utils.csproj]

    19 Warning(s)
    2 Error(s)

@jonathanpeppers jonathanpeppers marked this pull request as draft August 28, 2020 13:01
@jonathanpeppers jonathanpeppers marked this pull request as ready for review August 28, 2020 20:22
@jonathanpeppers
Copy link
Member Author

jonathanpeppers commented Aug 28, 2020

After more review, a build step is failing due to the %BUILD_SOURCEVERSIONMESSAGE% environment variable.

gradle prints this:

22:10:23.716 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Environment for process 'Gradle build daemon': {ISMONOBRANCH=False, AGENT_JOBSTATUS=Succeeded, XA_BUILD_WIN_POOL=VSEng-MicroBuildVS2019, NUGETARTIFACTNAME=nupkgs, VSTS_PROCESS_LOOKUP_ID=vsts_67c5d303-15f5-4278-9a76-8956d291f1a1, ALLUSERSPROFILE=C:\ProgramData, RESOURCES_TRIGGERINGALIAS=, SYSTEM_PIPELINESTARTTIME=2020-08-27 22:41:21+01:00, AGENT_VERSION=2.174.1, SYSTEM_TEAMFOUNDATIONCOLLECTIONURI=https://devdiv.visualstudio.com/, AGENT_WORKFOLDER=C:\a, AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN=true, MSDEPLOY_HTTP_USER_AGENT=VSTS_011b8bdf-6d56-4f87-be0d-0092136884d9_build_12278_0, SystemDrive=C:, NUNITCONSOLEVERSION=3.11.1, DIRNAME=C:\a\1\s\external\Java.Interop\build-tools\gradle\, USERNAME=AzDevOps, AZURE_HTTP_USER_AGENT=VSTS_011b8bdf-6d56-4f87-be0d-0092136884d9_build_12278_0, AGENT_OSARCHITECTURE=X64, SYSTEM_HOSTTYPE=build, CMD_LINE_ARGS=-d --stacktrace --no-daemon -PjavaSourceVer=1.8 -PjavaTargetVer=1.8 jar, ProgramFiles(x86)=C:\Program Files (x86), AZURE_CONTAINER_NAME=vsts-devdiv, COMMON_TESTRESULTSDIRECTORY=C:\a\1\TestResults, AKAMSOWNERALIASES=@('gourik';'hlong';'brbrot';'cadsit';'romoya';'admurp'), SYSTEM_TEAMPROJECTID=0bdbc590-a062-4c3f-b0f6-9383f67865ee, DEFAULT_JVM_OPTS="-Xmx64m", PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL, DriverData=C:\Windows\System32\Drivers\DriverData, SYSTEM_JOBTIMEOUT=360, SYSTEM_WORKFOLDER=C:\a, XA_FORCE_COMPONENT_REFRESH=, TASK_DISPLAYNAME=nunit Java.Interop Tests, HOSTEDMAC=Hosted Mac Internal, APPIDENTITYTOKEN=6d01b5e3-8536-41d6-9bef-c83ce0f164a9, PIPELINE_WORKSPACE=C:\a\1, SYSTEM_DEFINITIONID=12278, SYSTEM_JOBPARALLELISMTAG=Private, DOTNETCOREVERSION=3.1.201, MonAgentClientLocation=C:\Packages\Plugins\Microsoft.Azure.Geneva.GenevaMonitoring\2.18.0.2\Monitoring\Agent, PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 85 Stepping 4, GenuineIntel, SYSTEM_PHASEID=bd52e764-7bfb-54d9-0a45-bcddcda4442d, GITHUB_ARTIFACTS_CONTEXT=vsts-devdiv artifacts, AGENT_SERVEROMDIRECTORY=C:\agent\externals\vstsom, ProgramFiles=C:\Program Files, PUBLIC=C:\Users\Public, windir=C:\windows, _SKIP=2, BUILD_DEFINITIONNAME=Xamarin.Android-PR, BUILD_REPOSITORY_CLEAN=True, BUILD_REASON=Manual, RUNNINGONCI=true, SYSTEM_JOBPOSITIONINPHASE=1, SYSTEM_PHASEATTEMPT=1, BUILD_REPOSITORY_NAME=xamarin/xamarin-android, BUILD_STAGINGDIRECTORY=C:\a\1\a, JAVA_HOME=C:\Users\AzDevOps\android-toolchain\jdk-11, PROMPT=$P$G, SYSTEM_COLLECTIONID=011b8bdf-6d56-4f87-be0d-0092136884d9, AGENT_OS=Windows_NT, AGENT_HOMEDIRECTORY=C:\agent, SYSTEM_UNSAFEALLOWMULTILINESECRET=true, AGENT_MACHINENAME=xawin1d690000OL, DD_PROJ_GUID=0bdbc590-a062-4c3f-b0f6-9383f67865ee, MACMOJAVEBUILDPOOL=VSEng-Xamarin-RedmondMac-Android-Untrusted, SYSTEM_TASKDEFINITIONSURI=https://devdiv.visualstudio.com/, =C:=C:\a\1\s\external\Java.Interop\tools\java-source-utils, APPDATA=C:\Users\AzDevOps\AppData\Roaming, JAVA_EXE=C:\Users\AzDevOps\android-toolchain\jdk-11/bin/java.exe, XA_COMMERCIAL_BUILD=true, XA_JDK11_FOLDER=jdk-11, SYSTEM_STAGENAME=win_build_test, Path=C:\agent\externals\git\cmd;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\PowerShell\7\;C:\Program Files\Git\cmd;C:\Users\AzDevOps\AppData\Local\Microsoft\WindowsApps, [email protected], HOSTEDWINVS2019=Hosted Windows 2019 with VS2019, agent.jobstatus=Succeeded, BUILD_SOURCEVERSIONAUTHOR=Jonathan Peppers, COMPUTERNAME=xawin1d690000OL, SYSTEM_DEBUG=true, BUILD_QUEUEDBYID=4907eedc-8e83-6418-970d-be28acc2b5b2, ComSpec=C:\windows\system32\cmd.exe, APP_BASE_NAME=gradlew, SYSTEM_JOBDISPLAYNAME=Build and Smoke Test, AGENT_BUILDDIRECTORY=C:\a\1, AGENT_RETAINDEFAULTENCODING=false, SYSTEM_PHASENAME=win_build_test, NUMBER_OF_PROCESSORS=8, MSBUILDLOGIMPORTS=1, BUILD_REPOSITORY_PROVIDER=GitHub, BUILD_SOURCEBRANCHNAME=head, SIGNING_ENDPOINT=Xamarin Code Signing Jenkins, SYSTEM_SERVERTYPE=Hosted, BUILD_BUILDNUMBER=head-fb9e5d517d1358f06a4fa6f987402b934f1a6cbd-1, BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=True, SYSTEM=build, TESTASSEMBLIESARTIFACTNAME=test-assemblies, AGENT_ID=1326307, PROCESSOR_LEVEL=6, SYSTEM_JOBIDENTIFIER=win_build_test.win_build_test.__default, BUILD_BINARIESDIRECTORY=C:\a\1\b, SYSTEM_TOTALJOBSINPHASE=1, AGENT_NAME=xawin1d690000OL, BUILD_SOURCESDIRECTORY=C:\a\1\s, AGENT_TEMPDIRECTORY=C:\a\_temp, BUILD_BUILDURI=vstfs:///Build/Build/4018577, PROCESSOR_ARCHITECTURE=AMD64, RESOURCES_TRIGGERINGCATEGORY=, BUILD_REQUESTEDFOR=Jonathan Peppers, PSModulePath=C:\Users\AzDevOps\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\windows\system32\WindowsPowerShell\v1.0\Modules, SYSTEM_TASKINSTANCEID=ecb52562-b867-5d86-aae7-56a38766a1ce, MSBUILDTARGETOUTPUTLOGGING=true, SYSTEM_ARTIFACTSDIRECTORY=C:\a\1\a, SYSTEM_PULLREQUEST_ISFORK=False, SYSTEM_TASKINSTANCENAME=MSBuild5, SYSTEM_ENABLEACCESSTOKEN=SecretVariable, SYSTEM_TASKDISPLAYNAME=nunit Java.Interop Tests, APP_HOME=C:\a\1\s\external\Java.Interop\build-tools\gradle\, BUILD_DEFINITIONVERSION=21, SYSTEM_PHASEDISPLAYNAME=Build and Smoke Test, SYSTEM_TIMELINEID=47f03bdb-5e05-4556-92a5-501e5da556e3, AGENT_TOOLSDIRECTORY=C:\a\_tool, BUILD_SOURCEVERSIONMESSAGE=[One .NET] fixes for satellite assemblies
Context: https://github.com/xamarin/xamarin-android/pull/5040
In trying to enable our F# MSBuild tests, one fails with:
    GenerateCompressedAssembliesNativeSourceFiles
Xamarin.Android.Common.targets(1933,3): error XAGCANSF7004: System.ArgumentException: An item with the same key has already been added. Key: [FSharp.Core.resources.dll, Xamarin.Android.Tasks.CompressedAssemblyInfo] [C:\a\1\s\external\Java.Interop\tools\java-source-utils\java-source-utils.csproj]
C:\a\1\s\external\Java.Interop\tools\java-source-utils\Directory.Build.targets(10,5): error MSB3073: The command ""C:\a\1\s\external\Java.Interop\build-tools\gradle\gradlew" -d --stacktrace --no-daemon -PjavaSourceVer=1.8 -PjavaTargetVer=1.8 jar" exited with code -1. [C:\a\1\s\external\Java.Interop\tools\java-source-utils\java-source-utils.csproj]

And it causes the <Exec/> task to fail 🤦 .

@jonathanpeppers jonathanpeppers merged commit 307355f into dotnet:master Sep 1, 2020
@jonathanpeppers jonathanpeppers deleted the destinationsubdirectory branch September 1, 2020 13:05
jonathanpeppers added a commit to jonathanpeppers/java.interop that referenced this pull request Sep 3, 2020
Context: dotnet/android#5044 (comment)
Context: https://docs.microsoft.com/visualstudio/msbuild/exec-task

An env var can break the build such as running the following powershell:

    > $env:FOO="`nFoo.cs(123,1) error: oh dear!"

A `\n` new line in the string is required to hit an issue.

This results in a confusing build error:

    (_BuildJava target) ->
        EXEC : Foo.cs(123,1) error : oh dear!

`gradlew -d` emits every environment variable, and if there is a line
that looks like an MSBuild error it will emit the error. We can set
`IgnoreStandardErrorWarningFormat="true"` to disable this behavior.

This could commonly happen on CI systems, if a commit message contains
an error message.

In the case of Azure DevOps, `%BUILD_SOURCEVERSIONMESSAGE%` will
contain the full commit message.

I also fixed the `_CleanJava` target that was not running at all.
`java-source-utils.csproj` now properly cleans now.
jonpryor pushed a commit to dotnet/java-interop that referenced this pull request Sep 8, 2020
Context: dotnet/android#5044 (comment)
Context: https://docs.microsoft.com/visualstudio/msbuild/exec-task

An env var can break the build such as running the following powershell:

	> $env:FOO="`nFoo.cs(123,1) error: oh dear!"

A new line in the string is required to hit an issue.  (```n`` is
PowerShell-ese for a `\n` newline.)

This results in a confusing build error:

	(_BuildJava target) ->
	    EXEC : Foo.cs(123,1) error : oh dear!

`gradlew -d` prints every environment variable, and if there is a line
that looks like an MSBuild error then `msbuild` will emit the error.
We can set `IgnoreStandardErrorWarningFormat="true"` to disable this.

This could commonly happen on CI systems, if a commit message contains
an error message.

In the case of Azure DevOps, `%BUILD_SOURCEVERSIONMESSAGE%` will
contain the full commit message.

I also fixed the `_CleanJava` target that was not running at all.
`java-source-utils.csproj` now properly cleans now.
@github-actions github-actions bot locked and limited conversation to collaborators Jan 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants