Binding fails with Visual Studio for Mac 8.8.6.15 and above

[akalghatgi]

Stuck with compiling binding with no information binding failure

Binding was generated correctly with older versions but after upgrading to 8.8.6.15 bindings few of the classes are missing.
When Class-parser is selected very few warnings, no exact error/warning for which class binding is not generated.
If we select jar2xml most of the errors/warning are related to Kotlin.

When Class-parser is selected following are the warning. and important classes are missing from generated binding.

obj/Release/api.xml(433, 8) : warning BG8401: Skipping 'Com.vvvvvvv.xxxxxxxx.Http.HttpRequest.Companion' due to a duplicate nested type name. (Java type: 'com.vvvvvvv.xxxxxxxx.http.HttpRequest')
obj/Release/api.xml(496, 8) : warning BG8401: Skipping 'Com.vvvvvvv.xxxxxxxx.Logger.LoggerLevel.Companion' due to a duplicate nested type name. (Java type: 'com.vvvvvvv.xxxxxxxx.logger.LoggerLevel')

When jar2xml is selected following are the warnings but few extra classes are generated in binding.

/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Android/Xamarin.Android.Bindings.targets(83,5): warning XA4231: The Android class parser value 'jar2xml' is deprecated and will be removed in a future version of Xamarin.Android. Update the project properties to use 'class-parse'.
JARTOXML : OpenJDK 64-Bit Server VM warning : ignoring option UseSplitVerifier; support was removed in 8.0
obj/Release/api.xml(156, 2) : warning BG8800: Unknown parameter type 'kotlin.jvm.internal.DefaultConstructorMarker' for member 'Com.vvvvv.xxxxxxxx.Configuration.Configuration (char[], string, string, char[], string, char[], int, kotlin.jvm.internal.DefaultConstructorMarker)'.
obj/Release/api.xml(254, 2) : warning BG8800: Unknown parameter type 'kotlin.jvm.internal.DefaultConstructorMarker' for member 'Com.vvvvv.xxxxxxxx.ContextDelegate.Companion.Companion (kotlin.jvm.internal.DefaultConstructorMarker)'.
obj/Release/api.xml(554, 2) : warning BG8700: Unknown return type 'kotlinx.serialization.KSerializer<com.vvvvv.xxxxxxxx.http.HttpRequest>' for member 'Com.vvvvv.xxxxxxxx.Http.HttpRequest.Companion.Serializer ()'.
obj/Release/api.xml(551, 2) : warning BG8800: Unknown parameter type 'kotlin.jvm.internal.DefaultConstructorMarker' for member 'Com.vvvvv.xxxxxxxx.Http.HttpRequest.Companion.Companion (kotlin.jvm.internal.DefaultConstructorMarker)'.
obj/Release/api.xml(515, 2) : warning BG8800: Unknown parameter type 'kotlinx.serialization.CompositeEncoder' for member 'Com.vvvvv.xxxxxxxx.Http.HttpRequest.Write_Self (Com.vvvvv.xxxxxxxx.Http.HttpRequest, kotlinx.serialization.CompositeEncoder, kotlinx.serialization.SerialDescriptor)'.
obj/Release/api.xml(465, 2) : warning BG8800: Unknown parameter type 'kotlinx.serialization.SerializationConstructorMarker' for member 'Com.vvvvv.xxxxxxxx.Http.HttpRequest.HttpRequest (int, string, string, string[], int, string[], string, kotlinx.serialization.SerializationConstructorMarker)'.
obj/Release/api.xml(601, 2) : warning BG8800: Unknown parameter type 'kotlin.jvm.internal.DefaultConstructorMarker' for member 'Com.vvvvv.xxxxxxxx.Http.PublicKeyPinningTrustManager.PublicKeyPinningTrustManager (string[], int, int, kotlin.jvm.internal.DefaultConstructorMarker)'.
obj/Release/api.xml(745, 2) : warning BG8800: Unknown parameter type 'kotlin.jvm.internal.DefaultConstructorMarker' for member 'Com.vvvvv.xxxxxxxx.Logger.LoggerLevel.Companion.Companion (kotlin.jvm.internal.DefaultConstructorMarker)'.
obj/Release/api.xml(520, 2) : warning BG8401: Skipping 'Com.vvvvv.xxxxxxxx.Http.HttpRequest.Companion' due to a duplicate nested type name. (Java type: 'com.vvvvv.xxxxxxxx.http.HttpRequest')
obj/Release/api.xml(732, 2) : warning BG8401: Skipping 'Com.vvvvv.xxxxxxxx.Logger.LoggerLevel.Companion' due to a duplicate nested type name. (Java type: 'com.vvvvv.xxxxxxxx.logger.LoggerLevel') 

Expected Behavior

As the bindings are generated with class-parser there is should be proper errors/warning so that handling can be done properly for getting desired classes in binding.

Actual Behavior

No error/warnings for not generating binding for the classes.

Version Information

=== Visual Studio Community 2019 for Mac (Preview) ===

Version 8.9 Preview (8.9 build 1451)
Installation UUID: 7a99ac0f-2574-48b1-b44d-6ad8eabcabff
GTK+ 2.24.23 (Raleigh theme)
Xamarin.Mac 6.18.0.23 (d16-6 / 088c73638)

Package version: 612000113

=== Mono Framework MDK ===

Runtime:
Mono 6.12.0.113 (2020-02/4fdfb5b1fd5) (64-bit)
Package version: 612000113

=== Roslyn (Language Service) ===

3.9.0-3.20619.14+df59a33fd9beff9790e01a2a1ab21e4a1e6921b3

=== NuGet ===

Version: 5.8.0.6860

=== .NET Core SDK ===

SDK: /usr/local/share/dotnet/sdk/5.0.102/Sdks
SDK Versions:
5.0.102
5.0.101
5.0.100
3.1.405
3.1.404
3.1.403
3.1.402
2.1.811
2.1.809
MSBuild SDKs: /Applications/Visual Studio.app/Contents/Resources/lib/monodevelop/bin/MSBuild/Current/bin/Sdks

=== .NET Core Runtime ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
5.0.2
5.0.1
5.0.0
5.0.0-rc.2.20475.5
3.1.11
3.1.10
3.1.9
3.1.8
2.1.23
2.1.22
2.1.21

=== .NET Core 3.1 SDK ===

SDK: 3.1.405

=== Xamarin.Profiler ===

Version: 1.6.15.68
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Updater ===

Version: 11

=== Xamarin.Android ===

Version: 11.2.0.0 (Visual Studio Community)
Commit: xamarin-android/d16-9/f908d16
Android SDK: /Users/akalghatgi@xxxxxx.com/Library/Developer/Xamarin/android-sdk-macosx
Supported Android versions:
8.1 (API level 27)

SDK Tools Version: 26.1.1
SDK Platform Tools Version: 30.0.4
SDK Build Tools Version: 30.0.2

Build Information:
Mono: 5e9cb6d
Java.Interop: xamarin/java.interop/d16-9@1d382be
ProGuard: Guardsquare/proguard@912d149
SQLite: xamarin/sqlite@cfe06e0
Xamarin.Android Tools: xamarin/xamarin-android-tools/main@ad80a42

=== Microsoft OpenJDK for Mobile ===

Java SDK: /Users/akalghatgi@xxxxxx.com/Library/Developer/Xamarin/jdk/microsoft_dist_openjdk_1.8.0.25
1.8.0-25
Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Android SDK Manager ===

Version: 16.9.0.21
Hash: 57e40ba
Branch: remotes/origin/main
Build date: 2021-01-08 01:57:14 UTC

=== Android Device Manager ===

Version: 16.9.0.14
Hash: 0fdccda
Branch: remotes/origin/main
Build date: 2021-01-08 01:57:36 UTC

=== Xamarin Designer ===

Version: 16.9.0.266
Hash: c4842c761
Branch: remotes/origin/c4842c761b9b6a95407f72278ca7fb42f8f7fdf0
Build date: 2021-01-07 06:17:08 UTC

=== Apple Developer Tools ===

Xcode 12.1 (17222)
Build 12A7403

=== Xamarin.Mac ===

Version: 7.3.0.27 (Visual Studio Community)
Hash: f4c9327fa
Branch: main
Build date: 2020-11-19 10:57:31-0500

=== Xamarin.iOS ===

Version: 14.9.0.27 (Visual Studio Community)
Hash: f4c9327fa
Branch: main
Build date: 2020-11-19 10:57:31-0500

=== Build Information ===

Release ID: 809001451
Git revision: cfd15313a6388ef8dada0182e22a058131c46f9d
Build date: 2021-01-15 08:42:21-05
Build branch: release-8.9
Xamarin extensions: cfd15313a6388ef8dada0182e22a058131c46f9d

=== Operating System ===

Mac OS X 10.15.7
Darwin 19.6.0 Darwin Kernel Version 19.6.0
Thu Oct 29 22:56:45 PDT 2020
root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64

@jonathanpeppers needed your guidance on this.

[jpobst]

jar2xml is deprecated and will be removed in .NET6. It does not have the support we wrote for Kotlin in it, so it's not going to produce very good bindings to Kotlin libraries.

Most of the messages telling you why classes are not bound are not warnings, they are simply messages in the build log. You'll want to turn on Diagnostic build logging and then Rebuild (not just Build) your solution.

If you look in the Output Log you should see messages like:

Kotlin: Hiding internal class my.InternalClass

Which will tell you why types are not being bound.

[akalghatgi]

@jpobst here is a part of Diagnostic logs

 Error while processing '[Method] java.time.temporal.Temporal adjustInto(java.time.temporal.Temporal temporal, long newValue)' in '[Class] java.time.temporal.ChronoField': Type 'java.time.temporal.Temporal' was not found.
    Error while processing '[Method] java.time.temporal.Temporal addTo(java.time.temporal.Temporal temporal, long amount)' in '[Class] java.time.temporal.ChronoUnit': Type 'java.time.temporal.Temporal' was not found.
    Error while processing '[Constructor] ContextDelegate.Companion(kotlin.jvm.internal.DefaultConstructorMarker $constructor_marker)' in '[Class] com.vvvvvv.xxxxxx.ContextDelegate.Companion': Type 'kotlin.jvm.internal.DefaultConstructorMarker' was not found.
    Error while processing '[Constructor] HttpRequest(int seen1, java.lang.String url, java.lang.String method, java.lang.String[] pinnedPublicKeys, int pinnedPublicKeyIndex, java.lang.String[] headers, java.lang.String body, kotlinx.serialization.SerializationConstructorMarker serializationConstructorMarker)' in '[Class] com.vvvvvv.xxxxxx.http.HttpRequest': Type 'kotlinx.serialization.SerializationConstructorMarker' was not found.
    Error while processing type '[Class] com.vvvvvv.xxxxxx.http.HttpRequest..serializer': Type 'kotlinx.serialization.internal.GeneratedSerializer' was not found.
    Error while processing '[Method] kotlinx.serialization.KSerializer<com.vvvvvv.xxxxxx.http.HttpRequest> serializer()' in '[Class] com.vvvvvv.xxxxxx.http.HttpRequest.Companion': Type 'kotlinx.serialization.KSerializer' was not found.
    Error while processing '[Constructor] HttpRequest.Companion(kotlin.jvm.internal.DefaultConstructorMarker $constructor_marker)' in '[Class] com.vvvvvv.xxxxxx.http.HttpRequest.Companion': Type 'kotlin.jvm.internal.DefaultConstructorMarker' was not found.
    Error while processing '[Constructor] LoggerLevel.Companion(kotlin.jvm.internal.DefaultConstructorMarker $constructor_marker)' in '[Class] com.vvvvvv.xxxxxx.logger.LoggerLevel.Companion': Type 'kotlin.jvm.internal.DefaultConstructorMarker' was not found.
  Done executing task "BindingsGenerator".

[jpobst]

Are those the class-parse or jar2xml logs? What are the types that you are missing from the binding? If you search the log for those type names, do you find anything?

[akalghatgi]

It is with class-parser.. I don't see any other error. attaching the logs for your reference.
BindingLogs.txt

[akalghatgi]

Type ,missing are HttpFunction, HttpRequest$Serializer etc...

[jpobst]

From the logs, we hid HttpFunction because it is an internal class in Kotlin:

Kotlin: Hiding internal class com/vmware/chameleon/http/HttpFunction

It looks like HttpRequest$Serializer caused a problem with our parser:

class-parse: method com/vmware/chameleon/http/HttpRequest$$serializer.patch(Lkotlinx/serialization/Decoder;Lcom/vmware/chameleon/http/HttpRequest;)Lcom/vmware/chameleon/http/HttpRequest;: Local variables array has 3 entries ('LocalVariableTableAttribute(LocalVariableTableEntry(Name='this', Descriptor='Lkotlinx/serialization/KSerializer;', StartPC=0, Index=0), LocalVariableTableEntry(Name='decoder', Descriptor='Lkotlinx/serialization/Decoder;', StartPC=0, Index=1), LocalVariableTableEntry(Name='old', Descriptor='Lcom/vmware/chameleon/http/HttpRequest;', StartPC=0, Index=2))'); descriptor has 2 entries!\
class-parse: method com/vmware/chameleon/http/HttpRequest$$serializer.patch(Lkotlinx/serialization/Decoder;Lcom/vmware/chameleon/http/HttpRequest;)Lcom/vmware/chameleon/http/HttpRequest;: Local variable type descriptor mismatch! Got 'Lkotlinx/serialization/Decoder;'; expected 'Lkotlinx/serialization/KSerializer;'.\
class-parse: method com/vmware/chameleon/http/HttpRequest$$serializer.patch(Lkotlinx/serialization/Decoder;Lcom/vmware/chameleon/http/HttpRequest;)Lcom/vmware/chameleon/http/HttpRequest;: Local variable type descriptor mismatch! Got 'Lcom/vmware/chameleon/http/HttpRequest;'; expected 'Lkotlinx/serialization/Decoder;'.\

If you can provide the .jar file we can take a look at what is causing that. But it also looks like it may be a private/internal class, in which case it won't be bound even if that gets fixed.

[akalghatgi]

chameleon-android-1.1.0.zip
here you go... rename it to .aar

[jpobst]

Filed the HttpRequest$Serializer issue as dotnet/java-interop#788.

[jonpryor]

When jar2xml is selected following are the warnings but few extra classes are generated in binding.

What "extra classes" are present in the binding when $(AndroidClassParser)=jar2xml? When I create a new binding project that only binds the classes.zip at dotnet/java-interop#788 (which is the classes.jar from chameleon-android-1.1.0.zip), with a Build action of @(EmbeddedJar), more types are present in the binding when class-parse is used vs jar2xml:

% msbuild /p:AndroidClassParser=jar2xml /v:diag > b-j2x.txt
% mv obj obj.j2x
% mv bin bin.j2x

% msbuild /p:AndroidClassParser=class-parse /v:diag > b-cp.txt 
% mv bin bin.cp
% mv obj obj.cp

% find obj.j2x -name \*.cs | wc -l 
      15
% find obj.cp -name \*.cs | wc -l
      17

More C# files are generated when class-parse is used (17 vs 15).

% monodis --typedef bin.j2x/Debug/Scratch.Gji788.dll| wc -l
      25
% monodis --typedef bin.cp/Debug/Scratch.Gji788.dll| wc -l
      30

More types are present when using class-parse (30 vs 25).

The added types?

% monodis --typedef bin.cp/Debug/Scratch.Gji788.dll | awk '{print $2}' | sort > cp.txt
% monodis --typedef bin.j2x/Debug/Scratch.Gji788.dll | awk '{print $2}' | sort > j2x.txt
% diff -u j2x.txt cp.txt

--- j2x.txt	2021-02-02 20:42:57.000000000 -0500
+++ cp.txt	2021-02-02 20:42:46.000000000 -0500
@@ -3,6 +3,7 @@
 Com.Vmware.Chameleon.BuildConfig
 Com.Vmware.Chameleon.ChameleonApplication
 Com.Vmware.Chameleon.ChameleonContextKt
+Com.Vmware.Chameleon.Configuration
 Com.Vmware.Chameleon.ContextDelegate
 Com.Vmware.Chameleon.Function.FunctionArg
 Com.Vmware.Chameleon.Function.FunctionOutput
@@ -10,11 +11,15 @@
 Com.Vmware.Chameleon.Function.IFunctionFactory
 Com.Vmware.Chameleon.Function.IFunctionFactoryInvoker
 Com.Vmware.Chameleon.Function.IFunctionInvoker
-Com.Vmware.Chameleon.Http.HttpResponse
+Com.Vmware.Chameleon.Http.HttpRequest
+Com.Vmware.Chameleon.Http.HttpRequest/Companion
+Com.Vmware.Chameleon.IChameleonContext
+Com.Vmware.Chameleon.IChameleonContextInvoker
 Com.Vmware.Chameleon.Logger.ICustomLogger
 Com.Vmware.Chameleon.Logger.ICustomLoggerInvoker
 Com.Vmware.Chameleon.Logger.LoggerKt
 Com.Vmware.Chameleon.Logger.LoggerLevel
+Com.Vmware.Chameleon.Logger.LoggerLevel/Companion
 Java.Interop.__TypeRegistrations
 Table
 _JniMarshal_PPLLZ_L

While more types are generated, this does show an interesting absence, which may be what you're after: Com.Vmware.Chameleon.Http.HttpResponse is present when using jar2xml but not class-parse.

Why?

  Kotlin: Hiding internal class com/vmware/chameleon/http/HttpResponse (TaskId:35)

It's a public class as far as Java and the JVM are concerned, but Kotlin wishes it to be an internal type, and our binding system attempts to "appropriately bind". Thus, HttpResponse isn't bound.

[jonpryor]

As the bindings are generated with class-parser there is should be proper errors/warning so that handling can be done properly for getting desired classes in binding.

I'm not entirely sure I understand what is being requested.

An interpretation is that all of the "Error" message emitted, such as those mentioned earlier -- which are currently treated as debug messages, not errors, as they do not appear as MSBuild warnings or errors -- should be displayed as MSBuild warnings or errors.

I am amenable to that suggestion. @jpobst would need to implement it. ;-)

Additionally, we do emit warnings, but you don't appear to like the ones emitted:

obj/Release/api.xml(433, 8) : warning BG8401: Skipping 'Com.vvvvvvv.xxxxxxxx.Http.HttpRequest.Companion' due to a duplicate nested type name. (Java type: 'com.vvvvvvv.xxxxxxxx.http.HttpRequest')

We should improve this warning so that it's actionable. Currently, it isn't directly actionable. A workaround is to rename the field, in Metadata.xml:

  <attr path="//class[@name='HttpRequest']/field[@name='Companion']" name="managedName">CompanionField</attr>

This removes the previous BG8401 warning, and updates the HttpRequest binding to now contain an HttpRequest.CompanionField field.