Skip to content

Conversation

@grendello
Copy link
Contributor

@grendello grendello commented Jun 3, 2020

Context: #4914
Context: d583b7c

The .NET5 version of the Mono runtime "hijacked" __Internal for its
own purposes and the change causes us trouble when trying to resolve
p/invokes from our managed code to our runtime. Instead we now use
xa-internal-api (added in d583b7c) to specify the p/invoke library
name.

@grendello grendello requested a review from jonpryor June 3, 2020 08:21
@garuma
Copy link
Contributor

garuma commented Jun 4, 2020

Our code will not directly address those p/invoke so the change should be fine for us. Unless there is a specific concern I'm not seeing?

@grendello
Copy link
Contributor Author

@garuma that's what I thought, but seeing so many Designer tests fail I thought the changes here broke them somehow, thanks!

@jonpryor
Copy link
Contributor

jonpryor commented Jun 4, 2020

@garuma
Copy link
Contributor

garuma commented Jun 4, 2020

Looks like it's not resolving monodroid when run on desktop:

[2020-06-04 19:19:38.4] Renderer >> 4 [monodroid] Calling into managed runtime init
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >> Unhandled Exception:
[2020-06-04 19:19:38.6] Renderer (error) >> System.DllNotFoundException: monodroid assembly:<unknown assembly> type:<unknown type> member:(null)
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at (wrapper managed-to-native) Android.Runtime.AndroidObjectReferenceManager._monodroid_gref_get()
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at Android.Runtime.AndroidObjectReferenceManager.get_GlobalReferenceCount () [0x00000] in /Users/builder/azdo/_work/2/s/xamarin-android/src/Mono.Android/Android.Runtime/AndroidRuntime.cs:107 
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at Java.Interop.JniRuntime+JniObjectReferenceManager.CreateGlobalReference (Java.Interop.JniObjectReference reference) [0x00011] in <b3fb8b2fd70944739118ed66394c3cef>:0 
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at Android.Runtime.AndroidObjectReferenceManager.CreateGlobalReference (Java.Interop.JniObjectReference value) [0x00000] in /Users/builder/azdo/_work/2/s/xamarin-android/src/Mono.Android/Android.Runtime/AndroidRuntime.cs:169 
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at Java.Interop.JniObjectReference.NewGlobalRef () [0x0000a] in <b3fb8b2fd70944739118ed66394c3cef>:0 
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at Java.Interop.JniRuntime..ctor (Java.Interop.JniRuntime+CreationOptions options) [0x001c4] in <b3fb8b2fd70944739118ed66394c3cef>:0 
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at Android.Runtime.AndroidRuntime..ctor (System.IntPtr jnienv, System.IntPtr vm, System.Boolean allocNewObjectSupported, System.IntPtr classLoader, System.IntPtr classLoader_loadClass, System.Boolean jniAddNativeMethodRegistrationAttributePresent) [0x00000] in /Users/builder/azdo/_work/2/s/xamarin-android/src/Mono.Android/Android.Runtime/AndroidRuntime.cs:25 
[2020-06-04 19:19:38.6] Renderer (error) >> 
[2020-06-04 19:19:38.6] Renderer (error) >>   at Android.Runtime.JNIEnv.Initialize (Android.Runtime.JnienvInitializeArgs* args) [0x000f8] in /Users/builder/azdo/_work/2/s/xamarin-android/src/Mono.Android/Android.Runtime/JNIEnv.cs:177 

@grendello
Copy link
Contributor Author

Looks like it's not resolving monodroid when run on desktop:

That's because because the XA runtime is named libmono-android.{debug,release}.{dll,dylib,so} on desktop... ugh

@grendello
Copy link
Contributor Author

/azp run

@grendello grendello marked this pull request as ready for review June 5, 2020 18:41
@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@grendello grendello changed the title [WIP] See if we can drop __Internal Stop using __Internal in our p/invoke calls Jun 5, 2020
@grendello
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@jonpryor jonpryor added the do-not-merge PR should not be merged. label Jun 5, 2020
@jonpryor
Copy link
Contributor

jonpryor commented Jun 5, 2020

Can't merge until designer-side changes are in place.

TODO: link to designer-side changes?

Copy link
Member

@jonathanpeppers jonathanpeppers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove these:

https://github.com/xamarin/xamarin-android/blob/d4e0738500fe6e8d5c46fcf6703b703873be577f/tests/MSBuildDeviceIntegration/Tests/XASdkDeployTests.cs#L59-L61

If this test passes with these three lines removed, I think it will be fixed on the .NET 5+ side.

@grendello grendello force-pushed the drop-internal branch 2 times, most recently from 455fb4d to e29a0e2 Compare August 19, 2020 19:10
@jonpryor
Copy link
Contributor

jonpryor commented Aug 24, 2020

@grendello grendello force-pushed the drop-internal branch 2 times, most recently from 78fa65f to 0ec1c85 Compare August 31, 2020 12:51
@grendello
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@grendello grendello removed the do-not-merge PR should not be merged. label Sep 1, 2020
@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@jonpryor
Copy link
Contributor

jonpryor commented Sep 3, 2020

The macOS Designer (and Windows Designer) tests fail:

[2020-09-02 13:54:27.1] Renderer (error) >> Unhandled Exception:
[2020-09-02 13:54:27.1] Renderer (error) >> System.DllNotFoundException: xa-internal-api assembly:<unknown assembly> type:<unknown type> member:(null)
[2020-09-02 13:54:27.1] Renderer (error) >>   at (wrapper managed-to-native) Android.Runtime.AndroidObjectReferenceManager._monodroid_gref_get()
[2020-09-02 13:54:27.1] Renderer (error) >>   at Android.Runtime.AndroidObjectReferenceManager.get_GlobalReferenceCount () [0x00000] in /Users/builder/azdo/_work/4/s/xamarin-android/src/Mono.Android/Android.Runtime/AndroidRuntime.cs:109 
[2020-09-02 13:54:27.1] Renderer (error) >>   at Java.Interop.JniRuntime+JniObjectReferenceManager.CreateGlobalReference (Java.Interop.JniObjectReference reference) [0x00011] in <83d07bcea5334da9adfcd55af65209e9>:0 
[2020-09-02 13:54:27.1] Renderer (error) >>   at Android.Runtime.AndroidObjectReferenceManager.CreateGlobalReference (Java.Interop.JniObjectReference value) [0x00000] in /Users/builder/azdo/_work/4/s/xamarin-android/src/Mono.Android/Android.Runtime/AndroidRuntime.cs:171 
[2020-09-02 13:54:27.1] Renderer (error) >>   at Java.Interop.JniObjectReference.NewGlobalRef () [0x0000a] in <83d07bcea5334da9adfcd55af65209e9>:0 
[2020-09-02 13:54:27.1] Renderer (error) >>   at Java.Interop.JniRuntime..ctor (Java.Interop.JniRuntime+CreationOptions options) [0x001c4] in <83d07bcea5334da9adfcd55af65209e9>:0 
[2020-09-02 13:54:27.1] Renderer (error) >>   at Android.Runtime.AndroidRuntime..ctor (System.IntPtr jnienv, System.IntPtr vm, System.Boolean allocNewObjectSupported, System.IntPtr classLoader, System.IntPtr classLoader_loadClass, System.Boolean jniAddNativeMethodRegistrationAttributePresent) [0x00000] in /Users/builder/azdo/_work/4/s/xamarin-android/src/Mono.Android/Android.Runtime/AndroidRuntime.cs:27 
[2020-09-02 13:54:27.1] Renderer (error) >>   at Android.Runtime.JNIEnv.Initialize (Android.Runtime.JnienvInitializeArgs* args) [0x000f8] in /Users/builder/azdo/_work/4/s/xamarin-android/src/Mono.Android/Android.Runtime/JNIEnv.cs:177 

@grendello
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@jonpryor
Copy link
Contributor

jonpryor commented Sep 9, 2020

Windows designer integration tests are failing:

[2020-09-07 13:18:07.1] Renderer >> 5 [monodroid] MonodroidRuntime::monodroid_dlopen ("C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v9.0\xa-internal-api", 0, 000000001dc4c1a0, 0000000000000000)
[2020-09-07 13:18:07.1] Renderer >> 7 [monodroid] Internal API library is required
[2020-09-07 13:18:07.2] ERROR: : ReadMessage failed
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at Xamarin.AndroidDesigner.Java.JavaCommunicationContext.ReadMessageLoop() in D:\a\1\s\designer\Xamarin.Designer.Android\Xamarin.AndroidDesigner\Java\JavaCommunicationContext.cs:line 237
[2020-09-07 13:18:07.3] ERROR: : Layout inflation failed
…

The Internal API library is required message is from MonodroidRuntime::init_internal_api_dso(): https://github.com/xamarin/xamarin-android/blob/030442e6aeaf51f42309a2433ef05f06a8b7fd8d/src/monodroid/jni/monodroid-glue.cc#L1086

Thus, the problem is that libxa-internal-api.dll isn't being loaded on Windows. :-(

@grendello grendello force-pushed the drop-internal branch 8 times, most recently from 486ce01 to 30ced74 Compare September 16, 2020 07:14
bool name_needs_free = false;
/* name is nullptr when we're P/Invoking __Internal, so remap to libxa-internal-api */
if (name == nullptr) {
if (name == nullptr || strstr (name, "xa-internal-api") != nullptr) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this instead be strstr (name, MONODROID_PATH_SEPARATOR "xa-internal-api") != nullptr, ensuring that we get e.g. /xa-internal-api?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather not do that. We can't be sure we'll always be getting a path here. Right now we do get a path (which points to a location where there's no libraries we ship) but it may change and I'd rather be flexible.

const char *last_sep = strrchr (the_path, MONODROID_PATH_SEPARATOR_CHAR);
if (last_sep != nullptr) {
tmp_name = utils.strdup_new (the_path, last_sep - the_path);
tmp_name = utils.string_concat (tmp_name, MONODROID_PATH_SEPARATOR, API_DSO_NAME);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a memory leak. The preceding line utils.strdup_new() does a new char[…], and utils.string_concat() also returns a new char[…] without a delete[] s1. Consequently, this sequence of statements "loses" the first new char[…] held by tmp_name, causing it to leak memory.

name_needs_free = true;
}
}
#else
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be #else // ndef WINDOWS

// Next lets try the location of the XA runtime DLL, libxa-internal-api.dll should be next to it.
HMODULE hm;
const void *func = reinterpret_cast<const void*>(&Java_mono_android_Runtime_initInternal);
if (GetModuleHandleEx (GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, static_cast<LPCSTR>(func), &hm) == 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like MonodroidRuntime::get_my_location()? Perhaps that should be used instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

heh, forgot that we had that one :)

// First try to see if it exist at the path pointed to by `name`. With p/invokes, currently (Sep 2020), we can't
// really trust the path since it consists of *some* directory path + p/invoke library name and it does not
// point to the location where the native library is. However, we still need to check the location first, should
// it point to the right place in the future.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO (fix after current PR build completes): to link to: mono/mono#20295 (comment)

bool found = probe_dll_at (name);
if (!found) {
// Next lets try the location of the XA runtime DLL, libxa-internal-api.dll should be next to it.
const char *path = get_my_location ();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Silly question: why does get_my_location() return a const char* when we're supposed to free() it? Isn't convention to return a non-const type for values which should be freed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const char* doesn't mean it cannot be free'd, it means it {should,must} not be modified. Pointers can point to r/o (by convention) data after all..

const char *path = get_my_location ();
found = probe_dll_at (path);
if (path != nullptr) {
free (reinterpret_cast<void*>(const_cast<char*>(path)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…cause this is just bizarre.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is, but we have a no-warnings policy, so... :)

Context: dotnet#4914
Context: dotnet@d583b7c

The .NET5 version of the Mono runtime "hijacked" `__Internal` for its
own purposes and the change causes us trouble when trying to resolve
p/invokes from our managed code to our runtime.  Instead we now use
`xa-internal-api` (added in d583b7c) to specify the p/invoke library
name.
@jonpryor
Copy link
Contributor

jonpryor commented Sep 17, 2020

Summary:

[Mono.Android] Don't use `__Internal` in our p/invoke calls (#4757)

Content:

Context: https://github.com/xamarin/xamarin-android/pull/4914
Context: d583b7c215fe9be2f732b4a52423be0aea7f0809

Context: https://github.com/mono/mono/issues/20295#issuecomment-679271621

Context? https://github.com/mono/mono/pull/17369
Context? https://github.com/mono/mono/commit/d5b6cf33b3d2a89189be4e6bddcf0207e9887ecd

As a runtime extension, Mono long supported `[DllImport("__Internal")]`
as a synonym for [`dlopen(NULL, 0)`][0].  Unfortunately, this
[did not work on Android][1] -- Bionic would SIGSEGV when the
filename parameter was `NULL` -- and as "fallout" of this change the
`loader_func` that Xamarin.Android registers with mono via
`mono_dl_fallback_register()` would be invoked *instead of*
`dlopen(NULL)` when loading `__Internal`…but *only* on Android.

Use of this feature remains widely used in Xamarin.Android, e.g.
[`java-interop` is mapped to `__Internal`][2], avoiding the need to
build and ship a separate `libjava-interop.so` library.

Unfortunately, as part of .NET 5, MonoVM will be overhauling the
semantics of `[DllImport("__Internal")]`, preventing `loader_func`
from being invoked on Android to resolve `__Internal`.

Consequently, we need to begin moving away from use of `__Internal`.

Begin this process, and remove all P/Invokes to `__Internal` within
`Mono.Android.dll` with P/Invokes to `xa-internal-api` (d583b7c2).

[0]: https://linux.die.net/man/3/dlopen
[1]: https://github.com/mono/mono/commit/9b73cd9c33507ba27829728ea82d10d98335e1d6
[2]: https://github.com/xamarin/xamarin-android/blob/faf1d16692ead1115d34fdf7b84c256c94a321da/src/monodroid/config.xml#L2

@jonpryor jonpryor merged commit 39576f2 into dotnet:master Sep 17, 2020
@grendello grendello deleted the drop-internal branch September 17, 2020 16:02
@github-actions github-actions bot locked and limited conversation to collaborators Jan 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants