`Object::cast_to` works incorrectly for GDExtension custom classes

[VanessB]

Copy of this issue#71332 from main repository, as it is related to GDExtension.

Issue description

Object::cast_to does not work properly in case of custom GDExtension classes. For example, a nullptr is NOT returned in some cases where an object can not be casted to the desired type:

(Let TestNode inherit from Node2D, some_node_2d be of type Node2D)

godot::Node::cast_to<TestNode>(some_node_2d) != nullptr;  // true, must be false, as Node2D can not be casted to TestNode.

From my experience, it seems that cast_to works with custom classes as if they are their base classes, as (in this example) I can cast all the classes inherited from Node2D to TestNode, and can not cast any other type (see steps to reproduce and reproduction project).

The GDScript, although, works fine with casting of custom classes, see example. C++ code for casting in-engine types also seems to work fine.

Steps to reproduce and example project with GDExtension source code can be seen here (issue#71332)

[dsnopek]

I was also caught by this in trying to port SG Physics 2D. Here's a PR that fixes it in my testing:

#1037

[akien-mga]

Reopening as #1037 was reverted.

[zhehangd]

I saw the very helpful background information provided by @dsnopek in #1042 , and did my own investigation tonight and would like to add some more.

Basically we have no control over which godot-cpp wrapper gets bound to an engine object. It all depends on who comes first that references this object in the extension (who calls object_get_instance_binding or object_set_instance and provides ___binding_callbacks first).

Here are three examples:

1. Our extension memnew a godot::PackedScene. The wrapper will forever be this godot::PackedScene.
   

   godot-cpp/src/classes/wrapped.cpp

   Line 46 in f5133c0

   godot::internal::gde_interface->object_set_instance(_owner, reinterpret_cast<GDExtensionConstStringNamePtr>(extension_class), this);

2. A PackedScene is initially passed to a function as a parameter in form of a Variant -- the wrapper will forever be a godot::Object.
   

   godot-cpp/src/variant/variant.cpp

   Line 414 in f5133c0

   return reinterpret_cast<Object *>(internal::gde_interface->object_get_instance_binding(obj, internal::token, &Object::___binding_callbacks));

• A PackedScene is initially passed to a function as a parameter in form of godot::PackedScene* or Ref<godot::PackedScene> -- the wrapper will forever be a godot::PackedScene.

(Note that these two places are buggy in the master branch so I am linking to my pending PRs.

godot-cpp/include/godot_cpp/core/method_ptrcall.hpp

Line 172 in 891bcca

return reinterpret_cast<T *>(godot::internal::gde_interface->object_get_instance_binding(

godot-cpp/include/godot_cpp/classes/ref.hpp

Line 245 in 517db66

return Ref<T>(reinterpret_cast<T *>(godot::internal::gde_interface->object_get_instance_binding(

I see two options here:

Option 1: give object_get_instance_binding the ability to find the true type of an object and the binding_callbacks of the corresponding godot-cpp wrapper class. Then we can do dynamic_cast<T*>(reinterpret_cast<Object*>(api_obj_ptr)) in Object::cast_to .

Option 2: Implement a magical engine.is_base_of(T::class(), engine.get_real_class(obj)) function, and Object::cast_to returns nullptr if that function returns false.

Not sure if they are easy to do as I have little knowledge of the Godot class system

[dsnopek]

Option 1: give object_get_instance_binding the ability to find the true type of an object and the binding_callbacks of the corresponding godot-cpp wrapper class. Then we can do dynamic_cast<T*>(reinterpret_cast<Object*>(api_obj_ptr)) in Object::cast_to

I have an almost finished implementation of this option. I'll make a PR (or really a pair of PRs since there will be Godot engine changes and godot-cpp) when it's ready!

[dsnopek]

Here's the pair of PR's I promised:

• Ensure GDExtension class is the correct type for the Godot engine class #1050
• Add GDExtension function to get Object class name godot#73511

They're still a work-in-progress. As noted on the first PR, there's a couple things I'd still like to do.