std::weak_ptr expiring for derived Python objects when a trampoline class is used

[nsoblath]

I have a setup where I'm using a std::weak_ptr to keep track of objects that can originate in C++ or Python, and in a certain situation the weak_ptr expires even if the corresponding Python object still exists. Am I doing something wrong or is there a Pybind11 issue? Any help will be appreciated.

I have a minimum working example to demonstrate the issue. The C++ base class is thing, and there's a "holder" class that contains a std::weak_ptr< thing >. I don't want the holder to influence the lifetime of the thing it holds, hence using weak_ptr. For the binding, I want to have a trampoline so I can have virtual functions handled properly in Python. For this example I've included a derived thing in Python called DerivedThing.

The issue I'm running into is that after passing a DerivedThing to thing_holder and returning to Python, when I try to access the DerivedThing via its pointer in thing_holder again, the weak_ptr has expired.

Here's the minimum working example. First the C++ classes:

class thing : public std::enable_shared_from_this< thing >
{
    public:
        thing() = default;
        virtual ~thing() = default;

        virtual std::string get_type() const
        {
            return std::string("thing");
        }
};

class thing_holder
{
    public:
        thing_holder() = default;
        virtual ~thing_holder() = default;

        void print_thing() const
        {
            std::cout << "thing expired? " << f_thing.expired() << std::endl;
            if( ! f_thing.expired() )
            {
                std::cout << "   type is " << f_thing.lock()->get_type() << std::endl;
            }
            return;
        }

        void set_thing( std::shared_ptr< thing > a_thing_ptr )
        {
            f_thing = std::weak_ptr< thing >( a_thing_ptr );
            print_thing();  // print here to check the pointer while still in C++
            return;
        }

    private:
        std::weak_ptr< thing > f_thing;
};

Next, the trampoline class and Python bindings:

class thing_trampoline : public thing //, public py::trampoline_self_life_support
{
    public:
        using thing::thing;

        std::string get_type() const override 
        {
            PYBIND11_OVERRIDE( std::string, thing, get_type, );
        }
};

std::list< std::string > export_thing( pybind11::module& mod )
{
    std::list< std::string > all_members;

    all_members.push_back( "Thing" );
    py::classh< thing, thing_trampoline >( mod, "Thing", "a thing" )
        .def( py::init< >() )
        .def( "get_type", &thing::get_type, "get the typename" )
        ;

    all_members.push_back( "ThingHolder" );
    py::class_< thing_holder >( mod, "ThingHolder", "holds things" )
        .def( py::init< >(), 
              py::return_value_policy::take_ownership )

        .def( "print_thing", &thing_holder::print_thing, "print thing" )
        .def( "set_thing", &thing_holder::set_thing, "set the thing" )
        ;

    return all_members;
}

And finally the Python script that includes the derived thing class:

class DerivedThing(dripline.core.Thing):
    def __init__(self):
        dripline.core.Thing.__init__(self)

    def get_type(self):
        return "DerivedThing"

if __name__ == '__main__':
    holder = thing_module.ThingHolder()

    print('Creating a thing')
    thing = thing_module.Thing()
    print(f'Type: {thing.get_type()}')
    
    print('Giving thing to the holder')
    holder.set_thing(thing)

    print('Printing after a return to python')
    holder.print_thing()

    print('\n')

    print('Creating a derived thing')
    derthing = DerivedThing()
    print(f'Type: {derthing.get_type()}')

    print('(Giving derived thing to the holder')
    holder.set_thing(derthing)

    print('Printing after a return to python')
    holder.print_thing()

The expected output is:

Creating a thing
Type: thing
Setting thing then printing
thing expired? 0
   type is thing
Printing after a return to python
thing expired? 0
   type is thing


Creating a derived thing
Type: DerivedThing
Setting thing then printing
thing expired? 0
   type is DerivedThing
Printing after a return to python
thing expired? 0
   type is DerivedThing

The actual output is:

Creating a thing
Type: thing
Setting thing then printing
thing expired? 0
   type is thing
Printing after a return to python
thing expired? 0
   type is thing


Creating a derived thing
Type: DerivedThing
Setting thing then printing
thing expired? 0
   type is DerivedThing
Printing after a return to python
thing expired? 1

The issue is the last line, which indicates that the weak_ptr is no longer aware that the object still exists. My hypothesis is that even though we're using py::smart_holder in the thing binding, the fact that we only keep weak_ptr< thing > in the C++ results in the C++ side losing track of the Python objects, whereas if we had a shared_ptr< thing >, according to the documentation, it would still work.

I tested a these variations:

• Thing created in Python (as in the above example) -- works
• DerivedThing created in Python (as in the above example) -- does not work
• DerivedThing created in Python with no thing_trampoline -- works (except there's no virtual function overriding)
• DerivedThing created in Python where thing_trampoline also inherits from py::trampoline_self_life_support (commented out in the above code) -- does not work

[nsoblath]

Discussed in and resolved by #5624