Copying from wrong address + segmentation fault issue

[davidhamb95]

Hi All,

Recently I have encountered with the situation (segmentation fault issue) which is difficult for me to debug.
I have a diamond-like hierarchy of classes in C++, which I want to use in python. I have written pybind wrapper but
had some problems after compiling and trying to call one of my methods.

Here is the C++ source code (animal.hpp):

#include <string>
#include <iostream>

namespace example {

class Creature {
public:
    virtual std::string typeName() const = 0;
    virtual ~Creature() = default;
};

class Animal : public Creature {
public:
    Animal() {
        std::cout << "Animal created: " << this << std::endl;
    }

    Animal(const Animal& animal) : Creature(animal) {
        std::cout << "Animal copied: from " << &animal << " to " << this << std::endl;
    }

    ~Animal() {
        std::cout << "Animal deleted: " << this << std::endl;
    }

     std::string typeName() const override {
        return _typeName;
     }

private:
    std::string _typeName = "animal";
};

class TerrestrialAnimal : public virtual Animal {
public:
    TerrestrialAnimal() {
        std::cout << "TerrestrialAnimal created: " << this << std::endl;
    }

    TerrestrialAnimal(const TerrestrialAnimal& animal) : Animal(animal) {
        std::cout << "TerrestrialAnimal copied: from " << &animal << " to " << this << std::endl;
    }

    ~TerrestrialAnimal() {
        std::cout << "TerrestrialAnimal deleted: " << this << std::endl;
    }

     std::string typeName() const override {
        return _typeName;
     }

private:
    std::string _typeName = "terrestrial";
};

class AquaticAnimal : public virtual Animal {
public:
    AquaticAnimal() {
        std::cout << "AquaticAnimal created: " << this << std::endl;
    }

    AquaticAnimal(const AquaticAnimal& animal) : Animal(animal) {
        std::cout << "AquaticAnimal copied: from " << &animal << " to " << this << std::endl;
    }

    ~AquaticAnimal() {
        std::cout << "AquaticAnimal deleted: " << this << std::endl;
    }

    std::string typeName() const override {
       return _typeName;
    }

private:
    std::string _typeName = "aquatic";
};

class Frog : public TerrestrialAnimal, public AquaticAnimal {
public:
    Frog() {
        std::cout << "Frog created: " << this << std::endl;
    }

    Frog(const Frog& animal) : Animal(animal), TerrestrialAnimal(animal), AquaticAnimal(animal) {
        std::cout << "Frog copied: from " << &animal << " to " << this << std::endl;
    }

    ~Frog() {
        std::cout << "Frog deleted: " << this << std::endl;
    }

    std::string typeName() const override {
       return _typeName;
    }

private:
    std::string _typeName = "frog";
};

class AnimalUsage {
public:
    AnimalUsage() {
        std::cout << "AnimalUsage created: " << this << std::endl;
    }
    
    AnimalUsage(const AnimalUsage& u) {
        std::cout << "AnimalUsage copied: from " << &u << " to " << this << std::endl;
    }

    ~AnimalUsage() {
        std::cout << "AnimalUsage deleted: " << this << std::endl;
    }

    const Animal& getAnimal() {
        return frog;
    }

    const AquaticAnimal& getAquaticAnimal() {
        return frog;
    }

    const Frog& getFrog() {
        return frog;
    }

private:
    Frog frog;
};

}

Corresponding pybind wrapper:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/functional.h>

#include "animal.hpp"

namespace py = pybind11;

PYBIND11_MODULE(examples, m) {
    using namespace example;

    py::class_<example::Animal> animal(m, "Animal");
    
    animal.def(py::init<>());
    animal.def("type_name", &example::Animal::typeName);

    py::class_<example::TerrestrialAnimal, example::Animal> terrestrialAnimal(m, "TerrestrialAnimal");
    
    terrestrialAnimal.def(py::init<>());
    terrestrialAnimal.def("type_name", &example::TerrestrialAnimal::typeName);

    py::class_<example::AquaticAnimal, example::Animal> aquaticAnimal(m, "AquaticAnimal");
    
    aquaticAnimal.def(py::init<>());
    aquaticAnimal.def("type_name", &example::AquaticAnimal::typeName);

    py::class_<example::Frog, example::TerrestrialAnimal, example::AquaticAnimal> frog(m, "Frog");
    
    frog.def(py::init<>());
    frog.def("type_name", &example::Frog::typeName);

    py::class_<example::AnimalUsage> animalUsage(m, "AnimalUsage");
    
    animalUsage.def(py::init<>());
    animalUsage.def(py::init<const example::AnimalUsage &>(), py::arg("u"));
    animalUsage.def("get_animal", &example::AnimalUsage::getAnimal);
    animalUsage.def("get_aquatic_animal", &example::AnimalUsage::getAquaticAnimal);
    animalUsage.def("get_frog", &example::AnimalUsage::getFrog);
}

After compiling this file and generating the lib by the name examples, I imported AnimalUsage class in my test file and tried to call get_animal method:

from examples import AnimalUsage

usage = AnimalUsage()
usage.get_animal()

This call ends with segmentation fault:

Animal created: 0x600000c800d8
TerrestrialAnimal created: 0x600000c80080
AquaticAnimal created: 0x600000c800a0
Frog created: 0x600000c80080
AnimalUsage created: 0x600000c80080
Animal copied: from 0x600000c80080 to 0x6000022804a0
[1]    12401 segmentation fault  ./test.py

In my source C++ example I keep Frog instance in AnimalUsage class and return it from three different methods: getAnimal, getAquaticAnimal and getFrog. I know that I shouldn't expect polymorphism to work here since the default return value policy is copy policy for references, but I can't find out the cause of segmentation fault.

Except of segfault issue, from Animal copy constructor log I also noticed that object is being copied from address 0x600000c80080 which is not the address of Animal which has ben created. It corresponds to the Frog address of the initial object. It seems to me that the copy should have been done from 0x600000c800d8 address. It seems that pybind detects at the runtime that returned Animal& object type is Frog, goes to that object and uses it to call Animal copy constructor.

I want to know whether those kind of issues are known, is the copy address correct and what causes segfault issue ?

FYI: everything is ok when I return by value instead of returning by reference from getAnimal method.

Thanks,
Davit

[jagerman]

Two things.

1. This does look like a bug, and it isn't immediately obvious to me what is happening here. The copy address being wrong is almost certainly the issue, but why it is wrong is not obvious.

2. Your binding code isn't correct for what you're trying to do; what you have here is an owned reference which requires a little extra binding work to tell pybind about how to handle it:

    animalUsage.def("get_animal", &example::AnimalUsage::getAnimal, py::return_value_policy::reference_internal);
    animalUsage.def("get_aquatic_animal", &example::AnimalUsage::getAquaticAnimal, py::return_value_policy::reference_internal);
    animalUsage.def("get_frog", &example::AnimalUsage::getFrog, py::return_value_policy::reference_internal);

That reference_internal does two things:

• it makes pybind reference the returned value without taking ownership.
• it tells pybind that the return reference is something that is managed as part of the parent, so it applies a keep-alive that keeps the parent alive as long as something in Python is still referencing a child.

[davidhamb95]

Thanks @jagerman,

After noticing this issue, I started to use reference_internal policy, but yet wanted to understand the source of the issue.
I'll open a bug regarding this topic.