Return reference to member field in PyO3

Question

Suppose I have a Rust struct like this

struct X{...}

struct Y{
   x:X
}

I’d like to be able to write python code that accesses X through Y

y = Y()
y.x.some_method()

What would be the best way to implement it in PyO3? Currently I made two wrapper classes

#[pyclass]
struct XWrapper{
   x:X
}
#[pyclass]
struct YWrapper{
   y:Y
}
#[pymethods]
impl YWrapper{
   #[getter]
   pub fn x(&self)->XWrapper{
      XWrapper{x:self.y.clone()}
   }
}

However, this requires clone(). I’d rather want to return reference. Of course I know that if X was a pyclass, then I could easily return PyRef to it. But the problem is that X and Y come from a Rust library and I cannot nilly-wily add #[pyclass] to them.

Asked By: alagris

||

Source

Answer 1

I don’t think what you say is possible without some rejigging of the interface:

Your XWrapper owns the x and your Y owns its x as well. That means creating an XWrapper will always involve a clone (or a new).

Could we change XWrapper so that it merely contains a reference to an x? Not really, because that would require giving XWrapper a lifetime annotation, and PyO3 afaik doesn’t allow pyclasses with lifetime annotation. Makes sense, because passing an object to python puts it on the python heap, at which point rust loses control over the object.

So what can we do?

Some thoughts: Do you really need to expose the composition structure of y to the python module? Just because that’s the way it’s organized within Rust doesn’t mean it needs to be that way in Python. Your YWrapper could provide methods to the python interface that behind the scenes forward the request to the x instance:

#[pymethods]
impl YWrapper{
   pub fn some_method(&self) {
     self.y.x.some_method();
   }
}

This would also be a welcome sight to strict adherents of the Law of Demeter 😉

I’m trying to think of other clever ways. Depending on some of the details of how y.x is accessed and modified by the methods of y itself, it might be possible to add a field x: XWrapper to the YWrapper. Then you create the XWrapper (including a clone of y.x) once when YWrapper is created, and from then on you can return references to that XWrapper in your pub fn x. Of course that becomes much more cumbersome when x gets frequently changed and updated via the methods of y…

In a way, this demonstrates the clash between Python’s ref-counted object model and Rust’s ownership object model. Rust enforces that you can’t arbitrarily mess with objects unless you’re their owner.

Answered By: Lagerbaer

Answer 2

It is indeed possible to share objects and return them or mutate them. Whatever, just as in Python. What Lagerbaer suggested works and it’s actually pretty ideal for small codes. However, if the number of methods increases there will be A LOT of repeating and boilerplate needed (and worse, folds every time you increase the depth of your nesting).

I have no idea if this is something that we are supposed to do. But from what I understood, the way to do it is using Py. God wish I had a habit of reading the docs thoroughly before experimenting.

In https://docs.rs/pyo3/latest/pyo3/#the-gil-independent-types in the MAIN PAGE of the doc says:
When wrapped in Py<…>, like with Py or Py, Python objects no longer have a limited lifetime which makes them easier to store in structs and pass between functions. However, you cannot do much with them without a Python<‘py> token, for which you’d need to reacquire the GIL.

A Py is "A GIL-independent reference to an object allocated on the Python heap." https://docs.rs/pyo3/latest/pyo3/prelude/struct.Py.html

In other words, to return pyclass objects, we need to wrap it like Py<pyclass_struct_name>.

Your example is too complicated and to be honest I don’t even understand what you are trying to do but here is an alternative version which suits my own usecase more closely. Since this is basically one of the only results that pops in Google I see it fit to paste it here even if it is not an exact response to the example provided above.

So here we go…

Suppose we have a Rust struct X and we cannot modify the lib as you mentioned. We need an XWrapper (let’s call it PyX) pyclass to hold it.

So we define them here:

// in lib.rs
pub struct X {}

// in py_bindings.rs
#[pyclass]
struct PyX{
    value: Py<X>,
}

impl_new_for!(PyX);

Then for the usage, all we have to do is to initialize the object with a GIL lock (assuming in the init of the XWrapper) and then define a getter for it. THE IMPORTANT NOTE HERE IS THAT YOU CALL clone_ref ON IT AND DO NOT RETURN THE OBJECT.

This is basically a nested class system afterwards and the nested object is immutable (has interior mutability tho) so it’s a fantastic way to nest your code as well.

In the example below, I used my needed X as a PyX in yet another wrapper called Api.

#[pyclass]
struct Api {
    x: PyX,
}

#[pymethods]
impl Api {
    #[new]
    fn __new__() -> PyResult<Self> {
        Python::with_gil(|py| {
            Ok(Self {
                x: Py::new(
                    py,
                    PyX::new(),
                ),
            })
        }
    }

    #[getter(x)]
    fn x(&mut self, py: Python) -> Py<Network> {
        self.x.clone_ref(py)
    }
}

Answered By: Davoodeh

Return reference to member field in PyO3

Question:

Answers: