Why does a yield from inside __next__() return generator object?

Question:

I am using yield to return the next value in the __next__() function in my class. However it does not return the next value, it returns the generator object.

I am trying to better understand iterators and yield. I might be doing it in the wrong way.

Have a look.

class MyString:
    def __init__(self,s):
        self.s=s

    def __iter__(self):
        return self

    def __next__(self):
        for i in range(len(self.s)):
            yield(self.s[i])

r=MyString("abc")
i=iter(r)
print(next(i))

This returns:

generator object __next__ at 0x032C05A0

Asked By: sachin

||

Answers:

next pretty much just calls __next__() in this case. Calling __next__ on your object will start the generator and return it (no magic is done at this point).


In this case, you might be able to get away with not defining __next__ at all:

class MyString:
    def __init__(self,s):
        self.s=s

    def __iter__(self):
        for i in range(len(self.s)):
            yield(self.s[i])
        # Or...
        # for item in self.s:
        #     yield item

If you wanted to use __iter__ and __next__ (to define an iterator rather than simply making an iterable), you’d probably want to do something like this:

class MyString:
    def __init__(self,s):
        self.s = s
        self._ix = None

    def __iter__(self):
        return self

    def __next__(self):
        if self._ix is None:
            self._ix = 0

        try:
            item = self.s[self._ix]
        except IndexError:
            # Possibly reset `self._ix`?
            raise StopIteration
        self._ix += 1
        return item
Answered By: mgilson

Let’s take a look at the purpose of the __next__ method. From the docs:

iterator.__next__()

Return the next item from the container. If there are no further items, raise the StopIteration exception.

Now let’s see what the yield statement does. Another excerpt from the docs:

Using a yield expression in a function’s body causes that function to
be a generator

And

When a generator function is called, it returns an iterator known as a
generator.

Now compare __next__ and yield: __next__ returns the next item from the container. But a function containing the yield keyword returns an iterator. Consequently, using yield in a __next__ method results in an iterator that yields iterators.


If you want to use yield to make your class iterable, do it in the __iter__ method:

class MyString:
    def __init__(self, s):
        self.s = s

    def __iter__(self):
        for s in self.s:
            yield s

The __iter__ method is supposed to return an iterator – and the yield keyword makes it do exactly that.


For completeness, here is how you would implement an iterator with a __next__ method. You have to keep track of the state of the iteration, and return the corresponding value. The easiest solution is probably to increment an index every time __next__ is called:

class MyString:
    def __init__(self,s):
        self.s = s
        self.index = -1

    def __iter__(self):
        return self

    def __next__(self):
        self.index += 1

        if self.index >= len(self.s):
            raise StopIteration

        return self.s[self.index]
Answered By: Aran-Fey

As far as I can tell, generator functions are just syntactic sugar for classes with a next function. Example:

>>> def f():
    i = 0
    while True:
        i += 1
        yield i


>>> x = f()
>>> x
<generator object f at 0x0000000000659938>
>>> next(x)
1
>>> next(x)
2
>>> next(x)
3
>>> class g(object):
    def __init__(self):
        self.i = 0

    def __next__(self):
        self.i += 1
        return self.i


>>> y = g()
>>> y
<__main__.g object at 0x000000000345D908>
>>> next(y)
1
>>> next(y)
2
>>> next(y)
3

In fact, I came here looking to see if there is any significant difference. Please shout if there is.

So, to answer the question, what you have is a class with a __next__ method that returns an object that also has a __next__ method. So the simplest thing to do would be to replace your yield with a return and to keep track of how far along you are, and to remember to raise a StopIteration when you reach the end of the array. So something like:

class MyString:
    def __init__(self,s):
        self.s=s
        self._i = -1

    def __iter__(self):
        return self

    def __next__(self):
        self._i += 1
        if self._i >= len(self.s):
            raise StopIteration
        return self.s[self._i]

That’s probably the simplest way to achieve what I think you’re looking for.

Answered By: P.Cowderoy

OBSERVATION

If next() function calls __next__() method, what’s happening in the following example.

Code:

class T:
    def __init__(self):
        self.s = 10
        
    def __iter__(self):
        for i in range(self.s):
            yield i
    
    def __next__(self):
        print('__next__ method is called.')
    
if __name__== '__main__':
    obj = T()
    k = iter(obj)
    print(next(k)) #0
    print(next(k)) #1
    print(next(k)) #2
    print(next(k)) #3
    print(next(k)) #4
    print(next(k)) #5
    print(next(k)) #6
    print(next(k)) #7
    print(next(k)) #8
    print(next(k)) #9
    print(next(k))
    print(next(k))

Terminal:

C:...>python test.py
0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    print(next(k))
StopIteration

WHAT IS HAPPENING?

It seams that next() function does not calling __next__ method. I cannot understand why python docs states that "next(iterator, default) Retrieve the next item from the iterator by calling its __next__() method." If someonw knows, let us help!

Case: __iter__ with __next__ in custom class with yield

So, if you want to use yield (in order to create a generator) with __iter__ and __next__ methods in a custom class, do not put just the yield into the __next__ method, but use it with __iter__(self) and return self.__next__() instead return self.

Code:

class T:
    def __init__(self):
        self.s = 10
        
    def __iter__(self):
        return self.__next__()
    
    def __next__(self):
        for i in range(self.s):
            yield i
    
if __name__== '__main__':
    obj = T()
    for i in obj:
        print(i)

Terminal:

C:...>python test.py
0
1
2
3
4
5
6
7
8
9

C:...>

Also, you can call from __iter__ any other method instead __next__().

Code:

class T:
    def __init__(self):
        self.s = 10
        
    def __iter__(self):
        return self.foo()
    
    def foo(self):
        for i in range(self.s):
            yield i
    
if __name__== '__main__':
    obj = T()
    for i in obj:
        print(i)

You will have exactly the same results.

Case: yield in __iter__ method without __next__ method

I don’t think it is a good idea to use yield in __iter__. Ok, it works, but I think that destroys the class API.

Case: __iter__ with __next__ in custom class without yield

Use these methods (__iter__ and __next__). In the __iter__ return self and do not forget to raise StopIteration in __next__ method.

Code:

class T:
    def __init__(self):
        self.s = 10
        
    def __iter__(self):
        self.__i = -1
        return self
    
    def __next__(self):
        while  self.__i < self.s-1:
            self.__i+=1
            return self.__i
        raise StopIteration
    
if __name__== '__main__':
    obj = T()
    for i in obj:
        print(i)
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.