Why does a yield from inside __next__() return generator object?
Question:
I am using yield
to return the next value in the __next__()
function in my class. However it does not return the next value, it returns the generator object.
I am trying to better understand iterators and yield
. I might be doing it in the wrong way.
Have a look.
class MyString:
def __init__(self,s):
self.s=s
def __iter__(self):
return self
def __next__(self):
for i in range(len(self.s)):
yield(self.s[i])
r=MyString("abc")
i=iter(r)
print(next(i))
This returns:
generator object __next__ at 0x032C05A0
Answers:
next
pretty much just calls __next__()
in this case. Calling __next__
on your object will start the generator and return it (no magic is done at this point).
In this case, you might be able to get away with not defining __next__
at all:
class MyString:
def __init__(self,s):
self.s=s
def __iter__(self):
for i in range(len(self.s)):
yield(self.s[i])
# Or...
# for item in self.s:
# yield item
If you wanted to use __iter__
and __next__
(to define an iterator rather than simply making an iterable), you’d probably want to do something like this:
class MyString:
def __init__(self,s):
self.s = s
self._ix = None
def __iter__(self):
return self
def __next__(self):
if self._ix is None:
self._ix = 0
try:
item = self.s[self._ix]
except IndexError:
# Possibly reset `self._ix`?
raise StopIteration
self._ix += 1
return item
Let’s take a look at the purpose of the __next__
method. From the docs:
iterator.__next__()
Return the next item from the container. If there are no further items, raise the StopIteration exception.
Now let’s see what the yield
statement does. Another excerpt from the docs:
Using a yield expression in a function’s body causes that function to
be a generator
And
When a generator function is called, it returns an iterator known as a
generator.
Now compare __next__
and yield
: __next__
returns the next item from the container. But a function containing the yield
keyword returns an iterator. Consequently, using yield
in a __next__
method results in an iterator that yields iterators.
If you want to use yield
to make your class iterable, do it in the __iter__
method:
class MyString:
def __init__(self, s):
self.s = s
def __iter__(self):
for s in self.s:
yield s
The __iter__
method is supposed to return an iterator – and the yield
keyword makes it do exactly that.
For completeness, here is how you would implement an iterator with a __next__
method. You have to keep track of the state of the iteration, and return the corresponding value. The easiest solution is probably to increment an index every time __next__
is called:
class MyString:
def __init__(self,s):
self.s = s
self.index = -1
def __iter__(self):
return self
def __next__(self):
self.index += 1
if self.index >= len(self.s):
raise StopIteration
return self.s[self.index]
As far as I can tell, generator functions are just syntactic sugar for classes with a next function. Example:
>>> def f():
i = 0
while True:
i += 1
yield i
>>> x = f()
>>> x
<generator object f at 0x0000000000659938>
>>> next(x)
1
>>> next(x)
2
>>> next(x)
3
>>> class g(object):
def __init__(self):
self.i = 0
def __next__(self):
self.i += 1
return self.i
>>> y = g()
>>> y
<__main__.g object at 0x000000000345D908>
>>> next(y)
1
>>> next(y)
2
>>> next(y)
3
In fact, I came here looking to see if there is any significant difference. Please shout if there is.
So, to answer the question, what you have is a class with a __next__ method that returns an object that also has a __next__ method. So the simplest thing to do would be to replace your yield
with a return
and to keep track of how far along you are, and to remember to raise a StopIteration when you reach the end of the array. So something like:
class MyString:
def __init__(self,s):
self.s=s
self._i = -1
def __iter__(self):
return self
def __next__(self):
self._i += 1
if self._i >= len(self.s):
raise StopIteration
return self.s[self._i]
That’s probably the simplest way to achieve what I think you’re looking for.
OBSERVATION
If next()
function calls __next__()
method, what’s happening in the following example.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
for i in range(self.s):
yield i
def __next__(self):
print('__next__ method is called.')
if __name__== '__main__':
obj = T()
k = iter(obj)
print(next(k)) #0
print(next(k)) #1
print(next(k)) #2
print(next(k)) #3
print(next(k)) #4
print(next(k)) #5
print(next(k)) #6
print(next(k)) #7
print(next(k)) #8
print(next(k)) #9
print(next(k))
print(next(k))
Terminal:
C:...>python test.py
0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):
File "test.py", line 25, in <module>
print(next(k))
StopIteration
WHAT IS HAPPENING?
It seams that next()
function does not calling __next__
method. I cannot understand why python docs states that "next(iterator, default)
Retrieve the next item from the iterator by calling its __next__()
method." If someonw knows, let us help!
Case: __iter__
with __next__
in custom class with yield
So, if you want to use yield
(in order to create a generator) with __iter__
and __next__
methods in a custom class
, do not put just the yield
into the __next__
method, but use it with __iter__(self)
and return self.__next__()
instead return self
.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
return self.__next__()
def __next__(self):
for i in range(self.s):
yield i
if __name__== '__main__':
obj = T()
for i in obj:
print(i)
Terminal:
C:...>python test.py
0
1
2
3
4
5
6
7
8
9
C:...>
Also, you can call from __iter__
any other method instead __next__()
.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
return self.foo()
def foo(self):
for i in range(self.s):
yield i
if __name__== '__main__':
obj = T()
for i in obj:
print(i)
You will have exactly the same results.
Case: yield
in __iter__
method without __next__
method
I don’t think it is a good idea to use yield
in __iter__
. Ok, it works, but I think that destroys the class API.
Case: __iter__
with __next__
in custom class without yield
Use these methods (__iter__
and __next__
). In the __iter__
return self
and do not forget to raise StopIteration
in __next__
method.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
self.__i = -1
return self
def __next__(self):
while self.__i < self.s-1:
self.__i+=1
return self.__i
raise StopIteration
if __name__== '__main__':
obj = T()
for i in obj:
print(i)
I am using yield
to return the next value in the __next__()
function in my class. However it does not return the next value, it returns the generator object.
I am trying to better understand iterators and yield
. I might be doing it in the wrong way.
Have a look.
class MyString:
def __init__(self,s):
self.s=s
def __iter__(self):
return self
def __next__(self):
for i in range(len(self.s)):
yield(self.s[i])
r=MyString("abc")
i=iter(r)
print(next(i))
This returns:
generator object __next__ at 0x032C05A0
next
pretty much just calls __next__()
in this case. Calling __next__
on your object will start the generator and return it (no magic is done at this point).
In this case, you might be able to get away with not defining __next__
at all:
class MyString:
def __init__(self,s):
self.s=s
def __iter__(self):
for i in range(len(self.s)):
yield(self.s[i])
# Or...
# for item in self.s:
# yield item
If you wanted to use __iter__
and __next__
(to define an iterator rather than simply making an iterable), you’d probably want to do something like this:
class MyString:
def __init__(self,s):
self.s = s
self._ix = None
def __iter__(self):
return self
def __next__(self):
if self._ix is None:
self._ix = 0
try:
item = self.s[self._ix]
except IndexError:
# Possibly reset `self._ix`?
raise StopIteration
self._ix += 1
return item
Let’s take a look at the purpose of the __next__
method. From the docs:
iterator.__next__()
Return the next item from the container. If there are no further items, raise the StopIteration exception.
Now let’s see what the yield
statement does. Another excerpt from the docs:
Using a yield expression in a function’s body causes that function to
be a generator
And
When a generator function is called, it returns an iterator known as a
generator.
Now compare __next__
and yield
: __next__
returns the next item from the container. But a function containing the yield
keyword returns an iterator. Consequently, using yield
in a __next__
method results in an iterator that yields iterators.
If you want to use yield
to make your class iterable, do it in the __iter__
method:
class MyString:
def __init__(self, s):
self.s = s
def __iter__(self):
for s in self.s:
yield s
The __iter__
method is supposed to return an iterator – and the yield
keyword makes it do exactly that.
For completeness, here is how you would implement an iterator with a __next__
method. You have to keep track of the state of the iteration, and return the corresponding value. The easiest solution is probably to increment an index every time __next__
is called:
class MyString:
def __init__(self,s):
self.s = s
self.index = -1
def __iter__(self):
return self
def __next__(self):
self.index += 1
if self.index >= len(self.s):
raise StopIteration
return self.s[self.index]
As far as I can tell, generator functions are just syntactic sugar for classes with a next function. Example:
>>> def f():
i = 0
while True:
i += 1
yield i
>>> x = f()
>>> x
<generator object f at 0x0000000000659938>
>>> next(x)
1
>>> next(x)
2
>>> next(x)
3
>>> class g(object):
def __init__(self):
self.i = 0
def __next__(self):
self.i += 1
return self.i
>>> y = g()
>>> y
<__main__.g object at 0x000000000345D908>
>>> next(y)
1
>>> next(y)
2
>>> next(y)
3
In fact, I came here looking to see if there is any significant difference. Please shout if there is.
So, to answer the question, what you have is a class with a __next__ method that returns an object that also has a __next__ method. So the simplest thing to do would be to replace your yield
with a return
and to keep track of how far along you are, and to remember to raise a StopIteration when you reach the end of the array. So something like:
class MyString:
def __init__(self,s):
self.s=s
self._i = -1
def __iter__(self):
return self
def __next__(self):
self._i += 1
if self._i >= len(self.s):
raise StopIteration
return self.s[self._i]
That’s probably the simplest way to achieve what I think you’re looking for.
OBSERVATION
If next()
function calls __next__()
method, what’s happening in the following example.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
for i in range(self.s):
yield i
def __next__(self):
print('__next__ method is called.')
if __name__== '__main__':
obj = T()
k = iter(obj)
print(next(k)) #0
print(next(k)) #1
print(next(k)) #2
print(next(k)) #3
print(next(k)) #4
print(next(k)) #5
print(next(k)) #6
print(next(k)) #7
print(next(k)) #8
print(next(k)) #9
print(next(k))
print(next(k))
Terminal:
C:...>python test.py
0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):
File "test.py", line 25, in <module>
print(next(k))
StopIteration
WHAT IS HAPPENING?
It seams that next()
function does not calling __next__
method. I cannot understand why python docs states that "next(iterator, default)
Retrieve the next item from the iterator by calling its __next__()
method." If someonw knows, let us help!
Case: __iter__
with __next__
in custom class with yield
So, if you want to use yield
(in order to create a generator) with __iter__
and __next__
methods in a custom class
, do not put just the yield
into the __next__
method, but use it with __iter__(self)
and return self.__next__()
instead return self
.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
return self.__next__()
def __next__(self):
for i in range(self.s):
yield i
if __name__== '__main__':
obj = T()
for i in obj:
print(i)
Terminal:
C:...>python test.py
0
1
2
3
4
5
6
7
8
9
C:...>
Also, you can call from __iter__
any other method instead __next__()
.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
return self.foo()
def foo(self):
for i in range(self.s):
yield i
if __name__== '__main__':
obj = T()
for i in obj:
print(i)
You will have exactly the same results.
Case: yield
in __iter__
method without __next__
method
I don’t think it is a good idea to use yield
in __iter__
. Ok, it works, but I think that destroys the class API.
Case: __iter__
with __next__
in custom class without yield
Use these methods (__iter__
and __next__
). In the __iter__
return self
and do not forget to raise StopIteration
in __next__
method.
Code:
class T:
def __init__(self):
self.s = 10
def __iter__(self):
self.__i = -1
return self
def __next__(self):
while self.__i < self.s-1:
self.__i+=1
return self.__i
raise StopIteration
if __name__== '__main__':
obj = T()
for i in obj:
print(i)