The Iterator Protocol: __iter__, __next__, and Infinite Lazy Sequences
Master the iterator protocol: __iter__ returns self, __next__ yields values or raises StopIteration. Build custom iterators for lazy evaluation.
Wait. I just realized something. Generators implement iter and next. That's why they work in for loops. But I've never written those methods myself. I just... yield. It's like yesterday with context managers — we wrote enter and exit, and suddenly the with statement worked. But we never wrote those methods by hand before that either.
Exactly. You're connecting the dots. Yesterday you saw that context managers are protocols — you implement enter and exit, and suddenly with knows how to use your object. Same thing today. Implement iter and next, and suddenly for loops know how to use your object. You've discovered the secret. A generator is a shortcut. It is syntactic sugar for the iterator protocol. Today you are going to see what is hiding underneath that sugar — and when a hand-written class iterator is better than a generator.
But generators seem simpler. Why would I ever go back to writing iter and next myself?
Because sometimes the state you need to track is not linear. You are building an order paginator yesterday that needs to remember: current page, total pages, whether it has reached the end. A generator would work. But an iterator class lets you store that state as attributes, add helper methods, and be explicit about what is happening. Also — and this matters — infinite iterators.
Infinite? Like... an iterator that never stops?
Exactly. A generator that yields forever. Or an iterator class that, when you call next(), always has another value. Imagine an infinite stream of order IDs, or Fibonacci numbers, or a repeating pattern. An iterator can model that.
Let me start with the protocol itself. You know for order in batch: works on objects that are iterable. What does "iterable" mean?
Something you can... iterate over? With a for loop?
That is the definition, yes. But let's be precise. An iterable is an object that has an __iter__ method. That's it. __iter__ is the protocol. When you write for order in batch:, Python calls batch.__iter__(). That call returns an iterator.
An iterator is an object that has two methods: __iter__ (which returns self) and __next__ (which returns the next value or raises StopIteration). Every time the for loop runs, it calls next() on the iterator. When there are no more values, __next__ raises StopIteration and the for loop ends.
Wait. So __iter__ returns an iterator, which also has __iter__? That seems redundant.
I know it feels that way. Here is the thinking: An iterator's __iter__ always returns self. That way, if you call iter(iterator) a second time, you get the same iterator back. An iterable might return a fresh iterator each time __iter__ is called. That distinction matters.
Here is the mental picture using the sheet music analogy. A score is an iterable. A finger following the score is an iterator. The score's __iter__ says "here is a finger on page 1." The finger's __iter__ says "I am the finger."
So every time I call iter(score), I get a new finger starting at the beginning. But if I call iter(finger), I get the same finger.
Exactly. The finger is stateful. It remembers where it is on the page. The score is stateless — it is just the sheet music.
Now let me show you the skeleton of an iterator class. An OrderPageIterator that fetches pages of orders one at a time:
class OrderPageIterator:
def __init__(self, total_pages):
self.current_page = 0
self.total_pages = total_pages
def __iter__(self):
return self # Iterator returns itself
def __next__(self):
if self.current_page >= self.total_pages:
raise StopIteration # Signal end of iteration
self.current_page += 1
return f'Page {self.current_page}' # Return the next value
# Use it
for page in OrderPageIterator(total_pages=3):
print(page)
# Output:
# Page 1
# Page 2
# Page 3
It raises an exception to signal the end? That does not feel very Pythonic.
It absolutely does feel weird the first time. But it is actually elegant. StopIteration is not an error — it is a signal. The for loop catches it and exits cleanly. You never see the exception unless you call next() directly after the iteration is supposed to end.
So if I call next() on the iterator manually, and there are no more values, it raises StopIteration?
Yes. Watch:
iterator = OrderPageIterator(total_pages=2)
print(next(iterator)) # Page 1
print(next(iterator)) # Page 2
print(next(iterator)) # StopIteration raised — for loop would catch this
The for loop is just syntactic sugar for this pattern:
iterator = iter(batch) # Call __iter__
while True:
try:
value = next(iterator) # Call __next__
# do something with value
except StopIteration:
break
So the for loop is handling the StopIteration exception invisibly. That is why the iteration stops.
Exactly. The protocol is: __iter__ returns an iterator, __next__ returns values until it raises StopIteration. Everything else is sugar.
Now let's look at the harder case: pagination. Your iterator needs to do work — fetch data from a data source. It is lazy, meaning it only fetches the page when you ask for it:
def page_fetcher(page_number):
"""Simulate fetching a page of orders from a database."""
orders = [
{'id': f'ORD-{page_number}-{i}', 'total': 100 + i}
for i in range(1, 4)
]
return orders
class OrderPageIterator:
def __init__(self, total_pages):
self.current_page = 0
self.total_pages = total_pages
def __iter__(self):
return self
def __next__(self):
if self.current_page >= self.total_pages:
raise StopIteration
self.current_page += 1
page = page_fetcher(self.current_page)
return page
# Use it
for page in OrderPageIterator(total_pages=3):
print(f'Fetched {len(page)} orders')
So the iterator is lazy — it only calls page_fetcher when the for loop asks for the next value. If you break out early, you only fetched the pages you needed.
That is the whole point. Lazy evaluation. If your database has a million pages, your iterator does not fetch all a million pages upfront. It fetches one per iteration. If you break after 10, you only fetched 10.
Okay, so when would I use an iterator class instead of a generator? The generator would be shorter.
True. A generator version would look like this:
def order_page_generator(total_pages):
for page_num in range(1, total_pages + 1):
page = page_fetcher(page_num)
yield page
for page in order_page_generator(total_pages=3):
print(f'Fetched {len(page)} orders')
Very clean. But now imagine you want to add methods. You want to ask the iterator: "Can you skip ahead to page 10?" Or "What page are we on right now?" Or "Reset to the beginning."
With a generator, those become harder — the generator function is already running, frozen at a yield. You cannot easily add methods that modify its internal state. With an iterator class, you just add methods:
class OrderPageIterator:
def __init__(self, total_pages):
self.current_page = 0
self.total_pages = total_pages
def __iter__(self):
return self
def __next__(self):
if self.current_page >= self.total_pages:
raise StopIteration
self.current_page += 1
return page_fetcher(self.current_page)
def skip_to_page(self, page_num):
"""Jump to a specific page."""
self.current_page = page_num - 1
def current_page_number(self):
"""What page are we on?"""
return self.current_page
# Use it
iterator = OrderPageIterator(total_pages=10)
iterator.skip_to_page(5) # Jump to page 5
page = next(iterator) # Fetch page 5
So an iterator class is a container with state and methods. A generator is a function that yields. They both follow the iterator protocol, but a class gives you more control.
That is the distinction. For simple linear cases, generators are fine. For complex state management or methods, an iterator class is clearer.
Now, infinite iterators. Watch this:
class InfiniteCounter:
def __init__(self, start=0):
self.current = start
def __iter__(self):
return self
def __next__(self):
self.current += 1
return self.current
# Use it — be careful, this is infinite
counter = InfiniteCounter()
for i, num in enumerate(counter):
if i >= 5:
break
print(num)
# Output: 1, 2, 3, 4, 5
It never raises StopIteration. It just keeps yielding forever.
Exactly. An infinite iterator. When you combine it with something like itertools.islice or a break condition, it becomes useful. The standard library has infinite iterators built in — you will see those tomorrow. But the point is: an iterator can be stateful, can have methods, and can be infinite. A generator can do all three too, but sometimes a class is clearer.
So my task is to build an OrderPageIterator that fetches pages lazily, using iter and next?
Exactly. Initialize with total_pages. Each time __next__ is called, fetch and return the next page. When you have exhausted all pages, raise StopIteration. Use the page_fetcher function provided to simulate the database call.
And the whole thing is lazy — no pages are fetched until the iteration asks for them.
That is it. You are implementing the iterator protocol from scratch. By the end of today, you will understand why generators work in for loops — because they implement this same protocol under the hood. And tomorrow you will see that the standard library has infinite iterators waiting for you in itertools.
Practice your skills
Sign up to write and run code in this lesson.
The Iterator Protocol: __iter__, __next__, and Infinite Lazy Sequences
Master the iterator protocol: __iter__ returns self, __next__ yields values or raises StopIteration. Build custom iterators for lazy evaluation.
Wait. I just realized something. Generators implement iter and next. That's why they work in for loops. But I've never written those methods myself. I just... yield. It's like yesterday with context managers — we wrote enter and exit, and suddenly the with statement worked. But we never wrote those methods by hand before that either.
Exactly. You're connecting the dots. Yesterday you saw that context managers are protocols — you implement enter and exit, and suddenly with knows how to use your object. Same thing today. Implement iter and next, and suddenly for loops know how to use your object. You've discovered the secret. A generator is a shortcut. It is syntactic sugar for the iterator protocol. Today you are going to see what is hiding underneath that sugar — and when a hand-written class iterator is better than a generator.
But generators seem simpler. Why would I ever go back to writing iter and next myself?
Because sometimes the state you need to track is not linear. You are building an order paginator yesterday that needs to remember: current page, total pages, whether it has reached the end. A generator would work. But an iterator class lets you store that state as attributes, add helper methods, and be explicit about what is happening. Also — and this matters — infinite iterators.
Infinite? Like... an iterator that never stops?
Exactly. A generator that yields forever. Or an iterator class that, when you call next(), always has another value. Imagine an infinite stream of order IDs, or Fibonacci numbers, or a repeating pattern. An iterator can model that.
Let me start with the protocol itself. You know for order in batch: works on objects that are iterable. What does "iterable" mean?
Something you can... iterate over? With a for loop?
That is the definition, yes. But let's be precise. An iterable is an object that has an __iter__ method. That's it. __iter__ is the protocol. When you write for order in batch:, Python calls batch.__iter__(). That call returns an iterator.
An iterator is an object that has two methods: __iter__ (which returns self) and __next__ (which returns the next value or raises StopIteration). Every time the for loop runs, it calls next() on the iterator. When there are no more values, __next__ raises StopIteration and the for loop ends.
Wait. So __iter__ returns an iterator, which also has __iter__? That seems redundant.
I know it feels that way. Here is the thinking: An iterator's __iter__ always returns self. That way, if you call iter(iterator) a second time, you get the same iterator back. An iterable might return a fresh iterator each time __iter__ is called. That distinction matters.
Here is the mental picture using the sheet music analogy. A score is an iterable. A finger following the score is an iterator. The score's __iter__ says "here is a finger on page 1." The finger's __iter__ says "I am the finger."
So every time I call iter(score), I get a new finger starting at the beginning. But if I call iter(finger), I get the same finger.
Exactly. The finger is stateful. It remembers where it is on the page. The score is stateless — it is just the sheet music.
Now let me show you the skeleton of an iterator class. An OrderPageIterator that fetches pages of orders one at a time:
class OrderPageIterator:
def __init__(self, total_pages):
self.current_page = 0
self.total_pages = total_pages
def __iter__(self):
return self # Iterator returns itself
def __next__(self):
if self.current_page >= self.total_pages:
raise StopIteration # Signal end of iteration
self.current_page += 1
return f'Page {self.current_page}' # Return the next value
# Use it
for page in OrderPageIterator(total_pages=3):
print(page)
# Output:
# Page 1
# Page 2
# Page 3
It raises an exception to signal the end? That does not feel very Pythonic.
It absolutely does feel weird the first time. But it is actually elegant. StopIteration is not an error — it is a signal. The for loop catches it and exits cleanly. You never see the exception unless you call next() directly after the iteration is supposed to end.
So if I call next() on the iterator manually, and there are no more values, it raises StopIteration?
Yes. Watch:
iterator = OrderPageIterator(total_pages=2)
print(next(iterator)) # Page 1
print(next(iterator)) # Page 2
print(next(iterator)) # StopIteration raised — for loop would catch this
The for loop is just syntactic sugar for this pattern:
iterator = iter(batch) # Call __iter__
while True:
try:
value = next(iterator) # Call __next__
# do something with value
except StopIteration:
break
So the for loop is handling the StopIteration exception invisibly. That is why the iteration stops.
Exactly. The protocol is: __iter__ returns an iterator, __next__ returns values until it raises StopIteration. Everything else is sugar.
Now let's look at the harder case: pagination. Your iterator needs to do work — fetch data from a data source. It is lazy, meaning it only fetches the page when you ask for it:
def page_fetcher(page_number):
"""Simulate fetching a page of orders from a database."""
orders = [
{'id': f'ORD-{page_number}-{i}', 'total': 100 + i}
for i in range(1, 4)
]
return orders
class OrderPageIterator:
def __init__(self, total_pages):
self.current_page = 0
self.total_pages = total_pages
def __iter__(self):
return self
def __next__(self):
if self.current_page >= self.total_pages:
raise StopIteration
self.current_page += 1
page = page_fetcher(self.current_page)
return page
# Use it
for page in OrderPageIterator(total_pages=3):
print(f'Fetched {len(page)} orders')
So the iterator is lazy — it only calls page_fetcher when the for loop asks for the next value. If you break out early, you only fetched the pages you needed.
That is the whole point. Lazy evaluation. If your database has a million pages, your iterator does not fetch all a million pages upfront. It fetches one per iteration. If you break after 10, you only fetched 10.
Okay, so when would I use an iterator class instead of a generator? The generator would be shorter.
True. A generator version would look like this:
def order_page_generator(total_pages):
for page_num in range(1, total_pages + 1):
page = page_fetcher(page_num)
yield page
for page in order_page_generator(total_pages=3):
print(f'Fetched {len(page)} orders')
Very clean. But now imagine you want to add methods. You want to ask the iterator: "Can you skip ahead to page 10?" Or "What page are we on right now?" Or "Reset to the beginning."
With a generator, those become harder — the generator function is already running, frozen at a yield. You cannot easily add methods that modify its internal state. With an iterator class, you just add methods:
class OrderPageIterator:
def __init__(self, total_pages):
self.current_page = 0
self.total_pages = total_pages
def __iter__(self):
return self
def __next__(self):
if self.current_page >= self.total_pages:
raise StopIteration
self.current_page += 1
return page_fetcher(self.current_page)
def skip_to_page(self, page_num):
"""Jump to a specific page."""
self.current_page = page_num - 1
def current_page_number(self):
"""What page are we on?"""
return self.current_page
# Use it
iterator = OrderPageIterator(total_pages=10)
iterator.skip_to_page(5) # Jump to page 5
page = next(iterator) # Fetch page 5
So an iterator class is a container with state and methods. A generator is a function that yields. They both follow the iterator protocol, but a class gives you more control.
That is the distinction. For simple linear cases, generators are fine. For complex state management or methods, an iterator class is clearer.
Now, infinite iterators. Watch this:
class InfiniteCounter:
def __init__(self, start=0):
self.current = start
def __iter__(self):
return self
def __next__(self):
self.current += 1
return self.current
# Use it — be careful, this is infinite
counter = InfiniteCounter()
for i, num in enumerate(counter):
if i >= 5:
break
print(num)
# Output: 1, 2, 3, 4, 5
It never raises StopIteration. It just keeps yielding forever.
Exactly. An infinite iterator. When you combine it with something like itertools.islice or a break condition, it becomes useful. The standard library has infinite iterators built in — you will see those tomorrow. But the point is: an iterator can be stateful, can have methods, and can be infinite. A generator can do all three too, but sometimes a class is clearer.
So my task is to build an OrderPageIterator that fetches pages lazily, using iter and next?
Exactly. Initialize with total_pages. Each time __next__ is called, fetch and return the next page. When you have exhausted all pages, raise StopIteration. Use the page_fetcher function provided to simulate the database call.
And the whole thing is lazy — no pages are fetched until the iteration asks for them.
That is it. You are implementing the iterator protocol from scratch. By the end of today, you will understand why generators work in for loops — because they implement this same protocol under the hood. And tomorrow you will see that the standard library has infinite iterators waiting for you in itertools.