The Python Iterator Protocol: How for Loops Really Work
Every for loop you've written calls __iter__ and __next__ under the hood. Learn the iterator protocol, build a custom iterable, and finally understand what generators ARE.
I was looking at Amir's order processing pipeline last night. He's using a custom class in a for loop. Not a list, not a generator — a class. I didn't know you could do that.
You picked exactly the right thing to notice. What did the class look like?
It had __iter__ and __next__ methods. I recognized __iter__ because we've done dunder methods — that's the special method naming. But __next__? I don't know what that does.
Before I explain those, let me show you something. You've written hundreds of for loops. This:
orders = [{"id": 1, "total": 49.99}, {"id": 2, "total": 129.00}]
for order in orders:
print(order["total"])
What do you think Python is actually doing with that line for order in orders?
Going through the list one item at a time? Moving an internal index?
That is what it looks like. Here is what actually happens. Python calls iter(orders), which returns an iterator object. Then it calls next() on that iterator, over and over, until it gets a StopIteration exception. When it catches that exception, the loop ends. That is the entire for loop, every time, for everything.
That's it? The whole thing is just iter() then repeated next() until StopIteration?
Every for loop you have ever written. Every list comprehension. Every for x in range(10). All of it is iter() → next() → StopIteration. We can prove it:
orders = [{"id": 1, "total": 49.99}, {"id": 2, "total": 129.00}]
# What Python does behind the scenes:
iterator = iter(orders) # Step 1: get the iterator
order = next(iterator) # Step 2: first call to next()
print(order["total"]) # 49.99
order = next(iterator) # Step 3: second call
print(order["total"]) # 129.0
next(iterator) # Step 4: no more items
# Raises StopIteration
That is what for order in orders compiles to.
So iter() and next() are built-in functions. And they... call something on the object?
iter(obj) calls obj.__iter__(). next(obj) calls obj.__next__(). Those are the two methods. Any object that has both of those methods is an iterator. Any object that has __iter__ is an iterable.
Wait, what's the difference between an iterable and an iterator?
An iterable has __iter__. A list is an iterable. A string is an iterable. A dict is an iterable. When you call iter(my_list), it gives you back a list iterator — a separate object that tracks where you are in the list. That list iterator is the actual iterator: it has both __iter__ and __next__.
So the list itself is not the iterator. The list just knows how to make one.
Exactly. Call iter() on a list and you get back a list_iterator object. That object has a position. Every call to next() advances it one step.
Can I see that?
Yes:
orders = ["order-1", "order-2", "order-3"]
it = iter(orders)
print(type(it)) # <class 'list_iterator'>
print(it is orders) # False — a different object
print(next(it)) # order-1
print(next(it)) # order-2
# Now make a fresh iterator
it2 = iter(orders)
print(next(it2)) # order-1 — starts over
So every time I call iter() I get a fresh cursor. A different object that can travel through the list independently.
And that is why you can have two for loops over the same list and they do not interfere — each one gets its own iterator from iter().
Okay. So to make a class that works in a for loop, I need to give it __iter__ and __next__. __iter__ returns the iterator object — usually self. __next__ returns the next value, or raises StopIteration when done.
I was going to build up to that. You just got there.
Week 4, remember? I'm reading ahead.
Fair. Let me show you what it looks like in code:
class OrderBatch:
"""Yields orders in batches of N."""
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
self._index = 0
def __iter__(self):
return self # this object is its own iterator
def __next__(self):
if self._index >= len(self.orders):
raise StopIteration # done — signal the for loop to stop
batch = self.orders[self._index : self._index + self.batch_size]
self._index += self.batch_size
return batch
Now you can do:
orders = list(range(1, 8)) # [1, 2, 3, 4, 5, 6, 7]
batcher = OrderBatch(orders, batch_size=3)
for batch in batcher:
print(batch)
# [1, 2, 3]
# [4, 5, 6]
# [7]
So __iter__ returns self because the class is both the iterable and the iterator. And __next__ is where the actual logic lives — advance the index, return the slice, raise StopIteration when you fall off the end.
Right. The key line is raise StopIteration. That is your signal to the for loop: I'm done. Without it, the loop runs forever.
What happens if I iterate batcher a second time?
Try it:
batcher = OrderBatch(orders, batch_size=3)
for batch in batcher:
print(batch)
# [1, 2, 3], [4, 5, 6], [7]
for batch in batcher: # second loop
print(batch)
# nothing — _index is already past the end
Because __iter__ returns self, you are getting the same iterator back — with the same _index that is already exhausted. The second loop starts, calls __next__ immediately, sees index >= len, raises StopIteration, and the loop body never runs.
That's what happens to generators too. Once they're exhausted, they're done.
Stop right there. Say that again.
Generators... they're exhausted after one pass. You can't loop over a generator twice. And now I'm seeing — is that because a generator IS an iterator? Does it implement __iter__ and __next__?
Every generator you've written since Week 1. Every yield statement. Python is making __iter__ and __next__ for you automatically. A generator object is just an iterator with a very clever __next__: instead of incrementing an index, it resumes the function from where it left off at the last yield.
So yield is just... a way of writing __next__ without writing a class. Python compiles the generator function into an object with __iter__ and __next__ already built in.
That is exactly what it is. Here is the same logic as a generator:
def order_batches(orders, batch_size):
index = 0
while index < len(orders):
yield orders[index : index + batch_size]
index += batch_size
And as a class with __iter__ / __next__:
class OrderBatch:
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self.orders):
raise StopIteration
batch = self.orders[self._index : self._index + self.batch_size]
self._index += self.batch_size
return batch
Same behavior. The generator is the short version. The class is the long version. Both implement the iterator protocol.
And that's why Week 1 generators were exhausted after one loop! They're iterators. Iterators don't reset. If I wanted to loop over a generator twice, I'd have to call the generator function again to get a fresh iterator.
That is it. And now you understand something most developers get wrong for years: the difference between a list and a generator is not just memory. A list is an iterable that gives you a fresh iterator every time. A generator is an iterator that you consume once.
So if I'm designing a class that I want to be safely re-iterable — like OrderBatch — I should split the iterable from the iterator. The class is the iterable. __iter__ returns a new iterator each time.
That is the advanced version, and you got there on your own. Here is what that looks like:
class OrderBatch: # the iterable
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
def __iter__(self): # creates a FRESH iterator each time
return OrderBatchIterator(self.orders, self.batch_size)
class OrderBatchIterator: # the iterator
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self.orders):
raise StopIteration
batch = self.orders[self._index : self._index + self.batch_size]
self._index += self.batch_size
return batch
Now OrderBatch can be looped over multiple times safely, just like a list.
The iterable is the recipe. The iterator is the chef working through the tickets. When you start a new loop, you get a new chef — fresh start.
That is the cleanest way I've heard it. Actually stealing that analogy.
So where does this connect back to everything else? Closures? Decorators?
The connection is how Python works. Every time you write a for loop, Python's machinery runs the same protocol. When you write a decorator, you're relying on closures — which are functions with captured state. When you write a generator, you're relying on the iterator protocol — which is an object with captured state tracked across calls. It's all the same idea: objects and functions that carry their own state and respond to a defined protocol.
And the special methods — __iter__, __next__, __repr__, __init__, __enter__, __exit__ — those are how Python lets you plug into its internal machinery. You're not hacking the language. You're using the same mechanism the language already uses.
The whole language is protocols. len(x) calls x.__len__(). x + y calls x.__add__(y). with open(f) calls f.__enter__() and f.__exit__(). You're not adding behavior to Python. You're teaching your objects to speak Python's native language.
I finally feel like I understand Python, not just its syntax.
You've earned that. Day 24 was closures. Day 25 was decorators. Day 26 and 27 were the patterns. Today was the foundation of everything — the protocol that makes for loops work, the mechanism that makes generators possible, the reason special methods exist. You now understand the design of the language, not just how to use it.
Day 29 is the quiz. What should I actually study?
Don't study. Review. Go back to Day 24. Trace the closure. Go back to Day 25. Write a decorator from memory. Come back to today and implement an iterator from scratch. If you can do those three things without looking, you're ready.
And Day 30?
Day 30 you take that first quiz again. The same one from Day 1. I want to see your scores.
Practice your skills
Sign up to write and run code in this lesson.
The Python Iterator Protocol: How for Loops Really Work
Every for loop you've written calls __iter__ and __next__ under the hood. Learn the iterator protocol, build a custom iterable, and finally understand what generators ARE.
I was looking at Amir's order processing pipeline last night. He's using a custom class in a for loop. Not a list, not a generator — a class. I didn't know you could do that.
You picked exactly the right thing to notice. What did the class look like?
It had __iter__ and __next__ methods. I recognized __iter__ because we've done dunder methods — that's the special method naming. But __next__? I don't know what that does.
Before I explain those, let me show you something. You've written hundreds of for loops. This:
orders = [{"id": 1, "total": 49.99}, {"id": 2, "total": 129.00}]
for order in orders:
print(order["total"])
What do you think Python is actually doing with that line for order in orders?
Going through the list one item at a time? Moving an internal index?
That is what it looks like. Here is what actually happens. Python calls iter(orders), which returns an iterator object. Then it calls next() on that iterator, over and over, until it gets a StopIteration exception. When it catches that exception, the loop ends. That is the entire for loop, every time, for everything.
That's it? The whole thing is just iter() then repeated next() until StopIteration?
Every for loop you have ever written. Every list comprehension. Every for x in range(10). All of it is iter() → next() → StopIteration. We can prove it:
orders = [{"id": 1, "total": 49.99}, {"id": 2, "total": 129.00}]
# What Python does behind the scenes:
iterator = iter(orders) # Step 1: get the iterator
order = next(iterator) # Step 2: first call to next()
print(order["total"]) # 49.99
order = next(iterator) # Step 3: second call
print(order["total"]) # 129.0
next(iterator) # Step 4: no more items
# Raises StopIteration
That is what for order in orders compiles to.
So iter() and next() are built-in functions. And they... call something on the object?
iter(obj) calls obj.__iter__(). next(obj) calls obj.__next__(). Those are the two methods. Any object that has both of those methods is an iterator. Any object that has __iter__ is an iterable.
Wait, what's the difference between an iterable and an iterator?
An iterable has __iter__. A list is an iterable. A string is an iterable. A dict is an iterable. When you call iter(my_list), it gives you back a list iterator — a separate object that tracks where you are in the list. That list iterator is the actual iterator: it has both __iter__ and __next__.
So the list itself is not the iterator. The list just knows how to make one.
Exactly. Call iter() on a list and you get back a list_iterator object. That object has a position. Every call to next() advances it one step.
Can I see that?
Yes:
orders = ["order-1", "order-2", "order-3"]
it = iter(orders)
print(type(it)) # <class 'list_iterator'>
print(it is orders) # False — a different object
print(next(it)) # order-1
print(next(it)) # order-2
# Now make a fresh iterator
it2 = iter(orders)
print(next(it2)) # order-1 — starts over
So every time I call iter() I get a fresh cursor. A different object that can travel through the list independently.
And that is why you can have two for loops over the same list and they do not interfere — each one gets its own iterator from iter().
Okay. So to make a class that works in a for loop, I need to give it __iter__ and __next__. __iter__ returns the iterator object — usually self. __next__ returns the next value, or raises StopIteration when done.
I was going to build up to that. You just got there.
Week 4, remember? I'm reading ahead.
Fair. Let me show you what it looks like in code:
class OrderBatch:
"""Yields orders in batches of N."""
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
self._index = 0
def __iter__(self):
return self # this object is its own iterator
def __next__(self):
if self._index >= len(self.orders):
raise StopIteration # done — signal the for loop to stop
batch = self.orders[self._index : self._index + self.batch_size]
self._index += self.batch_size
return batch
Now you can do:
orders = list(range(1, 8)) # [1, 2, 3, 4, 5, 6, 7]
batcher = OrderBatch(orders, batch_size=3)
for batch in batcher:
print(batch)
# [1, 2, 3]
# [4, 5, 6]
# [7]
So __iter__ returns self because the class is both the iterable and the iterator. And __next__ is where the actual logic lives — advance the index, return the slice, raise StopIteration when you fall off the end.
Right. The key line is raise StopIteration. That is your signal to the for loop: I'm done. Without it, the loop runs forever.
What happens if I iterate batcher a second time?
Try it:
batcher = OrderBatch(orders, batch_size=3)
for batch in batcher:
print(batch)
# [1, 2, 3], [4, 5, 6], [7]
for batch in batcher: # second loop
print(batch)
# nothing — _index is already past the end
Because __iter__ returns self, you are getting the same iterator back — with the same _index that is already exhausted. The second loop starts, calls __next__ immediately, sees index >= len, raises StopIteration, and the loop body never runs.
That's what happens to generators too. Once they're exhausted, they're done.
Stop right there. Say that again.
Generators... they're exhausted after one pass. You can't loop over a generator twice. And now I'm seeing — is that because a generator IS an iterator? Does it implement __iter__ and __next__?
Every generator you've written since Week 1. Every yield statement. Python is making __iter__ and __next__ for you automatically. A generator object is just an iterator with a very clever __next__: instead of incrementing an index, it resumes the function from where it left off at the last yield.
So yield is just... a way of writing __next__ without writing a class. Python compiles the generator function into an object with __iter__ and __next__ already built in.
That is exactly what it is. Here is the same logic as a generator:
def order_batches(orders, batch_size):
index = 0
while index < len(orders):
yield orders[index : index + batch_size]
index += batch_size
And as a class with __iter__ / __next__:
class OrderBatch:
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self.orders):
raise StopIteration
batch = self.orders[self._index : self._index + self.batch_size]
self._index += self.batch_size
return batch
Same behavior. The generator is the short version. The class is the long version. Both implement the iterator protocol.
And that's why Week 1 generators were exhausted after one loop! They're iterators. Iterators don't reset. If I wanted to loop over a generator twice, I'd have to call the generator function again to get a fresh iterator.
That is it. And now you understand something most developers get wrong for years: the difference between a list and a generator is not just memory. A list is an iterable that gives you a fresh iterator every time. A generator is an iterator that you consume once.
So if I'm designing a class that I want to be safely re-iterable — like OrderBatch — I should split the iterable from the iterator. The class is the iterable. __iter__ returns a new iterator each time.
That is the advanced version, and you got there on your own. Here is what that looks like:
class OrderBatch: # the iterable
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
def __iter__(self): # creates a FRESH iterator each time
return OrderBatchIterator(self.orders, self.batch_size)
class OrderBatchIterator: # the iterator
def __init__(self, orders, batch_size):
self.orders = orders
self.batch_size = batch_size
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self.orders):
raise StopIteration
batch = self.orders[self._index : self._index + self.batch_size]
self._index += self.batch_size
return batch
Now OrderBatch can be looped over multiple times safely, just like a list.
The iterable is the recipe. The iterator is the chef working through the tickets. When you start a new loop, you get a new chef — fresh start.
That is the cleanest way I've heard it. Actually stealing that analogy.
So where does this connect back to everything else? Closures? Decorators?
The connection is how Python works. Every time you write a for loop, Python's machinery runs the same protocol. When you write a decorator, you're relying on closures — which are functions with captured state. When you write a generator, you're relying on the iterator protocol — which is an object with captured state tracked across calls. It's all the same idea: objects and functions that carry their own state and respond to a defined protocol.
And the special methods — __iter__, __next__, __repr__, __init__, __enter__, __exit__ — those are how Python lets you plug into its internal machinery. You're not hacking the language. You're using the same mechanism the language already uses.
The whole language is protocols. len(x) calls x.__len__(). x + y calls x.__add__(y). with open(f) calls f.__enter__() and f.__exit__(). You're not adding behavior to Python. You're teaching your objects to speak Python's native language.
I finally feel like I understand Python, not just its syntax.
You've earned that. Day 24 was closures. Day 25 was decorators. Day 26 and 27 were the patterns. Today was the foundation of everything — the protocol that makes for loops work, the mechanism that makes generators possible, the reason special methods exist. You now understand the design of the language, not just how to use it.
Day 29 is the quiz. What should I actually study?
Don't study. Review. Go back to Day 24. Trace the closure. Go back to Day 25. Write a decorator from memory. Come back to today and implement an iterator from scratch. If you can do those three things without looking, you're ready.
And Day 30?
Day 30 you take that first quiz again. The same one from Day 1. I want to see your scores.