Python Iterables, Iterators & Generators: A Complete Guide

By /

In Python, an iterable object is any object you can loop through using a for loop or while loop.

Common examples of iterables are:

  • Lists
  • Strings
  • Dictionaries
  • Tuples
  • Sets

For example, you can loop through a list of numbers:

numbers = [1, 2, 3]

for number in numbers:
    print(number)

Output:

1
2
3

You can use a similar approach to loop through the characters of a string:

word = "Testing"

for character in word:
    print(character)

Output:

T
e
s
t
i
n
g

This comprehensive guide teaches you everything you need to know about iterables, iterators, and generators in Python. After reading this guide, you understand:

  • What the yield keyword really means
  • What makes an object iterable
  • What makes a function iterator
  • What is a generator
  • How a for loop works behind the scenes

And much more.

Introduction: Iterables and Iterators in Python

An iterable object means it implements the __iter__ method under the hood. This method returns an iterator that can be used to loop through the iterable.

Let’s take a look at an example of a traditional non-iterable object. Here is a simple Python class called Course:

class Course:
    participants = ["Alice", "Bob", "Charlie"]

Let’s create a Course object of that class:

course = Course()

Now, let’s try to loop through the Course object and try to print each participant on the course:

for student in course:
    print(student)

Output:

Traceback (most recent call last):
  File "example.py", line 7, in <module>
    for student in course:
TypeError: 'Course' object is not iterable

Of course, this piece of code fails. The error says it all—The Course is not iterable.

Trying to loop through this course object is meaningless. Python has no idea what you’re trying to do.

To make the above code work, you need to make the Course class iterable. To convert the Course class into an iterable, implement the two special methods:

  1. __iter__.
  2. __next__.

Here is the updated version of the Course class:

class Course:
    participants = ["Alice", "Bob", "Charlie"]

    def __iter__(self):
        return iter(self.participants)

    def __next__(self):
        while True:
            try:
                value = next(self)
            except StopIteration:
                break
        return value

Now you can loop the Course objects with the previously failing for loop syntax:

course = Course()

for student in course:
    print(student)

Output:

Alice
Bob
Charlie

It works!

At this point, you probably have no idea what the above code does. No worries! That’s what you will be focusing on for the rest of this guide. The above example is just an introduction to what you can do with iterables and iterators in Python.

To understand how this code works, you need to better understand iterators and iterables. Furthermore, you need to learn what the special methods __iter__ and __next__ actually do.

Iterables and Iterators in Python

So, an iterable object in Python is something you can loop through using a for (or while) loop.

But what makes an object iterable in the first place?

To qualify as an iterable, the object has to implement the __iter__() method under the hood.

Let’s think about some common iterable types in Python. For example, a list is clearly iterable as you can loop through it using a for loop syntax, right?

Now, let’s see what special methods a list object implements behind the scenes. To do this, call the dir() function on a list:

numbers = [1, 2, 3, 4, 5]
print(dir(numbers))

Here is the output:

Python all list methods

This output is a list of all the methods that list objects implement under the hood. You can see the list has the __iter__() method. This verifies that a Python list is indeed iterable.

This __iter__() method is important because it’s what makes looping through a list possible.

For a for loop to work, it calls the __iter__() method of a list. This method returns an iterator. The loop then uses this iterator object to step through all the values.

But what on earth is an iterator then?

An iterator is an object with a state. An iterator object remembers where it is during an iteration. Iterators also know how to get the next value in the collection. They do this by using the __next__() method, which by the way is a method every iterator needs to have.

Let’s continue with the numbers list example and grab the iterator of the list:

numbers = [1, 2, 3, 4, 5]
iter_numbers = iter(numbers)

(By the way, calling the iter(numbers) is the same as calling numbers.__iter__() )

Now, let’s call the dir() function on the iterator object to see what methods it has:

print(dir(iter_numbers))

Result:

Iterator methods in Python

There’s the __next__() method that I talked about. This is what characterizes an iterator. Without this method, the iterator would not work.

Woah… So much new information!

Let’s recap what you have learned thus far.

  • A Python list is iterable. In other words, you can call a for loop on it. To be iterable, a list implements the __iter__() method under the hood.
  • This __iter__() method returns an iterator object. The for loop uses the iterator object to actually step through the list values.
  • To qualify as an iterator, the iterator must implement the __next__() method for obtaining the next value in the list.

Now, let’s continue exploring the numbers list object. Let’s call the __next__() method on the numbers iterator a bunch of times to see what happens:

>>> numbers = [1, 2, 3, 4, 5]    # specify a numbers list
>>> iter_numbers = iter(numbers) # get the iterator of the list
>>> next(iter_numbers)           # get the next value of the list using the iterator
1
>>> next(iter_numbers)
2
>>> next(iter_numbers)
3
>>> next(iter_numbers)
4
>>> next(iter_numbers)
5

(By the way, calling the next(numbers) is the same as calling numbers.__next__() )

Calling next(iter_numbers) always returns the next number in the numbers list. But how on earth is this possible?

This is possible because the iterator object has an internal state. It remembers where it left off when __next__() was called last time. So when the __next__() method is called again, the iterator object knows what value comes next and delivers it to the caller.

As you can see from the above example, the last value you received was 5. This means you have reached the end of the numbers list.

To see what happens if you continue calling the next() function, let’s call it once more:

>>> next(iter_numbers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>

Because there are no more values to access on the list, a StopIteration exception is thrown. At this point, the iterator is exhausted.

The cool part of the above example is you just did what a for loop does under the hood!

When you call a for loop on an iterable, such as a list:

  • It calls the __iter__() method on a list to retrieve an iterator object.
  • Then it calls the __next__() method until there are no values left.
  • When there are no values left, a StopIteration exception is thrown. The for loop handles the exception for you. Thus you never see it.

Let’s simulate the behavior of a for loop again. This time, let’s make it less manual and use a while loop instead:

numbers = [1, 2, 3, 4, 5]

# Get the iterator from numbers list
iter_numbers = iter(numbers)

# Start retrieving the next values indefinitely
while True:
    try:
        # Try to get the next value from the iterator and print it
        number = next(iter_numbers)
        print(number)
    # If the iterator has no more values, escape the loop
    except StopIteration:
        break

Output:

1
2
3
4
5

The above code construct works exactly the same way as a for…in loop operates on a list.

Awesome! Now you understand what are iterables and iterators in Python. Also, you now know how a for loop truly works when iterating over a list.

Next, let’s take a look at how you can implement custom iterables and iterators.

How To Create Iterators and Iterables

There is nothing special about Python’s native iterables, such as a list or tuple. You can convert any custom classes to iterables too!

To do this, you need to implement the __iter__() and __next__() methods in your class.

Example

I’m sure you’re familiar with the built-in range() function in Python. You can use it like this:

for i in range(4):
    print(i)

Output:

0
1
2
3

To demonstrate iterables and iterators, let’s create a custom implementation of range(). Here is a class RangeValues that mimics the behavior of the range() function:

class RangeValues:
    def __init__(self, start_value, end_value):
        self.current_value = start_value
        self.end_value = end_value
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current_value >= self.end_value:
            raise StopIteration
        value = self.current_value
        self.current_value += 1
        return value

Now, let’s go through this implementation line by line to understand what’s happening.

Lines 2–4

  • The __init__() method makes it possible to initialize a RangeValues object with start and end values. For example: RangeValues(0, 10).

Lines 6–7

  • The __iter__() method makes the class iterable. In other words, it is possible to call for i in RangeValues(0,10). This method returns an iterator. In this case, it returns the class itself, because the class is an iterator as it implements the __next__() method.

Lines 9–14

These lines specify how the iterator behaves when someone calls a for loop on the RangeValues object.

  • The __next__() method is responsible for going through the values from start to end. It raises a StopIteration exception when it reaches the end of the range.
  • If the iterator hasn’t reached the end yet, it continues returning the current value (and increments it for the next round).

Now that you understand how the code works, let’s finally test the RangeValues class:

for i in RangeValues(1,5):
    print(i)

Output:

1
2
3
4

As you can see, this function now works just like the range() function in Python.

Building an example iterable/iterator like this is a great way to get a better understanding of how the iterables work in Python.

Make sure to tweak the code and see what happens. Also, feel free to come up with your own iterable concepts and put them to test. If you only read this article without experimenting with the code, chances are you won’t understand the concepts well enough.

Now, let’s move on to generators that offer a more readable way to write iterators.

Generators—Readable Iterators in Python

If you take a look at the above example of RangeValues class, you see it’s daunting to read.

Luckily, Python provides you with generators to remedy this problem.

A generator is an iterator whose implementation is easier to read. The readability advantage stems from the fact that a generator lets you omit the implementation of __iter__() and __next__() methods.

Because a generator is also an iterator, it doesn’t return a single value. Instead, it yields (delivers) values one at a time. Besides, the generator keeps track of the state of the iteration to know what value to deliver next.

For example, let’s turn the RangeValues class from the earlier example into a generator:

def range_values(start, end):
    current = start
    while current < end:
        yield current
        current += 1

Let’s test the function:

for i in range_values(0,5):
    print(i)

Output:

0
1
2
3
4

The range_values works exactly like the RangeValues class but the implementation is way cleaner.

First of all, you don’t need to specify a class. Instead, you can use a generator function like the above. Then, you don’t need to implement the __next__() and __iter__() methods.

How Does ‘Yield’ Work?

In the previous example, you turned iterable into a generator to make it more readable.

But in the above example, you saw a new keyword yield. If you have never seen it before, there is no way for you to tell how it works.

When you write a generator function, you don’t return values. This is because, as you might recall, a generator only knows the current value and how to get the next value. Thus, a generator doesn’t store values in memory. This is what makes generators memory efficient.

To create a generator, you cannot return values. Instead, you need to yield them.

When Python encounters the yield keyword, it delivers a value and pauses the execution until the next call.

For example, if you have a huge text file with a lot of words, you cannot store the words in a Python list. This is because there is not enough memory for you to do that. To iterate over the words in the file, you cannot store them in memory.

This is where a generator comes in handy.

A generator picks the first word, yields it to you, and moves to the next one. It does this until there are no words in the list. This way, you don’t need to store the words in your Python program to go through them.

Make sure to read Yield vs Return to get a better understanding.

Infinite Stream of Elements with Generators

Iterators and generators only care about the current value and how to get the next one. It’s thus possible to create an infinite stream of values because you don’t need to store them anywhere.

Example

Let’s create an infinite iterator that produces all the numbers after a starting value. Let’s use a generator function to keep it readable:

def infinite_values(start):
    current = start
    while True:
        yield current
        current += 1

(If you want to see how this could be done with a class, here it is.)

This iterator produces values from start to infinity.

Let’s run it. (Warning: An infinite loop):

infinite_nums = infinite_values(0)

for num in infinite_nums:
    print(num)

Output:

0
1
2
3
4
5
.
.
.

Syntactically it looks as if the infinite_nums really was an infinite list of numbers after 0.

In reality, it’s nothing but an iterator that stores the current value and knows how to get the next one.

Conclusion

Today you learned what iterable means in Python.

An iterable is something that can be looped over in Python.

On a low level, an iterable is an object that implements the __iter__() method which returns an iterator.

An iterator is an object with a state. It remembers where it’s at during an iteration. To qualify as an iterator, an object must implement the __next__() method for obtaining the next value in the iteration process.

Thanks for reading. Happy coding!

Scroll to Top