Python 3.11: Exception Groups

Python 3.11 just released, and we are going to take a look at some of the new features in this version. Today, we will look at Exception Groups.

Exception Groups

We tweeted about exception groups previously as a part of our EuroPython coverage. In this article we will take a deeper look at this feature.

Exception Groups are described in PEP 654. This feature allows us to group multiple exceptions together and raise the whole group of exceptions. Why would we need to group exceptions? The PEP lists out some motivations. Here are some of the common ones

• Suppose we are running many tasks concurrently (either threads or asyncio) and are waiting for them to finish. Multiple tasks fail with an exception. We want to individually catch and handle each exception. We can use exception groups here.
• Sometimes we call an API and retry on failure. After some number retries we give up. We now want to know what was the exception for each of the attempts. Exception groups allow us to group the exceptions from each of the retries so that they can be handled individually.
• Lets say we get an exception in a piece of code, and while handling that exception, we get another exception. We want to handle both exceptions elsewhere, then exception groups can be used to raise both exceptions.

Creating Exception Groups

You can create exception groups using the new ExceptionGroup class. It takes a message and a list of exceptions.

def fn():
    e = ExceptionGroup("multiple exceptions",
            [FileNotFoundError("unknown filename file1.txt"), 
             FileNotFoundError("unknown filename file2.txt"), 
             KeyError("key")])
    raise e

The code above takes two FileNotFoundError and a KeyError and groups them together into an ExceptionGroup. We can then raise the exception group, just like a normal exception.

When we run the code above, we get an output like this

  + Exception Group Traceback (most recent call last):
  |   File "exgroup.py", line 10, in <module>
  |     fn()
  |   File "exgroup.py", line 8, in fn
  |     raise e
  | ExceptionGroup: multiple exceptions (3 sub-exceptions)
  +-+---------------- 1 ----------------
    | FileNotFoundError: unknown filename file1.txt
    +---------------- 2 ----------------
    | FileNotFoundError: unknown filename file2.txt
    +---------------- 3 ----------------
    | KeyError: 'key'
    +------------------------------------

The traceback here shows all the exceptions that are a part of the group.

Handling Exception Groups

Now that we have seen how to raise an exception group, let us take a look at how to handle such exceptions.

Normally, we use try ... except to handle exceptions. You can certainly do this

try:
    fn()
except ExceptionGroup as e:
    print(e.exceptions)

The exceptions attribute here contains all the exceptions that are a part of the group. The problem with this is that it makes it difficult to handle the individual exceptions within the group.

For this, Python 3.11 has a new except* syntax. except* can match exceptions that are within an ExceptionGroup

try:
    fn()
except* FileNotFoundError as eg:
    print("File Not Found Errors:")
    for e in eg.exceptions:
        print(e)
except* KeyError as eg:
    print("Key Errors:")
    for e in eg.exceptions:
        print(e)

Here we use except* to directly match the FileNotFoundError and the KeyError which are within the ExceptionGroup. except* filters the ExceptionGroup and selects those exceptions that match the type specified.

Key differences with try ... except

Let us look at a few key points from this snippet

try:
    fn()
except* FileNotFoundError as eg:
    print("File Not Found Errors:")
    for e in eg.exceptions:
        print(e)
except* KeyError as eg:
    print("Key Errors:")
    for e in eg.exceptions:
        print(e)

First, when we use the as syntax to assign to a variable, then we get an ExceptionGroup object, not a plain exception. In the code above, eg is an ExceptionGroup. For the first except* block, eg will contain all the FileNotFoundErrors from the original exception group. In the second except*, eg will contain all the KeyErrors.

Second, a matching except* block gets executed at most once. In the above code, even if there is more than one FileNotFoundError, the except* FileNotFoundError block will run only once. All the FileNotFoundErrors will be contained in eg and you need to handle all of them in the except* block.

Another important difference between except and except* is that you can match more than one exception block with except*. In the snippet above, the code in except* FileNotFoundError and except* KeyError will both be executed because the exception group contains both types of exceptions.

An example of ExceptionGroup

Now that we have seen how to use ExceptionGroup, let us take a look at it in action. In the code below, we use asyncio to read two files. (We use TaskGroup for this, another new feature in python 3.11. More on TaskGroup coming in the next article).

import asyncio

async def read_file(filename):
    with open(filename) as f:
        data = f.read()
    return data


async def main():
    try:
        async with asyncio.TaskGroup() as g:
            g.create_task(read_file("unknown1.txt"))
            g.create_task(read_file("unknown2.txt"))
        print("All done")
    except* FileNotFoundError as eg:
        for e in eg.exceptions:
            print(e)
    
asyncio.run(main())

We call read_file in lines 12 and 13 to read two files.

Neither of the files are existing, so both the tasks will give FileNotFoundError. TaskGroup will wrap both the failures in an ExceptionGroup and we should use except* to handle both the errors.

When to use ExceptionGroup?

Some of the newer additions to Python, like the TaskGroup that we saw above now use ExceptionGroup. In the future, we will see exception groups being used in many more places. However, keep in mind that you will probably be still using try ... except much more, and except* will be for those specific places where you really, really need exception groups. Most of the time, the library or framework will use exception groups, much like TaskGroup above, rather than regular code.

Which brings us to the question: When should you use exception groups in your own code?

Here, I would refer back to the motivating examples in the PEP. There are certain situations where multiple exceptions need to be raised and the all the exceptions need to be handled. Exception groups are perfect for this.

Most of the time though, good old single exceptions and try ... except should be just fine.

Some Advanced Cases

Now that we have understood the basic usage, let us look at some more advanced cases. You will not need these most of the time, but read on if you want to understand how these cases work

Nested Exception Groups

Exception groups can be used anywhere we use regular exceptions, which means you can have one exception group as a part of another exception group

def fn():
    e = ExceptionGroup("multiple exceptions",
            [ExceptionGroup("file not found",
                [FileNotFoundError("unknown filename file1.txt"), 
                 FileNotFoundError("unknown filename file2.txt")]), 
             KeyError("missing key")])
    raise e

Keep in mind that when using except* ... as eg to match exceptions, then eg will maintain the structure of the original exception group.

try:
    fn()
except* FileNotFoundError as eg:
    # eg will be 
    # ExceptionGroup("multiple exceptions",
    #     [ExceptionGroup("file not found",
    #         [FileNotFoundError("unknown file1"),
    #          FileNotFoundError("unknown file2")])])

Note that any empty branches of the tree will not be present.

Unhandled Exceptions

In the case that any exceptions are unhandled, then an ExceptionGroup containing the unhandled exceptions will be propogated up the call chain. Lets say we only handled FileNotFoundError

try:
    fn()
except* FileNotFoundError as eg:
    ...

The exception group containing just the unhandled KeyError will be propagated and can be handled elsewhere. As before, structure is maintained, so if the KeyError is nested somewhere, it will remain in the same position when propogated. Also, any empty branches which were fully handled will be removed from the exception group.

The subgroup and split methods

The ExceptionGroup class contains two methods: subgroup and split which can be used to manually work with the exception tree.

subgroup is used to select matching exceptions from the tree. You can pass a predicate function to subgroup and any exceptions that match the predicate will be selected. (predicate function is one that returns True or False)

This code will select the exceptions that have "file1" in the message

e = ExceptionGroup("multiple exceptions",
        [ExceptionGroup("file not found",
            [FileNotFoundError("unknown filename file1.txt"), 
             FileNotFoundError("unknown filename file2.txt")]), 
         KeyError("missing key")])

eg = e.subgroup(lambda ex: "file1" in str(ex))

# eg will be
# ExceptionGroup('multiple exceptions', 
#     [ExceptionGroup('file not found', 
#         [FileNotFoundError('unknown filename file1.txt')])])

As before, the structure of the tree is maintained and empty branches are removed.

split is similar to subgroup, but it returns two trees: one containing the matches and another containing the remaining nodes.

match,rest = e.split(lambda ex: "file1" in str(ex))

# match will be 
# ExceptionGroup('multiple exceptions', 
#     [ExceptionGroup('file not found', 
#         [FileNotFoundError('unknown filename file1.txt')])])

# rest will be
# ExceptionGroup('multiple exceptions', 
#     [ExceptionGroup('file not found', 
#         [FileNotFoundError('unknown filename file2.txt')]), 
#      KeyError('missing key')])

Python 3.11 internally uses split to decide which part of the exception group tree is matched in an except* and which parts should continue propagating.

Most of the time you will use except* instead of subgroup or split, but these two methods are available for the rare case when you need to select based on something other than type or want to manually process the exception group.