The Definitive Guide to Python Exceptions

Three years after my definitive guide on Python classic, static, class and abstract methods, it seems to be time for a new one. Here, I would like to dissect and discuss Python exceptions.

Dissecting the Base Exceptions

In Python, the base exception class is named BaseException. Being rarely used in any program or library, it ought to be considered as an implementation detail. But to discover how it's implemented, you can go and read Objects/exceptions.c in the CPython source code. In that file, what is interesting is to see that the BaseException class defines all the basic methods and attribute of exceptions. The basic well-known Exception class is then simply defined as a subclass of BaseException, nothing more:

/*
 *    Exception extends BaseException
 */
SimpleExtendsException(PyExc_BaseException, Exception,
                       "Common base class for all non-exit exceptions.");

 The only other exceptions that inherit directly from BaseException are GeneratorExit, SystemExit, and KeyboardInterrupt. All the other builtin exceptions inherit from Exception. The whole hierarchy can be seen by running pydoc2 exceptions or pydoc3 builtins.

Here are the graphs representing the builtin exceptions inheritance in Python 2 and Python 3 (generated using this script).

  

  

The BaseException.__init__ signature is actually BaseException.__init__(*args). This initialization method stores any arguments that are passed in the args attribute of the exception. This can be seen in the exceptions.c source code and is true for both Python 2 and Python 3:

static int
BaseException_init(PyBaseExceptionObject *self, PyObject *args, PyObject *kwds)
{
    if (!_PyArg_NoKeywords(Py_TYPE(self)->tp_name, kwds))
        return -1;

    Py_INCREF(args);
    Py_XSETREF(self->args, args);

    return 0;
}

The only place where this args attribute is used is in the BaseException.__str__ method. This method uses self.args to convert an exception to a string:

static PyObject *
BaseException_str(PyBaseExceptionObject *self)
{
    switch (PyTuple_GET_SIZE(self->args)) {
    case 0:
        return PyUnicode_FromString("");
    case 1:
        return PyObject_Str(PyTuple_GET_ITEM(self->args, 0));
    default:
        return PyObject_Str(self->args);
    }
}

This can be translated in Python to:

def __str__(self):
    if len(self.args) == 0:
        return ""
    if len(self.args) == 1:
        return str(self.args[0])
    return str(self.args)

Therefore, the message to display for an exception should be passed as the first and the only argument to the BaseException.__init__ method.

Defining Your Exceptions Properly

As you may already know, in Python, exceptions can be raised in any part of the program. The basic exception is called Exception and can be used anywhere in your program. In real life, however, no program nor library should ever raise Exception directly—it's not specific enough to be helpful.

Since all exceptions are expected to be derived from the base class Exception, this base class can easily be used as a catch-all:

try:
    do_something()
except Exception:
    # THis will catch any exception!
    print("Something terrible happened")

To define your own exceptions correctly, there are a few rules and best practices that you need to follow:

• Always inherit from (at least) Exception: 

  class MyOwnError(Exception):
      pass

  
  
• Leverage what we saw earlier about BaseException.__str__: it uses the first argument passed to BaseException.__init__ to be printed, so always calls BaseException.__init__ with only one argument.
• When building a library, define a base class inheriting from Exception. It will make it easier for consumers to catch any exception from the library:

  class ShoeError(Exception):
      """Basic exception for errors raised by shoes"""
  
  class UntiedShoelace(ShoeError):
      """You could fall"""
  
  class WrongFoot(ShoeError):
      """When you try to wear your left show on your right foot"""

  
  It then makes it easy to use except ShoeError when doing anything with that piece of code related to shoes. For example, Django does not do that for some of its exceptions, making it hard to catch "any exceptions raised by Django."
• Provide details about the error. This is extremely valuable to be able to correctly log errors or take further action and try to recover:

  class CarError(Exception):
      """Basic exception for errors raised by cars"""
      def init(self, car, msg=None):
          if msg is None:
              # Set some default useful error message
              msg = "An error occured with car %s" % car
          super(CarError, self).init(msg)
          self.car = car
  
  class CarCrashError(CarError):
      """When you drive too fast"""
      def init(self, car, other_car, speed):
          super(CarCrashError, self).init(
              car, msg="Car crashed into %s at speed %d" % (other_car, speed))
          self.speed = speed
          self.other_car = other_car

  
  Then, any code can inspect the exception to take further action: 

  try:
      drive_car(car)
  except CarCrashError as e:
      # If we crash at high speed, we call emergency
      if e.speed >= 30:
          call_911()

  
  For example, this is leveraged in Gnocchi to raise specific application exceptions (NoSuchArchivePolicy) on expected foreign key violations raised by SQL constraints: 

  try:
      with self.facade.writer() as session:
          session.add(m)
  except exception.DBReferenceError as e:
      if e.constraint == 'fk_metric_ap_name_ap_name':
          raise indexer.NoSuchArchivePolicy(archive_policy_name)
      raise

  
  
• Inherits from builtin exceptions types when it makes sense. This makes it easier for programs to not be specific to your application or library: 

  class CarError(Exception):
      """Basic exception for errors raised by cars"""
  
  class InvalidColor(CarError, ValueError):
      """Raised when the color for a car is invalid"""

  
  That allows many programs to catch errors in a more generic way without noticing your own defined type. If a program already knows how to handle a ValueError, it won't need any specific code nor modification.

Organization

There is no limitation on where and when you can define exceptions. As they are, after all, normal classes, they can be defined in any module, function, or class—even as closures.

Most libraries package their exceptions into a specific exception module: SQLAlchemy has them in sqlalchemy.exc, requests has them in requests.exceptions, Werkzeug has them in werkzeug.exceptions, etc.

That makes sense for libraries to export exceptions that way, as it makes it very easy for consumers to import their exception module and know where the exceptions are defined when writing code to handle errors.

This is not mandatory, and smaller Python modules might want to retain their exceptions into their sole module. Typically, if your module is small enough to be kept in one file, don't bother splitting your exceptions into a different file/module.

While this wisely applies to libraries, applications tend to be different beasts. Usually, they are composed of different subsystems, where each one might have its own set of exceptions. This is why I generally discourage going with only one exception module in an application, but to split them across the different parts of one's program. There might be no need of a special myapp.exceptions module.

For example, if your application is composed of an HTTP REST API defined into the module myapp.http and of a TCP server contained into myapp.tcp, it's likely they can both define different exceptions tied to their own protocol errors and cycle of life. Defining those exceptions in a myapp.exceptions module would just scatter the code for the sake of some useless consistency. If the exceptions are local to a file, just define them somewhere at the top of that file. It will simplify the maintenance of the code.

Wrapping Exceptions

Wrapping exception is the practice by which one exception is encapsulated into another:

class MylibError(Exception):
    """Generic exception for mylib"""
    def __init__(self, msg, original_exception)
        super(MylibError, self).__init__(msg + (": %s" % e))
        self.original_exception = original_exception

try:
    requests.get("http://example.com")
except requests.exceptions.ConnectionError as e:
     raise MylibError("Unable to connect", e)

This makes sense when writing a library which leverages other libraries. If a library uses requests and does not encapsulate requests exceptions into its own defined error classes, it will be a case of layer violation. Any application using your library might receive a requests.exceptions.ConnectionError, which is a problem because:

1. The application has no clue that the library was using requests and does not need/want to know about it.
2. The application will have to import requests.exceptions itself and therefore will depend on requests—even if it does not use it directly.
3. As soon as mylib changes from requests to e.g. httplib2, the application code catching requests exceptions will become irrelevant.

The Tooz library is a good example of wrapping, as it uses a driver-based approach and depends on a lot of different Python modules to talk to different backends (ZooKeeper, PostgreSQL, etcd…). Therefore, it wraps exception from other modules on every occasion into its own set of error classes. Python 3 introduced the raise from form to help with that, and that's what Tooz leverages to raise its own error.

It's also possible to encapsulate the original exception into a custom defined exception, as done above. That makes the original exception available for inspection easily.

Catching and Logging

When designing exceptions, it's important to remember that they should be targeted both at humans and computers. That's why they should include an explicit message, and embed as much information as possible. That will help to debug and write resilient programs that can pivot their behavior depending on the attributes of exception, as seen above.

Also, silencing exceptions completely is to be considered as bad practice. You should not write code like that:

try:
    do_something()
except Exception:
    # Whatever
    pass

Not having any kind of information in a program where an exception occurs is a nightmare to debug.

If you use (and you should) the logging library, you can use the exc_info parameter to log a complete traceback when an exception occurs, which might help debugging on severe and unrecoverable failure:

try:
    do_something()
except Exception:
    logging.getLogger().error("Something bad happened", exc_info=True)

Further Reading

If you understood everything so far, congratulations, you might be ready to handle exception in Python! If you want to have a broader scope on exceptions and what Python misses, I encourage you to read about condition systems and discover the generalization of exceptions—that I hope we'll see in Python one day!

I hope this will help you to build better libraries and application. Feel free to shoot any questions my way in the comments section!