Understanding Annotations

The Python programming language was not built with a static type system, but since Python 3.5 we have had a formal system for writing comments in Python code that are easy for linters to extract type information from.

The result is the Python ecosystem has linters that can warn a developer when they’re code is not type safe. This document focuses specifically on the standard mypy type checking linter.

Type annotations are effectively glorified comments and are only enforced at static time. And like all comments are irrelevant at runtime!

This means that when we write type annotations we do not have any guarantees that only apply at runtime and this is an important concept to consider when understanding how to make type safe code.

Lesson 1: annotations are about what something has, not what something is

Let’s take the following function

def double(num: int) -> int: ...

At static time, the type of double is a “Callable object that takes one argument (either positionally or as a keyword “num”) that is an integer and returns an integer”.

The type system can only describe what attributes are on the type, it cannot enforce identity. So this function cannot specify whether it’s the same or a different integer that is returned from the function, only that whatever is returned can be used like an integer.

This “limitation” applies to all types, for example:

class MyObj:
    def clone(self) -> Self: ...

This clone method must return an object that matches the type of the instance it’s called on, but there is no restriction on whether that object is the same instance or a new instance.

Lesson 2: What something has is a description of its API surface

Python has an “object model” that allows us to change the behaviour of an object by implementing special “dunder” attributes on that object - https://docs.python.org/3/reference/datamodel.html

For example an object that implements __call__ is a callable object.

You can actually see this if you create a function in Python and see what attributes it has:

Python 3.12.9 (main, Feb 17 2025, 09:33:34) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def my_function(): pass
...
>>> dir(my_function)
['__annotations__', '__builtins__', **'__call__'**, '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__getstate__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__type_params__']
>>>

Every object in python has object as its ancestor and so (with some extra nuance we’ll skip here) everything at least has the following on it:

https://github.com/python/typeshed/blob/c01e731dd1fb6d6ae3c9ef00e2f19a89ab1ffb24/stdlib/builtins.pyi#L102C1-L135C62
class object:
    __doc__: str | None
    __dict__: dict[str, Any]
    __module__: str
    __annotations__: dict[str, Any]
    @property
    def __class__(self) -> type[Self]: ...
    @__class__.setter
    def __class__(self, type: type[Self], /) -> None: ...
    def __init__(self) -> None: ...
    def __new__(cls) -> Self: ...
    # N.B. ``object.__setattr__`` and ``object.__delattr__`` are heavily special-cased by type checkers.
    # Overriding them in subclasses has different semantics, even if the override has an identical signature.
    def __setattr__(self, name: str, value: Any, /) -> None: ...
    def __delattr__(self, name: str, /) -> None: ...
    def __eq__(self, value: object, /) -> bool: ...
    def __ne__(self, value: object, /) -> bool: ...
    def __str__(self) -> str: ...  # noqa: Y029
    def __repr__(self) -> str: ...  # noqa: Y029
    def __hash__(self) -> int: ...
    def __format__(self, format_spec: str, /) -> str: ...
    def __getattribute__(self, name: str, /) -> Any: ...
    def __sizeof__(self) -> int: ...
    # return type of pickle methods is rather hard to express in the current type system
    # see #6661 and https://docs.python.org/3/library/pickle.html#object.__reduce__
    def __reduce__(self) -> str | tuple[Any, ...]: ...
    def __reduce_ex__(self, protocol: SupportsIndex, /) -> str | tuple[Any, ...]: ...
    if sys.version_info >= (3, 11):
        def __getstate__(self) -> object: ...

    def __dir__(self) -> Iterable[str]: ...
    def __init_subclass__(cls) -> None: ...
    @classmethod
    def __subclasshook__(cls, subclass: type, /) -> bool: ...

Note

📓 This is why object and typing.Any are both wildcards because Any represents the absence of type checking and everything in python is an object

Whenever we assign a type annotation to a variable the static type checker will tell us if that variable satisfies that annotation or not.

So let’s say we have this protocol:

from typing import Protocol

class Processor(Protocol):
    def process(self) -> int | str: ...

Then a variable satisfies that protocol as long as it has a process method on it that can be called with with no arguments and always returns either a str or an int.

Whereas if we had this class:

class CommonLogic:
    def something(self) -> str: ...

Then a variable can be annotated as an instance of CommonLogic if this class is one of that object’s base classes (it’s inside the object’s mro)

Note

📓 Note that liskov checks only happen when a class is defined and when comparing an annotation against some variable, if a variable is a valid subclass, mypy won’t check if that subclass has any liskov violations when it says the variable is the correct type (a liskov violation is when the signature on the subclass is not compatible with the signature on the parent and so an instance of the subclass cannot be substituted in where an instance of the Parent is expected)

Lesson 3: Compatible function signatures

Understanding python signatures teaches us a lot about type annotations that applies to variables as well (because an input to a function is a variable like any other from the perspective of a function implementation!)

First of all, this:

def my_function(one: int) -> bool: ...

Is functionally the same as:

from collections.abc import Callable

my_function: Callable[[int], bool] = lambda one: True

And functionally the same as:

class _MyFunction:
    def __call__(self, one: int) -> bool: ...

my_function = _MyFunction()

In all these cases my_function is an object at runtime with a dunder call method that takes in a single argument (either positionally or named “one”) that is an int and will return a boolean.

There are two main ways of creating a type annotation that represents a callable object: with collections.abc.Callable or with a Protocol that implements a dunder call method.

In many cases defining a callable Protocol is going to age better than creating a Callable annotation because protocols can be named and can express much more complicated function signatures.

In the examples above we could have:

type MyFunction = Callable[[int], bool]

Like what we use to give a type to the first argument in the example with a lambda expression, but with this annotation we can’t specify that the name of the first argument is “one”, whereas we can do that if we used a protocol:

class MyFunction(Protocol):
    def __call__(self, one: int) -> bool: ...

Note

📓 Note that the ... is valid python syntax and is equivalent to pass but it only makes sense on Protocol methods or methods annotated with @abc.abstractmethod and this document is avoiding writing implementations for brevity and because what a function does is statically irrelevant when thinking about a function from where we call a function.

Because we can write a full signature with a protocol we can do nice things like use / , *, *args, **kwargs and define when an argument has a default.

# On a protocol, we can use ... to signify a variable has a default
# without repeating any expectation about what value the default should have
class Action(Protocol):
    def __call__(self, number: int, *, some_option: str = ...) -> None: ...

Lesson 4: required and optional arguments

For example to make a protocol where the first positional argument is not named we would do:

class MyFunction(Protocol):
    def __call__(self, one: int, /) -> bool: ...

Because the / says that all arguments before the slash are “positional only” and cannot be passed in with a name.

Let’s see this in action:

# All the examples below satisfies this Protocol
class MyFunction(Protocol):
    def __call__(self, one: int, /) -> bool: ...


def _example_exact_match(one: int, /) -> bool: ...

def _example_positional_only_but_different_name(two: int, /) -> bool: ...

def _example_can_be_positional_only_but_not_enforced(one: int) -> bool: ...

def _example_has_additional_arg_that_is_not_required(
    one: int, /, two: str = ""
) -> bool: ...

That last example is an interesting one. It’s best to think about substitution by asking yourself when you have a function that takes in some specific type, will it work if I pass in my object?

def takes_a_func(func: MyFunction) -> bool:
    return func(1)

So in this case when we do:

takes_a_func(_example_has_additional_args_but_they_are_not_required)

Our implementation of takes_a_func will work because when we pass only one positional argument into _example_has_additional_args_but_they_are_not_required the body of that function will take the value we provided and set the other argument to its default value of an empty string.

But if we instead had:

def _example_has_additional_arg_that_is_required(
    one: int, /, two: str
) -> bool: ...

When we do:

takes_a_func(_example_has_additional_args_but_they_are_not_required)

We get TypeError: _example_has_additional_arg_that_is_required() missing 1 required positional argument: 'two' at runtime!

Lesson 5: inputs are contravariant

Another thing to consider with compatibility with signatures (and also keep in mind that a class constructor is a callable object that returns an instance of that class) is whether the implementation that is provided for an annotation can handle the inputs the annotation say it will support.

For example consider this for the rest of the examples in this section

from typing import Protocol
import attrs


@attrs.frozen
class Common:
    number: int


class Doubler(Common):
    def double(self) -> int:
        return self.number * 2

class Tripler(Common):
    def triple(self) -> int:
        return self.number * 3


class ProcessCommon(Protocol):
    def __call__(self, logic: Common, /) -> int: ...

class ProcessDoubler(Protocol):
    def __call__(self, logic: Doubler, /) -> int: ...

class ProcessTripler(Protocol):
    def __call__(self, logic: Tripler, /) -> int: ...

Let’s create some implementations that satisfy our protocols:

# Satisfies ProcessDoubler
def process_double(logic: Doubler, /) -> int:
    return logic.double()

# Satisfies ProcessTripler
def process_triple(logic: Tripler, /) -> int:
    return logic.triple()

So, let’s play a game of substitution!

def takes_processor(processor: ProcessCommon) -> int:
    return processor(Doubler(number=2))

At runtime this works

takes_processor(process_double)

But this does not

takes_processor(process_triple)

And neither work at static time!:

example.py:47: error: Argument 1 to "takes_processor" has incompatible type
    "Callable[[Any, Doubler], int]"; expected "ProcessCommon"  [arg-type]

example.py:49: error: Argument 1 to "takes_processor" has incompatible type
    "Callable[[Any, Tripler], int]"; expected "ProcessCommon"  [arg-type]

The reason for this is a concept called “contravariance”. What this means is that the body of a function cannot expect additional attributes on the input and the caller must provide at least what is expected.

So when we compare process_double to ProcessCommon we see that process_double expects the input to have a method on it that ProcessCommon doesn’t know about.

You can see this in how there are no mypy errors for the implementation of takes_processor because that implementation is completely type safe. The only two requirements for ProcessCommon is that the object we pass in has Common in its mro somewhere which is true for instances of Common, Doubler and Tripler.

So we can pass in a function that doesn’t handle Common or Tripler, we create a situation where it’s statically correct to pass in an object our implementation doesn’t know how to handle.

Outside the function, we cannot expect the function will handle an object it doesn’t say it can handle.

So passing Callable[[Doubler], int] where we expect Callable[[Common], int] means we are using a function that doesn’t handle all the possibilities expressed by the type we need to satisfy.

Lesson 6: Outputs are covariant

The ‘opposite’ concept to contravariance is “covariance” which is what return annotations are.

An implementation may return more than what is expected, but the caller is not guaranteed that the returned object has any extra attributes without additional type narrowing.

Note

📓 Remember that an annotation is a description of our expectations for the benefit of humans and computers. These are glorified comments and have no influence over what the variable actually has at runtime!

So let’s consider our classes from before:

from typing import Protocol
import attrs

@attrs.frozen
class Common:
    number: int

class Doubler(Common):
    def double(self) -> int:
        return self.number * 2

class Tripler(Common):
    def triple(self) -> int:
        return self.number * 3

And implement a function that returns a Common:

# Let's ignore the implementation for a second and focus on the signature
def makes_common() -> Common: ...

When we use this function we expect that the variable we get back has Common in its mro and thus has a read-only property on it called number that is an integer. So we expect to always be able to do this:

made = makes_common()
print(made.number)

This means all of these implementations are type safe:

def make_common() -> Common:
    return Common(number=1)

def make_doubler() -> Common:
    return Doubler(number=2)

def make_tripler() -> Common:
    return Tripler(number=3)

And if we change the return annotations to be more specific and create:

def make_common() -> Common:
    return Common(number=1)

def make_doubler() -> Doubler:
    return Doubler(number=2)

def make_tripler() -> Tripler:
    return Tripler(number=3)

They all still satisfy:

class MakesCommon(Protocol):
    def __call__(self) -> Common: ...

Because whilst the object being returned in make_doubler and make_tripler has more that what is required, they still return something that has Common in it’s mro.

Note

📓 variance is a concept for understanding substitution rather than about understanding a specific implementation.

So to drive the point home, return types being covariant means that if we expect an object to return a type, we can provide an implementation that returns a more specialised form of that return type.

If we have an object that expects a specific type as an input, we can only provide an implementation that expects a more generalised form of that input type.

So if I expect a function that returns a Common I can provide an implementation that returns a Tripler:

class ReturnsCommon(Protocol):
    def __call__(self) -> Common: ...

# Satisfies ReturnsCommon
class ReturnsTripler(Protocol):
    def __call__(self) -> Tripler: ...

But if I have a function that returns a Tripler I cannot provide an implementation that returns a Common:

class ReturnsTripler(Protocol):
    def __call__(self) -> Tripler: ...

# Does not satisfy ReturnsTripler
class ReturnsCommon(Protocol):
    def __call__(self) -> Common: ...

Because a variable of type Common is possible to be an instance of Tripler but a variable with type Tripler cannot be a subclass of Common that isn’t also a subclass of Tripler.

And if I have a callable that expects a Common as input I cannot substitute in an implementation that expects a Tripler as an input:

class TakesCommon(Protocol):
    def __call__(self, obj: Common) -> None: ...

# Does not satisfy TakesCommon
class TakesTripler(Protocol):
    def __call__(self, obj: Tripler) -> None: ...

But if I expect a function that takes a Tripler I can provide an implementation that takes a Common

class TakesTripler(Protocol):
    def __call__(self, obj: Tripler) -> None: ...

# Satisfies TakesTripler
class TakesCommon(Protocol):
    def __call__(self, obj: Common) -> None: ...

Because a function that expects a Common means a Tripler is a possibility in that implementation but a function that expects a Tripler does not expect any subclass of Common that isn’t also a subclass of Tripler.

Lesson 7: What about outputs from contravariant types?

So let’s see this in action! Given the following for this example:

from typing import Protocol
import attrs


class HasIntOrStr(Protocol):
    @property
    def number(self) -> int | str: ...


@attrs.frozen
class HasOnlyInt:
    number: int

@attrs.frozen
class HasOnlyStr:
    number: str

@attrs.frozen
class HasIntOrStrOrList:
    number: int | str | list[int]



class CreatesIntOrString(Protocol):
    def __call__(self) -> HasIntOrStr: ...

class CreatesOnlyInt(Protocol):
    def __call__(self) -> HasOnlyInt: ...

class CreatesOnlyStr(Protocol):
    def __call__(self) -> HasOnlyStr: ...

With some implementations:

# Satisfies CreatesIntOrString and CreatesOnlyInt
def create_only_int() -> HasOnlyInt:
    return HasOnlyInt(number=1)

# Satisfies CreatesIntOrString and CreatesOnlyStr
def create_only_str() -> HasOnlyStr:
    return HasOnlyStr(number="asdf")

# Satisfies none of them
def create_int_or_str_or_list() -> HasIntOrStrOrList:
    return HasIntOrStrOrList(number=[1, 2])

And something that expects a CreatesIntOrString:

from typing import assert_never

def takes_creator(creator: CreatesIntOrString) -> None:
    created = creator()
    match created.number:
        case int():
        print(created.number + 1)
        case str():
        print(created.number.upper())
        case _:
        assert_never(created.number)

And use it

takes_creator(create_only_int)
takes_creator(create_only_str)
takes_creator(create_int_or_str_or_list)

The first two are completely fine but that last one is not!:

example.py:65: error: Argument 1 to "takes_creator" has incompatible type
    "Callable[[], HasIntOrStrOrList]"; expected "CreatesIntOrString"  [arg-type]

This is because creator is an input and so it is contravariant, and a callable object is generic to it’s parameters and return type. So we cannot expect the implementation to know it needs to deal with this extra list from the return type. This function says it only knows what to do with objects where the number is an int or str, it does not say it implements the ability to handle an object where the number is a list!