.. _understanding_annotations: Understanding Annotations ========================= The Python programming language was not built with a static type system, but since Python 3.5 we have had a formal system for writing comments in Python code that are easy for linters to extract type information from. The result is the Python ecosystem has linters that can warn a developer when they're code is not type safe. This document focuses specifically on the standard ``mypy`` type checking linter. Type annotations are effectively ``glorified comments`` and are only enforced at static time. And like all comments are irrelevant at runtime! This means that when we write type annotations we do not have any guarantees that only apply at runtime and this is an important concept to consider when understanding how to make type safe code. Lesson 1: annotations are about what something has, not what something is ------------------------------------------------------------------------- Let's take the following function .. code-block:: python def double(num: int) -> int: ... At static time, the type of ``double`` is a "Callable object that takes one argument (either positionally or as a keyword "num") that is an integer and returns an integer". The type system can only describe what attributes are on the type, it cannot enforce identity. So this function cannot specify whether it's the same or a different integer that is returned from the function, only that whatever is returned can be used like an integer. This "limitation" applies to all types, for example: .. code-block:: python class MyObj: def clone(self) -> Self: ... This ``clone`` method must return an object that matches the type of the instance it's called on, but there is no restriction on whether that object is the same instance or a new instance. Lesson 2: What something has is a description of its API surface ---------------------------------------------------------------- Python has an "object model" that allows us to change the behaviour of an object by implementing special "dunder" attributes on that object - https://docs.python.org/3/reference/datamodel.html For example an object that implements ``__call__`` is a callable object. You can actually see this if you create a function in Python and see what attributes it has:: Python 3.12.9 (main, Feb 17 2025, 09:33:34) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> def my_function(): pass ... >>> dir(my_function) ['__annotations__', '__builtins__', **'__call__'**, '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__getstate__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__type_params__'] >>> Every object in python has ``object`` as its ancestor and so (with some extra nuance we'll skip here) everything at least has the following on it: .. code-block:: python https://github.com/python/typeshed/blob/c01e731dd1fb6d6ae3c9ef00e2f19a89ab1ffb24/stdlib/builtins.pyi#L102C1-L135C62 class object: __doc__: str | None __dict__: dict[str, Any] __module__: str __annotations__: dict[str, Any] @property def __class__(self) -> type[Self]: ... @__class__.setter def __class__(self, type: type[Self], /) -> None: ... def __init__(self) -> None: ... def __new__(cls) -> Self: ... # N.B. ``object.__setattr__`` and ``object.__delattr__`` are heavily special-cased by type checkers. # Overriding them in subclasses has different semantics, even if the override has an identical signature. def __setattr__(self, name: str, value: Any, /) -> None: ... def __delattr__(self, name: str, /) -> None: ... def __eq__(self, value: object, /) -> bool: ... def __ne__(self, value: object, /) -> bool: ... def __str__(self) -> str: ... # noqa: Y029 def __repr__(self) -> str: ... # noqa: Y029 def __hash__(self) -> int: ... def __format__(self, format_spec: str, /) -> str: ... def __getattribute__(self, name: str, /) -> Any: ... def __sizeof__(self) -> int: ... # return type of pickle methods is rather hard to express in the current type system # see #6661 and https://docs.python.org/3/library/pickle.html#object.__reduce__ def __reduce__(self) -> str | tuple[Any, ...]: ... def __reduce_ex__(self, protocol: SupportsIndex, /) -> str | tuple[Any, ...]: ... if sys.version_info >= (3, 11): def __getstate__(self) -> object: ... def __dir__(self) -> Iterable[str]: ... def __init_subclass__(cls) -> None: ... @classmethod def __subclasshook__(cls, subclass: type, /) -> bool: ... .. note:: 📓 This is why ``object`` and ``typing.Any`` are both wildcards because ``Any`` represents the absence of type checking and everything in python is an ``object`` Whenever we assign a type annotation to a variable the static type checker will tell us if that variable satisfies that annotation or not. So let's say we have this protocol: .. code-block:: python from typing import Protocol class Processor(Protocol): def process(self) -> int | str: ... Then a variable satisfies that protocol as long as it has a ``process`` method on it that can be called with with no arguments and always returns either a ``str`` or an ``int``. Whereas if we had this class: .. code-block:: python class CommonLogic: def something(self) -> str: ... Then a variable can be annotated as an instance of ``CommonLogic`` if this class is one of that object's base classes (it's inside the object's `mro `_) .. note:: 📓 Note that liskov checks only happen when a class is defined and when comparing an annotation against some variable, if a variable is a valid subclass, mypy won't check if that subclass has any liskov violations when it says the variable is the correct type (a liskov violation is when the signature on the subclass is not compatible with the signature on the parent and so an instance of the subclass cannot be substituted in where an instance of the Parent is expected) Lesson 3: Compatible function signatures ---------------------------------------- Understanding python signatures teaches us a lot about type annotations that applies to variables as well (because an input to a function is a variable like any other from the perspective of a function implementation!) First of all, this: .. code-block:: python def my_function(one: int) -> bool: ... Is functionally the same as: .. code-block:: python from collections.abc import Callable my_function: Callable[[int], bool] = lambda one: True And functionally the same as: .. code-block:: python class _MyFunction: def __call__(self, one: int) -> bool: ... my_function = _MyFunction() In all these cases ``my_function`` is an object at runtime with a dunder call method that takes in a single argument (either positionally or named "one") that is an int and will return a boolean. There are two main ways of creating a type annotation that represents a callable object: with ``collections.abc.Callable`` or with a ``Protocol`` that implements a dunder call method. In many cases defining a callable ``Protocol`` is going to age better than creating a ``Callable`` annotation because protocols can be named and can express much more complicated function signatures. In the examples above we could have: .. code-block:: python type MyFunction = Callable[[int], bool] Like what we use to give a type to the first argument in the example with a lambda expression, but with this annotation we can't specify that the name of the first argument is "one", whereas we can do that if we used a protocol: .. code-block:: python class MyFunction(Protocol): def __call__(self, one: int) -> bool: ... .. note:: 📓 Note that the ``...`` is valid python syntax and is equivalent to ``pass`` but it only makes sense on Protocol methods or methods annotated with ``@abc.abstractmethod`` and this document is avoiding writing implementations for brevity and because what a function does is statically irrelevant when thinking about a function from where we call a function. Because we can write a full signature with a protocol we can do nice things like use ``/`` , ``*``, ``*args``, ``**kwargs`` and define when an argument has a default. .. code-block:: python # On a protocol, we can use ... to signify a variable has a default # without repeating any expectation about what value the default should have class Action(Protocol): def __call__(self, number: int, *, some_option: str = ...) -> None: ... Lesson 4: required and optional arguments ----------------------------------------- For example to make a protocol where the first positional argument is not named we would do: .. code-block:: python class MyFunction(Protocol): def __call__(self, one: int, /) -> bool: ... Because the ``/`` says that all arguments before the slash are "positional only" and cannot be passed in with a name. Let's see this in action: .. code-block:: python # All the examples below satisfies this Protocol class MyFunction(Protocol): def __call__(self, one: int, /) -> bool: ... def _example_exact_match(one: int, /) -> bool: ... def _example_positional_only_but_different_name(two: int, /) -> bool: ... def _example_can_be_positional_only_but_not_enforced(one: int) -> bool: ... def _example_has_additional_arg_that_is_not_required( one: int, /, two: str = "" ) -> bool: ... That last example is an interesting one. It's best to think about substitution by asking yourself when you have a function that takes in some specific type, will it work if I pass in my object? .. code-block:: python def takes_a_func(func: MyFunction) -> bool: return func(1) So in this case when we do: .. code-block:: python takes_a_func(_example_has_additional_args_but_they_are_not_required) Our implementation of ``takes_a_func`` will work because when we pass only one positional argument into ``_example_has_additional_args_but_they_are_not_required`` the body of that function will take the value we provided and set the other argument to its default value of an empty string. But if we instead had: .. code-block:: python def _example_has_additional_arg_that_is_required( one: int, /, two: str ) -> bool: ... When we do: .. code-block:: python takes_a_func(_example_has_additional_args_but_they_are_not_required) We get ``TypeError: _example_has_additional_arg_that_is_required() missing 1 required positional argument: 'two'`` at runtime! Lesson 5: inputs are contravariant ---------------------------------- Another thing to consider with compatibility with signatures (and also keep in mind that a class constructor is a callable object that returns an instance of that class) is whether the implementation that is provided for an annotation can handle the inputs the annotation say it will support. For example consider this for the rest of the examples in this section .. code-block:: python from typing import Protocol import attrs @attrs.frozen class Common: number: int class Doubler(Common): def double(self) -> int: return self.number * 2 class Tripler(Common): def triple(self) -> int: return self.number * 3 class ProcessCommon(Protocol): def __call__(self, logic: Common, /) -> int: ... class ProcessDoubler(Protocol): def __call__(self, logic: Doubler, /) -> int: ... class ProcessTripler(Protocol): def __call__(self, logic: Tripler, /) -> int: ... Let's create some implementations that satisfy our protocols: .. code-block:: python # Satisfies ProcessDoubler def process_double(logic: Doubler, /) -> int: return logic.double() # Satisfies ProcessTripler def process_triple(logic: Tripler, /) -> int: return logic.triple() So, let's play a game of substitution! .. code-block:: python def takes_processor(processor: ProcessCommon) -> int: return processor(Doubler(number=2)) At runtime this works .. code-block:: python takes_processor(process_double) But this does not .. code-block:: python takes_processor(process_triple) And **neither** work at static time!:: example.py:47: error: Argument 1 to "takes_processor" has incompatible type "Callable[[Any, Doubler], int]"; expected "ProcessCommon" [arg-type] example.py:49: error: Argument 1 to "takes_processor" has incompatible type "Callable[[Any, Tripler], int]"; expected "ProcessCommon" [arg-type] The reason for this is a concept called "contravariance". What this means is that the body of a function cannot expect additional attributes on the input and the caller must provide at least what is expected. So when we compare ``process_double`` to ``ProcessCommon`` we see that ``process_double`` expects the input to have a method on it that ``ProcessCommon`` doesn't know about. You can see this in how there are no mypy errors for the implementation of ``takes_processor`` because that implementation is completely type safe. The only two requirements for ``ProcessCommon`` is that the object we pass in has ``Common`` in its mro somewhere which is true for instances of ``Common``, ``Doubler`` and ``Tripler``. So we can pass in a function that doesn't handle ``Common`` or ``Tripler``, we create a situation where it's statically correct to pass in an object our implementation doesn't know how to handle. Outside the function, we cannot expect the function will handle an object it doesn't say it can handle. So passing ``Callable[[Doubler], int]`` where we expect ``Callable[[Common], int]`` means we are using a function that doesn't handle all the possibilities expressed by the type we need to satisfy. Lesson 6: Outputs are covariant ------------------------------- The 'opposite' concept to contravariance is "covariance" which is what return annotations are. An implementation may return more than what is expected, but the caller is not guaranteed that the returned object has any extra attributes without additional type narrowing. .. note:: 📓 Remember that an annotation is a description of our expectations for the benefit of humans and computers. These are glorified comments and have no influence over what the variable actually has at runtime! So let's consider our classes from before: .. code-block:: python from typing import Protocol import attrs @attrs.frozen class Common: number: int class Doubler(Common): def double(self) -> int: return self.number * 2 class Tripler(Common): def triple(self) -> int: return self.number * 3 And implement a function that returns a Common: .. code-block:: python # Let's ignore the implementation for a second and focus on the signature def makes_common() -> Common: ... When we use this function we expect that the variable we get back has ``Common`` in its mro and thus has a read-only property on it called ``number`` that is an integer. So we expect to always be able to do this: .. code-block:: python made = makes_common() print(made.number) This means all of these implementations are type safe: .. code-block:: python def make_common() -> Common: return Common(number=1) def make_doubler() -> Common: return Doubler(number=2) def make_tripler() -> Common: return Tripler(number=3) And if we change the return annotations to be more specific and create: .. code-block:: python def make_common() -> Common: return Common(number=1) def make_doubler() -> Doubler: return Doubler(number=2) def make_tripler() -> Tripler: return Tripler(number=3) They all still satisfy: .. code-block:: python class MakesCommon(Protocol): def __call__(self) -> Common: ... Because whilst the object being returned in ``make_doubler`` and ``make_tripler`` has more that what is required, they still return something that has ``Common`` in it's mro. .. note:: 📓 variance is a concept for understanding substitution rather than about understanding a specific implementation. So to drive the point home, return types being covariant means that if we expect an object to return a type, we can provide an implementation that returns a more specialised form of that return type. If we have an object that expects a specific type as an input, we can only provide an implementation that expects a more generalised form of that input type. So if I expect a function that returns a ``Common`` I can provide an implementation that returns a ``Tripler``: .. code-block:: python class ReturnsCommon(Protocol): def __call__(self) -> Common: ... # Satisfies ReturnsCommon class ReturnsTripler(Protocol): def __call__(self) -> Tripler: ... But if I have a function that returns a ``Tripler`` I cannot provide an implementation that returns a ``Common``: .. code-block:: python class ReturnsTripler(Protocol): def __call__(self) -> Tripler: ... # Does not satisfy ReturnsTripler class ReturnsCommon(Protocol): def __call__(self) -> Common: ... Because a variable of type ``Common`` is possible to be an instance of ``Tripler`` but a variable with type ``Tripler`` cannot be a subclass of ``Common`` that isn't also a subclass of ``Tripler``. And if I have a callable that expects a ``Common`` as input I cannot substitute in an implementation that expects a ``Tripler`` as an input: .. code-block:: python class TakesCommon(Protocol): def __call__(self, obj: Common) -> None: ... # Does not satisfy TakesCommon class TakesTripler(Protocol): def __call__(self, obj: Tripler) -> None: ... But if I expect a function that takes a ``Tripler`` I can provide an implementation that takes a ``Common`` .. code-block:: python class TakesTripler(Protocol): def __call__(self, obj: Tripler) -> None: ... # Satisfies TakesTripler class TakesCommon(Protocol): def __call__(self, obj: Common) -> None: ... Because a function that expects a ``Common`` means a ``Tripler`` is a possibility in that implementation but a function that expects a ``Tripler`` does not expect any subclass of ``Common`` that isn't also a subclass of ``Tripler``. Lesson 7: What about outputs from contravariant types? ------------------------------------------------------ So let's see this in action! Given the following for this example: .. code-block:: python from typing import Protocol import attrs class HasIntOrStr(Protocol): @property def number(self) -> int | str: ... @attrs.frozen class HasOnlyInt: number: int @attrs.frozen class HasOnlyStr: number: str @attrs.frozen class HasIntOrStrOrList: number: int | str | list[int] class CreatesIntOrString(Protocol): def __call__(self) -> HasIntOrStr: ... class CreatesOnlyInt(Protocol): def __call__(self) -> HasOnlyInt: ... class CreatesOnlyStr(Protocol): def __call__(self) -> HasOnlyStr: ... With some implementations: .. code-block:: python # Satisfies CreatesIntOrString and CreatesOnlyInt def create_only_int() -> HasOnlyInt: return HasOnlyInt(number=1) # Satisfies CreatesIntOrString and CreatesOnlyStr def create_only_str() -> HasOnlyStr: return HasOnlyStr(number="asdf") # Satisfies none of them def create_int_or_str_or_list() -> HasIntOrStrOrList: return HasIntOrStrOrList(number=[1, 2]) And something that expects a ``CreatesIntOrString``: .. code-block:: python from typing import assert_never def takes_creator(creator: CreatesIntOrString) -> None: created = creator() match created.number: case int(): print(created.number + 1) case str(): print(created.number.upper()) case _: assert_never(created.number) And use it .. code-block:: python takes_creator(create_only_int) takes_creator(create_only_str) takes_creator(create_int_or_str_or_list) The first two are completely fine but that last one is not!:: example.py:65: error: Argument 1 to "takes_creator" has incompatible type "Callable[[], HasIntOrStrOrList]"; expected "CreatesIntOrString" [arg-type] This is because ``creator`` is an input and so it is contravariant, and a callable object is generic to it's parameters and return type. So we cannot expect the implementation to know it needs to deal with this extra ``list`` from the return type. This function says it only knows what to do with objects where the number is an int or str, it does not say it implements the ability to handle an object where the number is a list!