.. _understanding_typevars: Understanding TypeVars ====================== Static type checking is by definition a static system of checks, which means anything that is determined at runtime is irrelevant to the type checker! However it can be necessary for the types discovered by the linter to depend on how a piece of code is called. For example, it can be common to define code that supports many different types but only operates on one of those types at a time and this is the situation where a TypeVar becomes very useful. Essentially, a TypeVar is an annotation where the concrete type it represents is determined by the caller rather than by the implementation. For example .. code-block:: python import attrs @attrs.frozen class Container[T_Item]: item: T_Item def get_item_twice(self) -> tuple[T_Item, T_Item]: return (self.item, self.item) # Revealed type is "tuple[builtins.str, builtins.str]" reveal_type(Container("asdf").get_item_twice()) # Revealed type is "tuple[builtins.int, builtins.int]" reveal_type(Container(1).get_item_twice()) # Revealed type is "tuple[builtins.bool, builtins.bool]" reveal_type(Container(True).get_item_twice()) Think of it like simple algebra where we need to "solve for X". .. note:: ❗ A TypeVar needs to be **bound** which means any time there is a TypeVar in an output, there needs to be a chance for the caller to provide the specific type that the type var represents in that context. So either as part of the definition of the enclosing class: .. code-block:: python class MyProtocol[T_Item](Protocol): @property def item(self) -> T_Item: ... @attrs.frozen class MyBaseClass[T_Item]: item: T_Item Or as an input to the function returning the TypeVar: .. code-block:: python class MyProtocol(Protocol): def process[T_Item](self, item: T_Item) -> T_Item: ... def process[T_Item](item: T_Item) -> T_Item: raise NotImplementedError() def extract[T_Item](container: Container[T_Item]) -> T_Item: raise NotImplementedError() TypeVars also have a concept called "variance". This is relevant when you have a bound type variable (like ``T_Vehicle``) and some specific implementations of it (like ``Car`` or ``Bicycle``), which, alongside the basic features guaranteed by ``Vehicle``, also have some "extra" attributes and methods; an extra "API surface", if you will. There are three types of this variance: - **covariant** - extra API surface is kept - **contravariant** - extra API surface is forgotten - **invariant** - is neither just covariant or just contravariant When you aren't using `PEP 695 `_ (inline squarebrackets) syntax or ``infer_variance=True`` when creating the TypeVar, mypy will enforce the variance of the TypeVar based on: - Inputs are always contravariant - Outputs are always covariant This means: - A type var that's only ever an output must be defined with ``covariant=True`` - A type var that's only ever an input must be defined with ``contravariant=True`` - A type var that appears as an input and as an output cannot be covariant or contravariant, so it must be invariant. .. note:: 🤔 Note that a TypeVar is only necessary when the API surface of an object changes based on the implementation. If a different implementation doesn't change what attributes or methods are available, then it can be represented as a Protocol. Giving an upper bound to a ``TypeVar`` -------------------------------------- By default a TypeVar statically has no methods or attributes on it at the point it appears to the linter as a TypeVar. .. code-block:: python def pass_through[T_Item](item: T_Item) -> T_Item: # at this point item statically has no attributes or methods on it return item my_variable: int = 1 # my_variable has all the attributes/methods that an int has after_pass_through = pass_through(my_variable) # after_pass_through is typed as the type use for the input, so it also # statically has all the attributes/methods that an int has It's possible to make it so that the code thinking about the TypeVar has specific attributes and methods that are statically guaranteed to be available: .. code-block:: python from typing import Protocol class HasProcess(Protocol): def process(self) -> None: ... def pass_through[T_Item: HasProcess](item: T_Item) -> T_Item: # Our type var is bound to ``HasProcess`` # This means whilst the object passed in is allowed to have any number # of additional attributes and methods, we are guaranteed that it at least # has a "process" method that takes in no arguments and returns None item.process() return item # Error, int has no "process" on it! pass_through(1) @attrs.define class MyProcessor: number: int def process(self) -> None: print(self.number) # Valid because an instance of MyProcessor has a method on it # called "process" that can be called with no paramaters and returns a None pass_through(MyProcessor(10)) There is also the ability to constrain the TypeVar to exact types rather than subtypes of the upper bound (as long as you have at least two constraints): .. code-block:: python from typing import TypeVar T_Item = TypeVar("T_Item", str, int) # Or with PEP 695 def item_twice[T_Item: (str, int)](item: T_Item) -> T_Item: return item + item The reason to do this rather than just saying the function takes and returns ``str | int`` is that this way we know that if we pass in eg a ``str`` we definitely get a ``str`` back. Prefer to not use contravariant type vars ----------------------------------------- As a general rule contravariant type vars impose restrictions when extending classes and can be avoided by replacing: .. code-block:: python from typing import Protocol class Thing[T_Item: Item](Protocol): def do_something(self, item: T_Item) -> None: ... With: .. code-block:: python from typing import Protocol class Thing[T_Item: Item](Protocol): @property def item(self) -> T_Item: ... def do_something(self) -> None: ... Such that the changeable API surface being acted on is separate from the signature of the function doing the action. As mentioned in :ref:`understanding_annotations`, a contravariant type var is used to represent a value where extra API surface is always dropped: .. code-block:: python import dataclasses from typing import TYPE_CHECKING, Protocol, TypeVar, cast T_COT_Item = TypeVar("T_COT_Item", contravariant=True) class Recorder(Protocol[T_COT_Item]): def record(self, item: T_COT_Item) -> None: ... @dataclasses.dataclass(frozen=True) class ItemA: a: int @dataclasses.dataclass(frozen=True) class ItemB(ItemA): b: int class RecorderA: def record(self, item: ItemA) -> None: print(item.a) class RecorderB: def record(self, item: ItemB) -> None: print(item.a, item.b) def record_things(recorder: Recorder[ItemA]) -> None: recorder.record(ItemA(a=1)) # This fails because Recorder[ItemB] cannot be used where Recorder[ItemA] is required record_things(RecorderB()) if TYPE_CHECKING: _RA: Recorder[ItemA] = cast(RecorderA, None) _RB: Recorder[ItemB] = cast(RecorderB, None) An alternative pattern is to create an intermediary object that is specific to what is being operated on. This is a bit of a subtle distinction in this example, but it would look like this: .. code-block:: python import dataclasses from typing import TYPE_CHECKING, Protocol, cast class Recorder(Protocol): def record(self) -> None: ... @dataclasses.dataclass(frozen=True) class ItemA: a: int class ItemARecorder: item: ItemA def record(self) -> None: print(self.item.a) @dataclasses.dataclass(frozen=True) class ItemB(ItemA): b: int class ItemBRecorder: item: ItemB def record(self) -> None: print(self.item.a, self.item.b) def record_things(recorder: Recorder) -> None: recorder.record() record_things(ItemARecorder(item=ItemA(a=1))) record_things(ItemBRecorder(item=ItemB(a=1, b=5))) if TYPE_CHECKING: _RA: Recorder = cast(ItemARecorder, None) _RB: Recorder = cast(ItemBRecorder, None) If this is the extent of requirements then it's likely reasonable to only need to have a record method directly on the items themselves, but it's easy to imagine a scenario where there's a 1:n relationship between item and "recording" functionality and this pattern lets us separate the action of this "record" from what that actually means so that it's the caller that controls what that means rather than the orchestrator. .. note:: Note that in both these situations, we are able to represent the two sides of the design coin such that the implementation is generic and the usage is not .. code-block:: python import dataclasses from typing import TYPE_CHECKING, Protocol, TypeVar, cast T_CO_Item = TypeVar("T_CO_Item", covariant=True) class ForImplementation(Protocol[T_CO_Item]): @property def item(self) -> T_CO_Item: ... def do_something(self) -> None: ... class ForUse(Protocol): def do_something(self) -> None: ... @dataclasses.dataclass(frozen=True) class Implementation: item: MyItem def do_something(self) -> None: self.item.take_over_the_world() if TYPE_CHECKING: _FI: ForImplementation[MyItem] = cast(Implementation, None) _FU: ForUse = cast(Implementation, None) Sharing TypeVars ---------------- There are many cases where it's not necessary to share TypeVars, but there are two scenarios where it can be useful: - If the TypeVar is bound to a particular type or has a default - When the TypeVar is used to create a class that is intended to be subclass'd In both of these scenarios it can reduce problems around drift to be using a common definition of the TypeVar so that shared uses update when the shape of the TypeVar change.