Django QuerySets

This page is specifically talking about adding useful types for Django Querysets when using Mypy, django-stubs and extended-mypy-django-plugin

Basics

The Django ORM has a concept called the “Queryset” which is an object that can be used to represent the rows that would be fetched from the table given some filter. The Queryset provides a chain API where you can build up the filter a bit at the time and defer resolving that filter until the entire filter has been built up.

We create these Querysets via “object managers” on the model. Every model has a default manager which defaults to the attribute objects.

Type Annotations for Custom Querysets

Note

📕 Note that technically the Django ORM only guarantees _default_manager (the underscore prefix is for namespacing rather than saying it’s a private attribute) and it’s the Meta.default_manager_name option which defaults to objects that means every model by default has a .objects on it.

So given this model:

from django.db import models
from typing import final


@final
class MyModel(models.Model):
    happy = models.BooleanField()

We have access to a MyModel.objects which has the type models.Manager[MyModel] and has methods on it (like .all(), .filter(), etc) that return a Queryset with the type models.QuerySet[MyModel] .

It’s sometimes useful to have custom methods on our Queryset so that we can extract common filters. For example instead of MyModel.objects.filter(happy=True) we may want MyModel.objects.happy().

To do this we must create a custom manager that returns instances of our custom Queryset:

from django.db import models
from typing import Self, final


class MyModelQuerySet(models.QuerySet["MyModel"]):
    def happy(self) -> Self:
        return self.filter(happy=True)


@final
class MyModel(models.Model):
    happy = models.BooleanField()

    objects = MyModelQuerySet.as_manager()

There are two very important things to note here that will make Querysets much nicer to work with:

  • We use models.QuerySet["MyModel"] instead of models.QuerySet as the latter is an implicit models.QuerySet[Any] and so Mypy won’t recognise the type of the objects we get when we resolve the Queryset

  • We use typing.Self as the return type when we’re returning a copy of the Queryset. This is more relevant when we have a Queryset that has subclasses (see below about abstract models), but it’s a good habit to use this for all Queryset methods regardless.

Note

📕 Also, it’s a good idea to use the @typing.final decorator on models that don’t have subclasses.

See Using @final on Concrete models

Type Annotations for Custom Managers

Sometimes it’s useful to make a custom manager that returns a Queryset that already has a filter. In these case we need to do some special incantations to make Mypy recognise what we are doing:

from django.db import models
from typing import Self, final


class MyModelQuerySet(models.QuerySet["MyModel"]):
    def happy(self) -> Self:
        return self.filter(happy=True)


class _MyModelAlreadyHappyManager(models.Manager["MyModel"]):
    def get_queryset(self) -> MyModelQuerySet:
        return MyModelQuerySet(self.model, using=self._db).happy()


MyModelAlreadyHappyManager = _MyModelAlreadyHappyManager.from_queryset(
    MyModelQuerySet
)

@final
class MyModel(models.Model):
    happy = models.BooleanField()

    objects = MyModelQuerySet.as_manager()
    happy_ones = MyModelAlreadyHappyManager()

In this example we now have MyModel.happy_ones.all() doing the same thing as MyModel.objects.happy()

The trick here is the line

MyModelAlreadyHappyManager = _MyModelAlreadyHappyManager.from_queryset(
    MyModelQuerySet
)

Which means that when we do

happy_ones = MyModelAlreadyHappyManager()

Mypy is able to know that Queryset methods on this manager will return instances of MyModelQuerySet.

An explanation of what from_queryset does

The from_queryset method is a reasonable simple method, it dynamically creates a new manager class where the manager uses the provided Queryset and also has on it a copy of the custom Queryset methods on the manager itself.

The first part of that doesn’t make too much difference if we have created our own custom get_queryset method, but the latter does make it possible for us to access Queryset methods directly from the manager (though we encourage accessing such methods from the Queryset rather than the manager, by doing say a objects.all() first to get a Queryset).

From a typing perspective it means that Mypy is able to see that this manager does indeed operate in terms of our custom Queryset.

Note

We can’t instead do the following:

class MySpecialManager(models.Manager.from_queryset(MyCustomQuerySet)):
    ...

because that would be relying on runtime behaviour to create a class and so statically we can’t have guarantees over the types.

Prefer Queryset methods over manager methods

An “interesting” part of how the Django ORM works is that any methods on the manager are copied onto the Queryset.

So for example we could do

from django.db import models
from typing import Self, final


# This is an example of what NOT to do!!
class MyModelAlreadyHappyManager(models.Manager["MyModel"]):
    def happy(self) -> models.QuerySet["MyModel"]:
        return self.filter(happy=True)


@final
class MyModel(models.Model):
    happy = models.BooleanField()

    objects = MyModelAlreadyHappyManager.as_manager()

And then we have MyModel.objects.happy() or even MyModel.objects.all().happy() where the latter is calling that method from the Queryset rather than from the model.

This behaviour can confuse django-stubs and it’s much preferable to instead require MyModel.objects.all().happy() so that you are accessing methods on the Queryset than to make the methods available on the manager too.

Abstract models

The Django ORM uses inheritance to build up an idea of what fields are on a model and it gives us the ability to create Abstract models that represent a portion of the fields that appears on the final concrete models.

For example I may want to group a number of models so that I can refer to them in terms of the known parent, or I may want a group of models to share some specific fields without manually duplicating those fields on each model.

To define an abstract model we use the abstract = True option on the Meta for that model

from django.db import models
from typing import final


class MyAbstractModel(models.Model):
    one = models.CharField()

    class Meta:
        abstract = True


@final
class ConcreteOne(MyAbstractModel):
    two = models.CharField()

    def complete_message(self) -> str:
        return self.one + self.two


@final
class ConcreteTwo(MyAbstractModel):
    three = models.CharField()

    def complete_message(self) -> str:
        return self.one + self.three

In this hierarchy we have two “concrete” models (they result in a table being created in the database) where “ConcreteOne” has the two fields “one” and “two”. And “ConcreteTwo” has the two fields “one” and “three”. And both models have a separate implementation of a “complete_message” function

Note

❗ Note that we can’t use the @abc.abstractmethod decorator on django models because the abc.ABCMeta metaclass that makes that decorator actually do anything conflicts with the metaclass that the Django ORM uses. It’s encouraged to document when a method is expected on models but only implement it on the models themselves and rely on having the extended Mypy django plugin to give us types that exclude the abstract models when accessing these common methods.

There are some things to be aware of with abstract models

Shared methods

It’s common to want to create methods on the abstract model that are used by the concrete models.

In these cases where the extended mypy django plugin is installed we want to use Concrete.cast_as_concrete to get a type that represents all of the concrete models

from extended_mypy_django_plugin import Concrete
from django.db import models


class MyAbstractModel(models.Model):
    one = models.CharField()

    class Meta:
        abstract = True

    def __repr__(self) -> str:
        concrete = Concrete.cast_as_concrete(self)
        return "{concrete.__class__.__name__} - {concrete.complete_message()}"

The Concrete.cast_as_concrete(self) in this case is equivalent to the type we get from Concrete[MyAbstractModel] which at Mypy time is turned into ConcreteOne | ConcreteTwo because that is the complete set of descendants to MyAbstractModel that doesn’t have Meta.abstract = True. Mypy will understand the return type of concrete.concrete_message() because both items in that union have implemented that method.

See below for information about shared .new() methods.

Foreign keys

Adding a foreign key to an abstract only makes sense if the related_name on the foreign key is either + which means no reverse relationship is registered on the foreign model, or if the related name is formatted in terms of the names of the subclasses so that the reverse relationship has a unique name. When a reverse relationship is necessary it is often better to instead declare the foreign key explicitly on each concrete model.

Shared Querysets regarding Abstract models

It can be desirable for the Queryset on each concrete model to share some Queryset methods. To do this we do the following

from django.db import models
from typing import Self


class MyAbstractQuerySet[T_Model: Concrete[MyAbstractModel]](models.QuerySet[T_Model]):
    def some_filter(self) -> Self:
        return self.filter(one="stuff")


class MyAbstractModel(models.Model):
    one = models.CharField()

    class Meta:
        abstract = True


class ConcreteOneQuerySet(MyAbstractQuerySet["ConcreteOne"]):
    pass


class ConcreteOne(MyAbstractModel):
    objects = ConcreteOneQuerySet.as_manager()

The two important things here are:

  • We don’t add a custom .objects to the abstract model

  • We define the base Queryset with a TypeVar so that the final concrete model given to models.QuerySet[] is defined by each subclass of the QuerySet.

Shared .new methods

It can b e useful to define a MyModel.new() method for use outside of the model instead of directly accessing MyModel.objects.create(). This is a little less straightforward on an abstract model. For this we are able to use the Concrete.cast_as_concrete to make this work:

from extended_mypy_django_plugin import Concrete
from django.db import models
from typing import Self


class MyAbstractModel(models.Model):
    one = models.CharField()

    class Meta:
        abstract = True

    @classmethod
    def new(cls, *, one: str) -> Self:
        made = Concrete.cast_as_concrete(cls).objects.create(one=one)
        # Convince mypy that we are returning an instance of cls
        # This is needed because cls.objects.create correctly doesn't statically
        # guarantee that it's an instance of cls that is returned
        assert isinstance(made, cls)
        return made

Note that when the concrete models have different foreign keys on them it can be useful to instead do something like

from extended_mypy_django_plugin import Concrete
from django.db import models
from typing import Self, TypedDict, NotRequired, final


class NewArgs(TypedDict):
    one: NotRequired[str]
    two: str


class PartialNew[T_Ret](Protocol):
    def __call__(self, **kwargs: Unpack[NewArgs]) -> T_Ret: ...


class MyAbstractModel(models.Model):
    one = models.CharField()
    two = models.CharField()

    class Meta:
        abstract = True

    @classmethod
    def _new[T_Ret: Concrete[MyAbstractModel](
        cls, partial_new: PartialNew[T_Ret], /, *, one: str = "", two: str
    ) -> T_Ret:
        made = partial_new(one=one, two=two)

        # This is especially useful if some common action needs to happen
        # to the instance that was just made
        made.some_post_creation_action()

        return made

@final
class ConcreteOne(MyAbstractModel):
    three = models.ForeignKey(SomeOtherModel, related_name="concretes_one")

    @classmethod
    def new(self, *, three: SomeOtherModel, **kwargs: Unpack[NewArgs]) -> Self:
        def _partial_new(**kwargs: Unpack[NewArgs]) -> Self:
            return self.objects.create(three=three, **kwargs)
        return self._new(_partial_new, **kwargs)

This way we are able to keep the details of the specialised field on each concrete model in that model itself and the abstract model does not need to concern itself with these details.

Unique Querysets

We need to have unique Queryset names otherwise django-stubs gets confused and it’s preferable to not confuse django-stubs (it can be a delicate flower!).

There a few ways we end up with non unique Querysets:

  • Models with the same name in different parts of the codebase that don’t define a custom Queryset with a unique name

  • Abstract models having its own concrete Queryset (see section above about abstract models)

  • Concrete models using the same custom Queryset

  • Multiple abstract models with the same name having a reverse relationship to the same model

Unique names

When you have multiple models with the same name, it becomes necessary to define their custom querysets with unique names so that django-stubs doesn’t create confusing errors where it’s substituted in the wrong custom queryset:

class _OnPremiseDispenserQuerySet(models.QuerySet["Dispenser"]):
    pass


class Dispenser(models.Model):
    objects = _OnPremiseDispenserQuerySet.as_manager()

Note that instead of referring to the Queryset by name we can use:

from extended_mypy_django_plugin import DefaultQuerySet

DefaultQuerySet[Dispenser]

This annotation will resolve to either models.QuerySet[Dispenser] if the model has no custom Queryset. However, since we have defined a custom Queryset, in this case it will resolve too _OnPremiseDispenserQuerySet.

You can also do this to have an explicit reference to the Queryset

class _OnPremiseDispenserQuerySet(models.QuerySet["Dispenser"]):
    pass


DispenserQuerySet = _OnPremiseDispenserQuerySet


class Dispenser(models.Model):
    objects = _OnPremiseDispenserQuerySet.as_manager()

Abstract models should not have their own custom managers

To ensure custom Querysets do not have conflicting names, it’s best to not have shared Querysets via inheritance.

Shared Querysets

Just like with the abstract models above, when sharing a Queryset, use a TypeVar so that the concrete model that the Queryset ends up with is determined by the final Queryset

class ActivePeriodQuerySet[T_Model: models.Model](models.QuerySet[T_Model]):
    ...


class MyModel1QuerySet(ActivePeriodQuerySet["MyModel1"]):
    pass


class MyModel1(models.Model):
    objects = MyQuery1Set()


class MyModel2QuerySet(ActivePeriodQuerySet["MyModel2"]):
    pass


class MyModel2(models.Model):
    objects = MyQuery2Set()

Using @final on Concrete models

It can be useful to decorate models that aren’t subclassed with the final decorator to convince Mypy that we never have subclasses of these models.

This is nice for example when we are working with Model.objects.create() . For example if we had a model like this

from django.db import models
from typing import Self


class MyModel(models.Model):
    @classmethod
    def new(cls) -> Self:
        return cls.objects.create() # mypy error!

In this scenario the return type of cls.objects.create() is not necessarily the same class if we can have subclasses.

Take for example the following code:

class ConcreteParentQuerySet(models.QuerySet["ConcreteParent"]):
    pass


class ConcreteParent(models.Model):
    objects = ParentQuerySet.as_manager()

    @classmethod
    def new(cls) -> Self:
        return cls.objects.create()


class ConcreteChild(ConcreteParent):
    pass

In this example, technically, and Mypy is correct for saying this, the return type of ConcreteChild.objects.create() is a ConcreteParent rather than a ConcreteChild and so cls.objects.create() in ConcreteParent cannot be guaranteed to be a typing.Self

By adding a @final to ConcreteParent we remove the possibility of subclasses that don’t redefine objects and so Mypy believes us when we say cls.objects.create() returns the same type as the class that .new is being called on.