.. _django_querysets:

Django QuerySets
================

This page is specifically talking about adding useful types for Django Querysets
when using `Mypy`_, `django-stubs`_ and `extended-mypy-django-plugin`_

.. _mypy: https://www.mypy-lang.org

.. _django-stubs: https://github.com/typeddjango/django-stubs

.. _extended-mypy-django-plugin: https://extended-mypy-django-plugin.readthedocs.io/en/latest/

Basics
------

The Django ORM has a concept called the "Queryset" which is an object that can
be used to represent the rows that would be fetched from the table given some
filter. The Queryset provides a chain API where you can build up the filter a
bit at the time and defer resolving that filter until the entire filter has been
built up.

We create these Querysets via "object managers" on the model. Every model has a
default manager which defaults to the attribute ``objects``.

Type Annotations for Custom Querysets
-------------------------------------

.. note::
    📕 Note that technically the Django ORM only guarantees ``_default_manager`` 
    (the underscore prefix is for namespacing rather than saying it's a private
    attribute) and it's the `Meta.default_manager_name`_ option which defaults
    to ``objects`` that means every model by default has a ``.objects`` on it.

.. _Meta.default_manager_name : https://docs.djangoproject.com/en/5.2/ref/models/options/#django.db.models.Options.default_manager_name

So given this model:

.. code-block:: python

    from django.db import models
    from typing import final


    @final
    class MyModel(models.Model):
        happy = models.BooleanField()

We have access to a ``MyModel.objects`` which has the type ``models.Manager[MyModel]``
and has methods on it (like ``.all()``, ``.filter()``, etc) that return a Queryset
with the type ``models.QuerySet[MyModel]`` .

It's sometimes useful to have custom methods on our Queryset so that we can
extract common filters. For example instead of ``MyModel.objects.filter(happy=True)``
we may want ``MyModel.objects.happy()``.

To do this we must create a custom manager that returns instances of our custom
Queryset:

.. code-block:: python

    from django.db import models
    from typing import Self, final


    class MyModelQuerySet(models.QuerySet["MyModel"]):
        def happy(self) -> Self:
            return self.filter(happy=True)


    @final
    class MyModel(models.Model):
        happy = models.BooleanField()

        objects = MyModelQuerySet.as_manager()

There are two very important things to note here that will make Querysets much
nicer to work with:

- We use ``models.QuerySet["MyModel"]`` instead of ``models.QuerySet`` as the latter
  is an implicit ``models.QuerySet[Any]`` and so Mypy won't recognise the type of
  the objects we get when we resolve the Queryset
- We use ``typing.Self`` as the return type when we're returning a copy of the
  Queryset. This is more relevant when we have a Queryset that has subclasses
  (see below about abstract models), but it's a good habit to use this for all
  Queryset methods regardless.

.. note::
   📕 Also, it's a good idea to use the ``@typing.final`` decorator on models
   that don't have subclasses.

   See :ref:`Using @final on Concrete models <final_on_concrete_models>`

Type Annotations for Custom Managers
------------------------------------

Sometimes it's useful to make a custom manager that returns a Queryset that
already has a filter. In these case we need to do some special incantations to
make Mypy recognise what we are doing:

.. code-block:: python

    from django.db import models
    from typing import Self, final


    class MyModelQuerySet(models.QuerySet["MyModel"]):
        def happy(self) -> Self:
            return self.filter(happy=True)


    class _MyModelAlreadyHappyManager(models.Manager["MyModel"]):
        def get_queryset(self) -> MyModelQuerySet:
            return MyModelQuerySet(self.model, using=self._db).happy()


    MyModelAlreadyHappyManager = _MyModelAlreadyHappyManager.from_queryset(
        MyModelQuerySet
    )

    @final
    class MyModel(models.Model):
        happy = models.BooleanField()

        objects = MyModelQuerySet.as_manager()
        happy_ones = MyModelAlreadyHappyManager()

In this example we now have ``MyModel.happy_ones.all()`` doing the same thing as
``MyModel.objects.happy()`` 

The trick here is the line

.. code-block:: python

    MyModelAlreadyHappyManager = _MyModelAlreadyHappyManager.from_queryset(
        MyModelQuerySet
    )

Which means that when we do

.. code-block:: python

    happy_ones = MyModelAlreadyHappyManager()

Mypy is able to know that Queryset methods on this manager will return instances
of ``MyModelQuerySet``.

An explanation of what from_queryset does
+++++++++++++++++++++++++++++++++++++++++

The ``from_queryset`` method is a reasonable simple method, it
`dynamically creates a new manager class`_ where the manager uses the provided
Queryset and also has on it a copy of the custom Queryset methods on the manager
itself.

.. _dynamically creates a new manager class: https://github.com/django/django/blob/a7b0e50eadba8f0420013605c70eb790280b0fd2/django/db/models/manager.py#L108

The first part of that doesn't make too much difference if we have created our
own custom ``get_queryset`` method, but the latter does make it possible for us to
access Queryset methods directly from the manager (though we encourage
accessing such methods from the Queryset rather than the manager, by doing say a
``objects.all()`` first to get a Queryset).

From a typing perspective it means that Mypy is able to see that this manager
does indeed operate in terms of our custom Queryset.

.. note::

   We can't instead do the following:

   .. code-block:: python

       class MySpecialManager(models.Manager.from_queryset(MyCustomQuerySet)):
           ...

   because that would be relying on runtime behaviour to create a class and so
   statically we can't have guarantees over the types.

Prefer Queryset methods over manager methods
--------------------------------------------

An "interesting" part of how the Django ORM works is that any methods on the
manager are copied onto the Queryset.

So for example we could do

.. code-block:: python

    from django.db import models
    from typing import Self, final


    # This is an example of what NOT to do!!        
    class MyModelAlreadyHappyManager(models.Manager["MyModel"]):
        def happy(self) -> models.QuerySet["MyModel"]:
            return self.filter(happy=True)


    @final
    class MyModel(models.Model):
        happy = models.BooleanField()

        objects = MyModelAlreadyHappyManager.as_manager()

And then we have ``MyModel.objects.happy()`` or even ``MyModel.objects.all().happy()``
where the latter is calling that method from the Queryset rather than from the
model.

This behaviour can confuse ``django-stubs`` and it's much preferable to instead
require ``MyModel.objects.all().happy()`` so that you are accessing methods on the
Queryset than to make the methods available on the manager too.

Abstract models
---------------

The Django ORM uses inheritance to build up an idea of what fields are on a
model and it gives us the ability to create Abstract models that represent a
portion of the fields that appears on the final concrete models.

For example I may want to group a number of models so that I can refer to them
in terms of the known parent, or I may want a group of models to share some
specific fields without manually duplicating those fields on each model.

To define an abstract model we use the ``abstract = True`` option on the ``Meta``
for that model

.. code-block:: python

    from django.db import models
    from typing import final


    class MyAbstractModel(models.Model):
        one = models.CharField()

        class Meta:
            abstract = True


    @final
    class ConcreteOne(MyAbstractModel):
        two = models.CharField()

        def complete_message(self) -> str:
            return self.one + self.two


    @final
    class ConcreteTwo(MyAbstractModel):
        three = models.CharField()

        def complete_message(self) -> str:
            return self.one + self.three

In this hierarchy we have two "concrete" models (they result in a table being
created in the database) where "ConcreteOne" has the two fields "one" and "two".
And "ConcreteTwo" has the two fields "one" and "three". And both models have a
separate implementation of a "complete_message" function

.. note::
    ❗ Note that we can't use the ``@abc.abstractmethod`` decorator on django models
    because the ``abc.ABCMeta`` metaclass that makes that decorator actually do
    anything conflicts with the metaclass that the Django ORM uses. It's encouraged
    to document when a method is expected on models but only implement it on the
    models themselves and rely on having the extended Mypy django plugin to give us
    types that exclude the abstract models when accessing these common methods.

There are some things to be aware of with abstract models

Shared methods
++++++++++++++

It's common to want to create methods on the abstract model that are used by the
concrete models.

In these cases where the `extended mypy django plugin`_ is installed we want to
use ``Concrete.cast_as_concrete`` to get a type that represents all of the concrete
models

.. _extended mypy django plugin: https://extended-mypy-django-plugin.readthedocs.io/en/latest/

.. code-block:: python

    from extended_mypy_django_plugin import Concrete
    from django.db import models


    class MyAbstractModel(models.Model):
        one = models.CharField()

        class Meta:
            abstract = True

        def __repr__(self) -> str:
            concrete = Concrete.cast_as_concrete(self)
            return "{concrete.__class__.__name__} - {concrete.complete_message()}"

The ``Concrete.cast_as_concrete(self)`` in this case is equivalent to the type we
get from ``Concrete[MyAbstractModel]`` which at Mypy time is turned into
``ConcreteOne | ConcreteTwo`` because that is the complete set of descendants to
``MyAbstractModel`` that doesn't have ``Meta.abstract = True``. Mypy will understand
the return type of ``concrete.concrete_message()`` because both items in that
union have implemented that method.

See below for information about shared ``.new()`` methods.

Foreign keys
++++++++++++

Adding a foreign key to an abstract only makes sense if the related_name on the
foreign key is either ``+`` which means no reverse relationship is registered on
the foreign model, or if the related name is formatted in terms of the names of
the subclasses so that the reverse relationship has a `unique name`_. When a
reverse relationship is necessary it is often better to instead declare the
foreign key explicitly on each concrete model.

.. _unique name: https://docs.djangoproject.com/en/5.1/topics/db/models/#abstract-related-name

Shared Querysets regarding Abstract models
++++++++++++++++++++++++++++++++++++++++++

It can be desirable for the Queryset on each concrete model to share some
Queryset methods. To do this we do the following

.. code-block:: python

    from django.db import models
    from typing import Self


    class MyAbstractQuerySet[T_Model: Concrete[MyAbstractModel]](models.QuerySet[T_Model]):
        def some_filter(self) -> Self:
            return self.filter(one="stuff")


    class MyAbstractModel(models.Model):
        one = models.CharField()

        class Meta:
            abstract = True


    class ConcreteOneQuerySet(MyAbstractQuerySet["ConcreteOne"]):
        pass


    class ConcreteOne(MyAbstractModel):
        objects = ConcreteOneQuerySet.as_manager()

The two important things here are:

- We don't add a custom ``.objects`` to the abstract model
- We define the base Queryset with a TypeVar so that the final concrete model
  given to ``models.QuerySet[]`` is defined by each subclass of the QuerySet.

Shared .new methods
+++++++++++++++++++

It can b e useful to define a ``MyModel.new()`` method for use outside of the model
instead of directly accessing ``MyModel.objects.create()``. This is a little less
straightforward on an abstract model. For this we are able to use the
``Concrete.cast_as_concrete`` to make this work:

.. code-block:: python

    from extended_mypy_django_plugin import Concrete
    from django.db import models
    from typing import Self


    class MyAbstractModel(models.Model):
        one = models.CharField()

        class Meta:
            abstract = True

        @classmethod
        def new(cls, *, one: str) -> Self:
            made = Concrete.cast_as_concrete(cls).objects.create(one=one)
            # Convince mypy that we are returning an instance of cls
            # This is needed because cls.objects.create correctly doesn't statically
            # guarantee that it's an instance of cls that is returned
            assert isinstance(made, cls)
            return made

Note that when the concrete models have different foreign keys on them it can
be useful to instead do something like

.. code-block:: python

    from extended_mypy_django_plugin import Concrete
    from django.db import models
    from typing import Self, TypedDict, NotRequired, final


    class NewArgs(TypedDict):
        one: NotRequired[str]
        two: str


    class PartialNew[T_Ret](Protocol):
        def __call__(self, **kwargs: Unpack[NewArgs]) -> T_Ret: ...


    class MyAbstractModel(models.Model):
        one = models.CharField()
        two = models.CharField()

        class Meta:
            abstract = True

        @classmethod
        def _new[T_Ret: Concrete[MyAbstractModel](
            cls, partial_new: PartialNew[T_Ret], /, *, one: str = "", two: str
        ) -> T_Ret:
            made = partial_new(one=one, two=two)

            # This is especially useful if some common action needs to happen
            # to the instance that was just made
            made.some_post_creation_action()

            return made

    @final
    class ConcreteOne(MyAbstractModel):
        three = models.ForeignKey(SomeOtherModel, related_name="concretes_one")

        @classmethod
        def new(self, *, three: SomeOtherModel, **kwargs: Unpack[NewArgs]) -> Self:
            def _partial_new(**kwargs: Unpack[NewArgs]) -> Self:
                return self.objects.create(three=three, **kwargs)
            return self._new(_partial_new, **kwargs)

This way we are able to keep the details of the specialised field on each
concrete model in that model itself and the abstract model does not need to
concern itself with these details.

Unique Querysets
----------------

We need to have unique Queryset names otherwise ``django-stubs`` gets confused and
it's preferable to not confuse ``django-stubs`` (it can be a delicate flower!).

There a few ways we end up with non unique Querysets:

- Models with the same name in different parts of the codebase that don't define
  a custom Queryset with a unique name
- Abstract models having its own concrete Queryset (see section above about
  abstract models)
- Concrete models using the same custom Queryset
- Multiple abstract models with the same name having a reverse relationship to
  the same model

Unique names
++++++++++++

When you have multiple models with the same name, it becomes necessary to define
their custom querysets with unique names so that ``django-stubs`` doesn't create
confusing errors where it's substituted in the wrong custom queryset:

.. code-block:: python

    class _OnPremiseDispenserQuerySet(models.QuerySet["Dispenser"]):
        pass


    class Dispenser(models.Model):
        objects = _OnPremiseDispenserQuerySet.as_manager()

Note that instead of referring to the Queryset by name we can use:

.. code-block:: python

    from extended_mypy_django_plugin import DefaultQuerySet

    DefaultQuerySet[Dispenser]

This annotation will resolve to either ``models.QuerySet[Dispenser]`` if
the model has no custom Queryset. However, since we have defined a custom Queryset,
in this case it will resolve too ``_OnPremiseDispenserQuerySet``.

You can also do this to have an explicit reference to the Queryset

.. code-block:: python

    class _OnPremiseDispenserQuerySet(models.QuerySet["Dispenser"]):
        pass


    DispenserQuerySet = _OnPremiseDispenserQuerySet


    class Dispenser(models.Model):
        objects = _OnPremiseDispenserQuerySet.as_manager()


Abstract models should not have their own custom managers
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++

To ensure custom Querysets do not have conflicting names, it's best to not
have shared Querysets via inheritance.

Shared Querysets
++++++++++++++++

Just like with the abstract models above, when sharing a Queryset, use a TypeVar
so that the concrete model that the Queryset ends up with is determined by the
final Queryset

.. code-block:: python

    class ActivePeriodQuerySet[T_Model: models.Model](models.QuerySet[T_Model]):
        ...


    class MyModel1QuerySet(ActivePeriodQuerySet["MyModel1"]):
        pass


    class MyModel1(models.Model):
        objects = MyQuery1Set()


    class MyModel2QuerySet(ActivePeriodQuerySet["MyModel2"]):
        pass


    class MyModel2(models.Model):
        objects = MyQuery2Set()

Set related_name to + on foreign keys on abstract models
++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Let's say for example we have multiple abstract models called ``AuditEntry`` that
all have on them ``support_user = models.ForeignKey("support.SupportUser")`` this
implicitly creates a reverse relationship on support_user such that
``support_user.audit_entry`` returns a reverse relationship manager that returns
those AuditEntry objects. This already is undefined behaviour and we should
ensure reverse relationships have `unique names`_ but from a static typing
perspective, if we follow the rule that abstract models don't have their own
custom Querysets this means the type of that reverse relationship is
``models.QuerySet[AuditEntry] | models.QuerySet[AuditEntry] | models.QuerySet[AuditEntry]``
as opposed to something like
``_AusAuditEntryQS | _UkAuditEntryQS | _EspAuditEntryQS`` and django-stubs will
get confused.

.. _unique names: https://docs.djangoproject.com/en/5.1/topics/db/models/#abstract-related-name

It's best to use ``related_name='+'`` when a reverse relationship isn't needed to
avoid this problem.

.. _final_on_concrete_models:

Using ``@final`` on Concrete models
-----------------------------------

It can be useful to decorate models that aren't subclassed with the
`final decorator`_ to convince Mypy that we never have subclasses of these
models.

.. _final decorator: https://docs.python.org/3/library/typing.html#typing.final

This is nice for example when we are working with ``Model.objects.create()`` .
For example if we had a model like this

.. code-block:: python

    from django.db import models
    from typing import Self


    class MyModel(models.Model):
        @classmethod
        def new(cls) -> Self:
            return cls.objects.create() # mypy error!

In this scenario the return type of ``cls.objects.create()`` is not necessarily
the same class if we can have subclasses.

Take for example the following code:

.. code-block:: python

    class ConcreteParentQuerySet(models.QuerySet["ConcreteParent"]):
        pass


    class ConcreteParent(models.Model):
        objects = ParentQuerySet.as_manager()

        @classmethod
        def new(cls) -> Self:
            return cls.objects.create()


    class ConcreteChild(ConcreteParent):
        pass

In this example, technically, and Mypy is correct for saying this, the return
type of ``ConcreteChild.objects.create()`` is a ``ConcreteParent`` rather than a
``ConcreteChild`` and so ``cls.objects.create()`` in ``ConcreteParent`` cannot be
guaranteed to be a ``typing.Self``

By adding a ``@final`` to ``ConcreteParent`` we remove the possibility of subclasses
that don't redefine ``objects`` and so Mypy believes us when we say
``cls.objects.create()`` returns the same type as the class that ``.new`` is being
called on.