True cursor pagination for Django

[mjtamlyn]

Problem

The default pagination approach used for Django querysets executes a .count() on the query. For tables with hundreds of thousands or millions of rows, this is very slow on Postgres. It also assumes that the set is stable, which somewhat defeats the point of using a cursor based pagination system in the first place.

Possible solution

I released today django-cursor-pagination which provides a paginator class which fulfils the relay cursor connection spec. It is available on PyPI.

Below is a pretty simple approach to using this in Graphene, with inspiration taken from graphene-gae - thanks to @ekampf. It would want some checks around full_args, and possibly a means to explicitly set the ordering. It would also need testing and documenting.

Should we use this in graphene? There are some caveats of using this approach. Most importantly, the ordering fields in my current implementation must uniquely determine the items in the queryset. (Discussion of this restriction). Perhaps we should provide both options?

import graphene
from cursor_pagination import CursorPaginator

def connection_from_cursor_paginated(queryset, args, connection_type, edge_type, pageinfo_type, **kwargs):
    paginator = CursorPaginator(queryset, queryset.query.order_by)
    full_args = dict(args, **kwargs)
    page = paginator.page(**full_args)

    edges = []
    for item in page:
        edge = edge_type(node=item, cursor=paginator.cursor(item))
        edges.append(edge)

    return connection_type(
        edges=edges,
        page_info=pageinfo_type(
            start_cursor=paginator.cursor(page[0]),
            end_cursor=paginator.cursor(page[-1]),
            has_previous_page=page.has_previous,
            has_next_page=page.has_next,
        )
    )


class CursorPaginatedConnection(graphene.relay.types.Connection):
    @classmethod
    def from_list(cls, queryset, args, context, info):
        connection = connection_from_cursor_paginated(queryset, args, connection_type=cls, edge_type=cls.edge_type, pageinfo_type=graphene.relay.PageInfo)
        connection.set_connection_data(queryset)
        return connection


class CursorPaginatedConnectionField(graphene.relay.ConnectionField):
    def __init__(self, *args, **kwargs):
        kwargs['connection_type'] = kwargs.pop('connection_type', CursorPaginatedConnection)
        super(CursorPaginatedConnectionField, self).__init__(*args, **kwargs)

[syrusakbary]

I think is worth to take a closer look into this approach.

Also, having an order that strictly determine all the items in the queryset is somewhat essential for pagination.

My ideas here are:

• Use CursorPaginatedConnection when possible (if a query order its defined).
• Use the current behavior otherwise.

Let's play with it in a branch that could let us see the edge cases

[mjtamlyn]

Looking at this a bit more in the context of #62 I wonder whether we can reduce the plumbing duplication needed for various different data backends. The core logical structure is the same for my Django implementation, for the GAE one, and for the rethinkdb cursor mentioned in #62.

Ideally what we want is an easy way to add a Cursor subclass or similar which has some defined API, possibly similar to what I defined in django-cursor-pagination. It needs to be able to:

• Be initialised with some data set
• Have some method like page(first, last, after, before), which returns an object with resultant set of items, and the data needed for PageInfo.
• Have a means of getting the cursor for each item

Ideally you could then specify the cursor as a kwarg and/or attribute of graphene.relay.ConnectionField and everything else would follow. A built in DjangoCursor could swap between using CursorPaginator or the current count/slice approach as needed.

[defrex]

I would love this.

I'll underscore one point. Right now you can't lookup a node's cursor without the full context of the connection. This means returning an edge from a mutation requires a ton of overhead and possibly knowledge you don't have. For that reason the ability to look up a cursor for a given node without any additional information should be a goal. This issue has been discussed a lot in #59.

Also, to solve the issue of a missing order value, you could order by primary key by default.

[crucialfelix]

The example at the top doesn't work now because you cannot make an abstract subclass of graphene.relay.Connection.

Just by importing this (not using it, just defining the class):

class CursorPaginatedConnection(graphene.relay.Connection):
    @classmethod
    def from_list(cls, queryset, args, context, info):
        connection = connection_from_cursor_paginated(queryset, args, connection_type=cls, edge_type=cls.edge_type, pageinfo_type=graphene.relay.PageInfo)
        connection.set_connection_data(queryset)
        return connection

raises:

AssertionError: You have to provide a node in ConnectionMeta.Meta

ConnectionMeta does some error checking to ensure that it's subclasses are usable:

https://github.com/graphql-python/graphene/blob/v1.2.0/graphene/relay/connection.py#L61

I've been running into issues with these meta classes quite a bit when working with graphene. You cannot write an abstract subclass of DjangoObjectType for the same reason — the error checking scolds you.

I guess these reusable things should be written as plain classes and then apply them as mixins. But then to actually use them we need to write factory methods to assemble a one off class and use it immediately.

# cannot do this
clients = connection.CursorPaginatedConnectionField(ClientNode)
# I will have to write a factory
clients = make_paginated_connection(ClientNode)

[mjtamlyn]

The example at the top worked on Graphene 0.X, you're right it doesn't work with 1.X.

I redefined Connection more completely on graphene 1.X to make it work:

class CursorPaginatedConnection(six.with_metaclass(graphene.relay.connection.ConnectionMeta, graphene.ObjectType)):
    ...

I also had to add a different default resolver to the field.

[necnec]

@mjtamlyn you said you also added a resolver?
Could you provide any working example with your custom paginator?

[ramusus]

@mjtamlyn could you please update your approach for latest

graphene==2.0.1
graphene-django==2.0.0

there is no such class as graphene.relay.connection.ConnectionMeta

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.