Easy way to implement repository pagination (RavenDB + NestJS tutorial 7)

We are going to learn how to add support for pagination when retrieving a large list of documents with RavenDB and NestJS, the smart and industry standard way! At the end of this tutorial, we will have an endpoint for which we can specify the page number and the page size of the retrieved database list. Let’s get started!

RavenDB is a very fast, robust and scalable database system which can support any modern web application. Still, it doesn’t mean that app developers should neglect pagination as returning more and more documents from the database require more work on the database, app performance will be impacted and also transfer over the wire will be slower. Hence, pagination should be implemented as soon as possible. When it comes to RavenDB, I think although you call the method to retrieve all documents, in case they are too many, only the first 1000 are to be retrieved as far as I know so you will need pagination anyway.

You can find the source code for this tutorial in this Github repo.

Let’s get our hands dirty and see some code!

PaginationContext and PaginatedResults – classes to support pagination

We need two support classes to start with: PaginationContext and PaginatedResults. The first one will specify the page number and the page size, values which come from the client or switched to default in case they are empty. The PaginatedResults will encapsulate the actual results of the pagination query with info about the page, total number of pages or the total count of documents. These values are then sent to the client in order to support UI construction. Let’s look at the definition of these two classes:

export class PaginationContext {
  public page: number;
  public pageSize: number;
}

export class PaginatedResults<TPaginated extends BaseEntity> {
  public page: number;
  public pageSize: number;
  public pagesTotal: number;
  public docsTotal: number;
  public results: TPaginated[];
}

These classes are located in the base.repo.ts file, but can be moved around if necessary. I left them there to avoid unnecessary complexities on project structure. We also need to define some defaults inside the BaseRepo class definition:

protected get defaultPaginationContext(): PaginationContext {
  return {
    page: 1,
    pageSize: 50,
  };
}

protected get maxPageSize(): number {
  return 200;
}

The defaultPaginationContext property is going to assign default values for the paginated query in case the client sends an empty parameter and the maxPageSize is a hardcoded value used to restrict page size to a maximum level, hence gaining protection from unwanted input.

The paginated query method

Now the most important part is the paginated query which will be defined in the BaseRepo class. It will be similar to the retrieveDocuments but it will add pagination logic. Let’s have a look:

public async retrieveDocumentsPaginated(pagination: PaginationContext = null): Promise<PaginatedResults<TEntity>> {
  pagination = pagination || this.defaultPaginationContext;

  if (!pagination.page) {
    pagination.page = this.defaultPaginationContext.page;
  }

  if (!pagination.pageSize) {
    pagination.pageSize = this.defaultPaginationContext.pageSize;
  }

  const session = this.documentStore.openSession();
  const query = () =>
    session.query({
      collection: this.descriptor.collection,
      documentType: this.descriptor.class,
    }).noTracking()

  // We do not allow for large page sizes
  if (pagination.pageSize > this.maxPageSize) {
    pagination.pageSize = this.maxPageSize;
  }

  const docsTotal = await query().count();
  const roundPages = Math.round(docsTotal / pagination.pageSize);
  const pagesTotal = roundPages >= 1 ? roundPages : 1;

  // Do some validation checks
  if (pagination.page <= 0) {
    pagination.page = 1;
  }

  if (pagination.page > pagesTotal) {
    pagination.page = pagesTotal;
  }

  const results = await query()
      .skip((pagination.page - 1) * pagination.pageSize)
      .take(pagination.pageSize)
      .all();

  session.dispose();

  return {
    page: pagination.page,
    pageSize: pagination.pageSize,
    pagesTotal,
    docsTotal,
    results: results.map(this.metadataRemove),
  }
}

Hope it doesn’t look too intimidating 😄 The idea is that first we make sure we have the right values in the page and pageSize parameters, for instance for the pageSize to not exceed what we allow. Then we can observe the query as lambda expression which is used first to find out the total number of records then to actually perform the results retrieve with skip and take. You shouldn’t fret too much to understand the whole logic, trust me it works, I tested it like all of my source codes.

Amendment to the API method

Let’s test this long method by calling it from the API. We already have some API for movies, which was used in all my RavenNest tutorials. We will modify it to accept query parameters which will then be passed to the retrieveDocumentsPaginated method we created earlier. It uses specific NestJS pipe transforms to ensure it has a default value and that we convert page and pageSize to numbers since they are string by default. The new method looks like this:

@Get()
async getMovies(@Query('page', new DefaultValuePipe(1), ParseIntPipe) page: number, @Query('pageSize', new DefaultValuePipe(50), ParseIntPipe) pageSize: number) {
  return await this.movieRepo.retrieveDocumentsPaginated({
    page, pageSize,
  });
}

The last step is to use Postman to call this API and see the results:

🔝 Without supplying the parameters, page = 1 and pageSize = 50 is assumed by default.

🔝 If we request page = 2 and pageSize = 2, we get the correct results.

🔝 If we request page = 2 and pageSize = 500, we notice that the pageSize is brought back to 200 which is our maximum accepted value. Also, the page is reset since 200 accommodates all the records, hence we receive back page 1 instead of 2.

Conclusion

Although this is done specifically for RavenDB, you can use the concepts to implement your own pagination in whatever technology you are working with. It’s certainly a good addition and during my career so far I was surprised that many apps do not have such a feature at the backend level. Hope this article is useful for you and don’t forget to subscribe to the newsletter and contact me if you have any questions!

Thanks for reading, I hope you found this article useful and interesting. If you have any suggestions don’t hesitate to contact me. If you found my content useful please consider a small donation. Any support is greatly appreciated! Cheers  😉