resolutionflow/docs/connectwise/best-practices/PSA-Pagination.md

1. Last updated

   Mar 31, 2023

2. Save as PDF

Overview

When working with ConnectWise PSA instances, you will run into a significant amount of data.  This guide should help you to work with and manipulate that data for your integration.  

Pagination Defined

The concept of pagination is similar to that of pages within a book or when you are navigating a website or forum and there is a button to go to the next page.  Simply put, when you have a large set of data, pagination breaks it up into different pages for consumption.  The below is a perfect example of the pagination concept that you have probably seen before.

Pagination with APIs works a little bit differently but has the same underlying concept. When you try to retrieve a list of objects that is too large, either for the computational costs associated with consumption or any of the plethora of networking concerns (waiting for a few gigs of data from an API call can have unexpected effects on your application), the API will respond with information about how to access the next page of results.  Instead of flipping through pages either physically or with a navigation tree, the API has a series of Link headers that allow you to do it programmatically without human interaction.

Business Case

ConnectWise products store a significant amount of data and often times developers may not understand the impact of their integration on a given environment.  We have set a hard limit of 1,000 records for the protection of the Partner environments.  Many integrations do not need data past this point and if we didn't have a set limit, it allow for the API to potentially lock records or cause overall performance problems.

Implementation

There are two different methods to Pagination, each has it's own benefit or drawback depending on the type of integration you are creating.

Type	Pros	Cons	Usage
Navigable	Allows you to create a UI that allows for human interaction in going back and forth between records.	Uses a large number of resources and results are returned slower as you get into the higher page sizes.	This example returns the first 100 results.
service/tickets?pagesize=100&page=1
Forward Only	Each call will return results at the same speed as the last.  Data is returned from an indexed location.	There is no way to page backward.  This means you can't design a navigable UI around it.	This example returns the first 100 results after Ticket#40941
service/tickets?pagesize=100&conditions=id>1231

Each version of paging will return a series of Link headers to help with navigation.  Let's dive into each Pagination Method and how to utilize them.

Navigable Pagination

The ConnectWise PSA API provides a vast wealth of information for developers to consume.  Most of the time, you will find that there is far more information than is needed for integration.  In order to ensure server availability, the API automatically sets the page size to 25 results.  The maximum page size for any request can be up to 1,000 records if properly specified in the request URL.

pagesize	The number of results returned by each call.  Defaulted to 25 and has a maximum of 1,000.  These values cannot be changed.
page	Starting with page 1, is the number of pages available based on the current pagesize.

When using paging, the response will include a Link header that looks like this:

1	/v``4``_``6``_release/apis/``3.0``/company/companies?pagesize=``50``&page=``1

If you notice, the header contains two separate URLs in this instance.  The first one is "next" and the second one is "last".  These refer to the next set of results and the final set of results.  If we navigate to the next page, we will get some additional URLs in the response Link header.

1	/v``4``_``6``_release/apis/``3.0``/company/companies?pagesize=``50``&page=``2

Now that we have navigated to the second page, we will find two additional URLs.  These are "prev" and "first" and in total we now have four including the original "next" and "last".

First	This link will display the first page available based on the current page size
Prev*	This link will display the previous page based on the current page position and page size
Next*	This link will display the next available page based on the current page position and page size
Last	This link will always display the final page available based on the current page size

*We will not return the next or prev links if there isn't a next or prev respectively.

In the above examples, we have four pages of results and depending on the page size, that number can change.  For instance, let's say we switch to the default page size of 25.  Instead of returning four pages, we actually get back 7 pages as there are only 160 results.

The goal of pagination is not to pull every possible result, every single time.  Instead, it is designed to be in conjunction with a UI component to create a set of navigation links.

Navigable Pagination closely follows RFC 5988.

Forward-Only Pagination

Released in 2018.5

Unlike Navigable Pagination, forward-only requires that you pass in a header to identify the type of pagination you would like to use.  For instance, we now accept a header called pagination-type, and when you set that to forward-only you will get to utilize the new features.  In addition to the new header, there is a new query parameter called pageId.  The pageId is the record in which you would like to begin with for paging.

• If you do not include the new header in your request, it will use the default paging method of navigable. 
• The Page query parameter will be ignored with forward-only paging as all results are technically page 1.
• You cannot use an Order By query parameter with forward-only as it must be ordered by the ID.
• There will always be a link header in the response for the next pageId.
• The pageId query parameter is treated like an additional condition of Id > pageId.
• The following pageId in the header will be the last Id you got in the request.

Forward-Only pagination does not work with the Audit Trail endpoints at this time.

How to Get All Records?

In most cases, when you experience pagination, you likely want all of the items in the list and aren’t very happy about the extra work required to page through the results. Here are some best practices to make it a little less painful to get all of your results:

Avoid it

Although this doesn't sound ideal, conceptually this is how you will be able to get the most performance.  Many APIs allow you to query for items based on specific criteria or only return certain subsets of the data.  The PSA API is no different and for example, allows you to query based on any number of fields including time ranges.  If you know anything about the data you are looking for, you can narrow down the result set from the beginning, in many cases eliminating pagination and giving you a performance boost in comparison to over-fetching and then filtering in your application.

Pay Attention to the Headers

We only return the paging headers if there is something to the page.  If you don't see the next page, don't try to query for the next page number.  Use the paging headers programmatically to achieve your results and avoid trying to make them on your own.

While looping

Depending on your programming language of choice, pagination can be a wonderful use case for While.  The basic workflow is that while you are getting a pagination token in your response, keep making subsequent requests.  In pseudo-code that might look like this:

Page = GetPageOfItems();
//process the data from the page, or add it to a larger array, etc.

while( Page->cursor )
    Page  = GetPageOfItems(Page->cursor);
    //process the data again
end

As soon as your API stops responding with a pagination cursor, then you can stop looping and continue execution with the complete set of data.  There are a couple of important points to remember with an approach like this:

• Any errors from the API that don’t return a cursor might prematurely stop your execution, so always check that you get a good response from the API.
• You probably want to include some checks to make sure that you stop bringing in more information at some upper limit, just in case the API responds with much more information than you expect and keeps paging, and paging, and paging…