How to Fetch Paginated Data in API Engine

Overview

Some external APIs return data in paginated responses, meaning that only a subset of records is included in each response, with additional data available via a "next link" provided by the API. The SuiteEngine API Engine has built-in support for handling pagination, allowing API requests to automatically retrieve subsequent pages of data.

The API Engine will automatically execute the Next Function Code (as defined in the API Function record) when the Next Link field on the API Message is populated. The value stored in this field will be transferred to the Link field of the next API Message, which can then be used to construct the next request. This data can either:

• Overwrite the endpoint URL entirely
• Be used as a parameter in the next API request

There are two approaches to fetching paginated data:

1. Using Mappings
2. Implementing the "Get Next Link" Interface

Option 1: Using Mappings

If the next link data is included in the response body, it can be extracted and mapped directly to the Next Link field in the API Message. This is the simplest approach, as it does not require any additional development effort.

Steps:

1. Open the API Set record.
2. Navigate to the API Mapping for the relevant API Call.
3. Locate the field in the response body that contains the pagination link.
4. Map this field to the Next Link field on the API Message. Note: This option may not always work, as some APIs structure pagination differently. If mapping alone does not correctly populate the Next Link field, use Option 2. Example

Option 2: Implementing the "Get Next Link" Interface

If direct mapping is not sufficient, you can implement the GetNextToken interface to dynamically extract the next link from the API response. This requires extending the "SENAPIRespNxtTokenProcessing" enumeration and providing custom logic.

Steps:

1. Extend the "SENAPIRespNxtTokenProcessing" enumeration.
2. Implement the GetNextToken procedure.
   • This procedure receives the API Message and the API Response Message as parameters.
   • It should extract the next link from the response and return it as a string.
3. Assign your new enumeration value to the Next Link Processing Method in the API Function record.

---

Option 1 Example (Get Orders)

In this example, the API response includes a next field containing the URL that should be used for the subsequent request:

{
  "count": 5,
  "next": "https://sample.com/orders/ca374345-1279-40d0-8a2b-eab6b826be0e/?limit=1&offset=1",
  "previous": null,
  "orders": [
    {
      "id": 57886556,
      "name": "John"
    }
  ]
}

Mapping Configuration

To correctly capture and use the next link, follow these steps:

1. Set Up the Root Node Mapping

   • Set the Source Type on the root node (RESPONSEOBJECT) as a Table Node.
   • Assign the Table Number to 70338454 (API Message).

2. Map the "next" Field

   • Locate the "next" element in the response.
   • Assign the Table Number to 70338454 (API Message).
   • Assign the Field Number to 210 (Next Token Link).

   This mapping ensures that the next URL is stored in the Next Token Link field, which will be used to execute the next API call.

3. Map the Orders Data

   • The orders array should be mapped as a Record Node.
   • Follow the standard mapping process outlined in How to Create API Mappings.

Defining the Next Function

Now, we need to define an API function to handle the next API call.

1. Create a New API Function

   • Open the API Functions list and create a new function called GET_ORDERS_NEXT.

2. Set the Next Function Code

   • Open the original API function (GET_ORDERS).
   • Set the Next Function Code to GET_ORDERS_NEXT.
   • Also, set the Next Function Code for GET_ORDERS_NEXT to itself. This ensures that if the API returns another next link, it continues fetching additional pages.

3. Reusing the Mapping

   • If the response schema for paginated calls is identical to the original request, you can reuse the mappings.
   • In GET_ORDERS_NEXT, set Use Mappings from Function to GET_ORDERS.
   • This eliminates the need to create a separate mapping for the paginated request.

Overriding the Endpoint URL

Since paginated requests may require modifying the request URL dynamically, follow these steps:

1. Select the GET_ORDERS_NEXT function.
2. From the action bar, go to Request > URL Overrides.
3. Configure the URL Override:
   • Name: Url
   • Value Processing Type: Parameter Record
   • Value Table No: 70338454 (API Message)
   • Value Field No: 220 (Token Link)

Testing the Pagination

1. Manually execute the GET_ORDERS function.
2. When prompted with a pop-up asking "Do you want to execute the next API call?", select Yes.
3. The system will process both API calls (GET_ORDERS and GET_ORDERS_NEXT), retrieving all paginated data.