Performance Testing Protocol for CKAN - ckan/ckan GitHub Wiki

Performance testing protocol for CKAN

The following document outlines a protocol for testing and reporting on the performance and load characteristics of a CKAN installation. The output of this protocol is a set of statistics describing:

  • the response times of a number of key views/pages
  • the behaviour of these response times to increased/parallel loading

Basic overview

The tests are performed against an installation of CKAN which has had a standard set of example data loaded. The example data can be found at

https://github.com/ckan/ckan-demo-data

A typical public CKAN instance will receive many more "read" requests than "write" requests, and the primary purpose of the testing is to evaluate the "read" performance of the application.

The testing occurs against a number of key views (pages) within the application. These include:

  • homepage
  • datasets overview
  • datasets overview (paginated)
  • datasets overview (filtered by tag)
  • several specific dataset overview pages
  • several specific dataset activity stream pages
  • several specific resource pages
  • resource download
  • groups overview
  • several specific group overview pages
  • several specific group activity stream pages
  • API package_list view
  • API tag_list view
  • API datastore_search view

The testing takes two parts: 1) performance testing and 2) load testing.

The aim of performance testing is to establish what the best-case response times are for each of these pages.

The aim of load testing is to establish the statistics of response times under realistic (and extreme) loading conditions.

Performance testing

For each potential target URL, the following tests are run:

  • a 1 minute "cold cache" single-threaded performance test
  • a 1 minute "warm cache" single-threaded performance test

In each test, as many requests as possible are made within the 30s test duration by a single process.

The output of each test will be:

  • 50/90/99th percentile response times
  • the number of unsuccessful (timeout/error) responses

Load testing

A load testing tool will be run that generates a predefined load against the CKAN instance by picking target URLs at random and requesting them in parallel. All tests last 5 minutes, and the following levels of load are tested:

  • 1 req/s
  • 10 req/s
  • 30 req/s
  • 50 req/s
  • 100 req/s

The output of each test will be:

  • overall 50/95/99th percentile response times
  • the number of unsuccessful (timeout/error) responses

See also

https://github.com/nickstenning/ckan-perf-testing