Waffle Server API - Gapminder/waffle-server GitHub Wiki

Waffle Server API

Note: all notations that look like ${SOME_TEXT} - mean that this is a placeholder and its real value should be either set by the user or WS itself.

Table of Contents

DDF dataset asset

GET /api/ddf/assets/${PATH_TO_DDF_DATASET_ASSET}

This route can be used for querying files from an assets directory under a particular dataset. So imagine we have a dataset hosted on the GitHub under the open-numbers account - its name is ddf--gapminder--systema_globalis. This dataset has a directory under the root called assets with has just a single file in it - world-50m.json. You want to fetch this files from dataset's develop branch.

Next, let's assume that this dataset is imported into the Waffle Server and you want to request this asset from it. To do this you need to construct the following query:

https://import-waffle-server.gapminder.org/api/ddf/assets/open-numbers/ddf--gapminder--systema_globalis/develop/assets/world-50m.json

If you want to fetch the same asset from a default dataset on the Waffle Server, you need to change this query to:

https://import-waffle-server.gapminder.org/api/ddf/assets/default/assets/world-50m.json

In this query we've replaced open-numbers/ddf--gapminder--systema_globalis/develop with a default keyword.

Important details: When using a default dataset you cannot specify a branch but when using any dataset other than default you must specify the branch under which the asset gets requested.

If you are requesting assets from private dataset - you must specify a dataset_access_token as a query parameter:

https://import-waffle-server.gapminder.org/api/ddf/assets/open-numbers/ddf--gapminder--systema_globalis/develop/assets/world-50m.json?dataset_assess_token=YOUR_ACCESS_TOKEN

Also, you should know that assets get served only from Waffle Server Threshing machine rather than whole Waffle Server cluster.

DDFQL

GET /api/ddf/ql?${DDFQL_QUERY_IN_URLON}

Inside of DDFQL_QUERY_IN_URLON, you can specify the format in which you want to get a response. By default wsJson is used.

Available formats: 1. wsJson 2. csv

Response in wsJson format:


{
  "headers": ["header1", "header2", "header3", "..."],
  "rows": [
    ["hader1Value1", "hader2Value1", "hader3Value1", "..."],
    ["hader1Value2", "hader2Value2", "hader3Value2", "..."],
    ["hader1Value3", "hader2Value3", "hader3Value3", "..."],
    "..."
  ]
}

DDFQL query is extendable with following properties: 1. dataset - takes a value of dataset name that should be queried. Example: open-numbers/ddf--gapminder--systema_globalis#stage. Where: - open-numbers - GitHub account name; - ddf--gapminder--systema_globalis - repo name; - #stage is a branch name (master branch considered as default). 2. version - takes a value of dataset repository commit. This commit can be chosen only if the dataset was imported at this commit or updated to this commit. 3. format - takes values described above (wsJson, csv)

Dataset and version choosing algorithm:

  • If no dataset nor version were given - then default dataset with according version will be used for data serving;
  • If only dataset were given - then latest successfully applied version of this dataset will be used;
  • If only version were given - then dataset will be inferred by this version and used for querying;
  • If dataset and version were given - then they will be used for query execution;
  • If dataset or version sent by the user were not found then error is generated with message explaining it;

Only successfully applied versions can be queried.

Not every successful request stores for future WS warmup, apart from that it:

  • should not have dataset property in DDFQL query;
  • should not have version property in DDFQL query;
  • should not have format property in DDFQL query;

GET /api/ddf/ql?$query={DDFQL_QUERY_IN_ENCODED_JSON}

Response is the same as in /api/ddf/ql?${DDFQL_QUERY_IN_URLON}

POST /api/ddf/ql

As a body for this post request - DDFQL should be sent with Content-Type set as application/json

Response is the same as in /api/ddf/ql?${DDFQL_QUERY_IN_URLON}

Waffle Server Import CLI

POST /api/ddf/cli/authenticate

This route is used for user authentication in WS and generates expirable authentication token (token's time to live is 1 hour).

Following parameters should be present in the request body:

  • email
  • password

Following response will be sent in case of a successful request:

{
  "success": true,
  "data": {
    "token": "${TOKEN_VALUE}"
  }
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/prestored-queries?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token acquired from /api/ddf/cli/authenticate

This route is used for getting all available datasets for the current user with their versions.

Following response will be sent in case of a successful request:

{
  "success": true,
  "data": [
    {
      "createdAt": "${TIME_OF_DATASET_CREATION}",
      "datasetName": "${NAME_OF_THE_DATASET}",
      "githubUrl": "${URL_TO_DATASET_REPO_ON_GITHUB_IN_SSH_FORMAT}",
      "version": "${COMMIT_WHICH_CAN_BE_QUERIED_ON_WS_FOR_THIS_DATASET}",
      "isDefault": "${IS_THIS_DATASET_DEFAULT}"
    },
    "..."
  ]
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/git-commits-list?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Following parameters should be present in the request query:

  • github - GitHub URL in ssh format

This route gets available commits for repo URL (repo URL should be in the ssh format).

Following response will be sent in case of a successful request:

{
  "success": true,
  "data": {
    "commits": [ "${commits available for given github url on WS}" ]
  }
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/commit-of-latest-dataset-version?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that acquired from /api/ddf/cli/authenticate

Serves metadata regarding latest dataset version

Following parameters should be provided within the query

  • github - GitHub URL in ssh format;

Following response will be sent in case of a successful request:

{
  "success": true,
  "data": {
    "github": "${DATASET_REPO_GITHUB_URL_IN_SSH_FORMAT}",
    "dataset": "${NAME_OF_THE_DATASET}",
    "commit": "${COMMIT_HASH}"
  }
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/transactions/latest/status?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Provides information about latest transaction performed for the given dataset.

Following parameters should be provided within the query:

  • datasetName - name of the dataset like open-numbers/ddf--gapminder--systema_globalis

Following response will be sent in case of the successful request:

{
  "success": true,
  "data": {
      "datasetName": "${NAME_OF_THE_DATASET}",
      "transaction": {
        "lastError": "${LATEST_OCCURRED_ERROR_IN_SCOPE_OF_TRANSACTION}",
        "commit": "${COMMIT_THAT_WAS_APPLIED}",
        "status": "${STATUS_OF_TRANSACTION}",
        "createdAt": "${TIME_OF_TRANSACTION_CREATION}"
      },
      "modifiedObjects": {
          "concepts": "${AMOUNT_OF_TOUCHED_IN_TRANSACTION_CONCEPTS}",
          "entities": "${AMOUNT_OF_TOUCHED_IN_TRANSACTION_ENTITIES}",
          "datapoints": "${AMOUNT_OF_TOUCHED_IN_TRANSACTION_DATAPOINTS}"
      }
  }
}

${STATUS_OF_TRANSACTION} might have either Completed or In progress value

Following response will be sent in case of the unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/datasets/private?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Fetches all available private datasets (only successfully imported)

Following response will be sent in case of the successful request:

{
  "success": true,
  "data": [
    {
      "name": "${NAME_OF_THE_PRIVATE_DATASET}",
      "githubUrl": "${URL_TO_DATASET_GITHUB_REPO}"
    },
    "..."
  ]
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/datasets/removable?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Fetches all datasets available for removal (only successfully imported and not default)

Following response will be sent in case of the successful request:

{
  "success": true,
  "data": [
    {
      "name": "${NAME_OF_THE_PRIVATE_DATASET}",
      "githubUrl": "${URL_TO_DATASET_GITHUB_REPO}"
    },
    "..."
  ]
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/datasets/inProgress?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Fetches all datasets that are currently being imported, updated or removed.

Following response will be sent in case of the successful request:

{
  "success": true,
  "data": [
    {
      "name": "${NAME_OF_THE_PRIVATE_DATASET}",
      "githubUrl": "${URL_TO_DATASET_GITHUB_REPO}"
    },
    "..."
  ]
}

Following response will be sent in case of the successful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

GET /api/ddf/cli/datasets/removalStatus?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Returns a status object that contains information on how many concepts, entities, and datapoints were already removed from a requested dataset that is being removed.

Following parameters should be provided in the query

  • datasetName - name of the dataset like open-numbers/ddf--gapminder--systema_globalis

Following response will be sent in case of the successful request:

{
  "success": true,
  "data": {
      "concepts": "${AMOUNT_OF_CONCEPTS_REMOVED_SO_FAR}",
      "entities": "${AMOUNT_OF_ENTITIES_REMOVED_SO_FAR}",
      "datapoints": "${AMOUNT_OF_DATAPOINTS_REMOVED_SO_FAR}",
    }
}

Following response will be sent in case of the successful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

POST /api/ddf/cli/update-incremental?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Starts dataset incremental update operation.

Following parameters should be provided in the request body:

  • hashFrom - from which hash commit update should be started;
  • hashTo - to which hash commit dataset should be updated;
  • github - URL to dataset GitHub repo in the ssh format;

Only 'Status Code 200' will be sent in case of the successful request - as this operation is non-blocking and happens in the background

Following response will be sent in case of the unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

POST /api/ddf/cli/import-dataset?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Starts dataset importing operation.

Following parameters should be provided in the request body:

  • commit - from which hash commit importing should be started
  • github - URL to dataset GitHub repo in the ssh format
  • repoType (optional, "public" by default) - URL to dataset GitHub repo in the ssh format

Only 'Status Code 200' will be sent in case of the successful request - as this operation is non-blocking and happens in the background

Following response will be sent in case of the unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

POST /api/ddf/cli/remove-dataset?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Starts dataset removing operation.

Following parameters should be provided in the request body:

  • datasetName - name of the imported dataset in a format like open-numbers/ddf--gapminder--systema_globalis

Only 'Status Code 200' will be sent in case of the successful request - as this operation is non-blocking and happens in the background

Following response will be sent in case of the unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

POST /api/ddf/cli/transactions/latest/rollback?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Starts dataset rollback operation, which removes all the changes that were done by the latest failed transaction.

Following parameters should be provided in the request body:

  • datasetName - name of the imported dataset in a format like open-numbers/ddf--gapminder--systema_globalis

Following response will be sent in case of the successful request:

{
  "success": true,
  "message": "${INFORMATION_MESSAGE_ABOUT_OPERATION_STATE}"
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

POST /api/ddf/cli/datasets/default?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Sets the default dataset, which will be used by default to serve data in response to DDFQL queries. If no default dataset is set and there is no dataset in DDFQL query - then an error is generated. Every change of the default dataset causes cache invalidation on WS. There should be only one default dataset.

Following parameters should be provided in the request body:

  • datasetName - name of the imported dataset in the following format open-numbers/ddf--gapminder--systema_globalis;
  • commit - version of dataset that should be considered default;

Following response will be sent in case of the successful request:

{
  "success": true,
  "data": {
    "name": "${DEFAULT_DATASET_NAME}",
    "commit": "${DEFAULT_DATASET_VERSION}",
    "createdAt": "${VERSION_APPLYING_TIME}"
  }
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

POST /api/ddf/cli/datasets/accessToken?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Generates access token for a private dataset. In case, if there are a couple of subsequent generations on the same dataset, the previous token will be overwritten. Only with this token user will be able to access private datasets.

Following parameters should be provided in the request body:

  • datasetName - name of the imported dataset in a format like open-numbers/ddf--gapminder--systema_globalis

Following response will be sent in case of the successful request::

{
  "success": true,
  "data": {
    "accessToken": "${DATASET_ACCESS_TOKEN}"
  }
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

POST /api/ddf/cli/cache/clean?waffle-server-token=${AUTH_TOKEN}

${AUTH_TOKEN} is a token that is acquired from /api/ddf/cli/authenticate

Cleans cache on WS. The cache will be cleaned globally (for all datasets available).

Following response will be sent in case of the successful request:

{
  "success": true,
  "message": "${INFORMATION_MESSAGE_ABOUT_OPERATION_STATE}"
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}

Gapminder Vizabi

GET /api/vizabi/mc_precomputed_shapes.json

This route is used on Vizabi Tools page only in mountain chart.

GET /api/vizabi/world-50m.json

This route is used on Vizabi Tools page only in map chart.

Development Waffle Server API

POST /api/development/populate-documents

Following parameters should be provided in the request body:

  • datasetName - Name from which data should be fetched
  • commit - version of given dataset
  • collection - to which collection query should be sent
  • query - query from which response should be populated. By 'populated' I mean substitute links to other collection with actual objects from them. Only ONE level deep.

Following response will be sent in case of the successful request:

{
  "success": true,
  "data": [ "Array of populated raw mongo documents" ]
}

Following response will be sent in case of an unsuccessful request:

{
  "success": false,
  "error": "${ERROR_MESSAGE}"
}