Request Verification API - medullan/navocat-collector GitHub Wiki

Request Verification API (RVA)

Background

The Request Verification API (RVA) is a tool used to verify analytics hits sent to the Collector. Such hits are classified currently as IDENTIFY, PROFILE, PAGE, TRACK and ENDSESSION. RVA will generate a record/log for each hit and stores useful information about all interactions/events that occurred for this hit. These interactions/events mentioned are things such as sending information to external sources such as Google Analytics (GA), writing information to disk or even exceptions that occurred during the processing of each hit.

This became necessary due to the fact that developers needed an easier way to verify whether analytics hooks set up on websites for the Collector were working properly and also if the correct information was being sent.

RVA is made up of two key components; API and the Collector Verifier Tool.

API Documentaion

Configuration

To configure RVA, the feature needs to be toggled on in your meda.yml. This is done by setting verification_api to true as seen in the snippet below. This will enable all components of RVA

snippet of meda.yml

verification_api:
    private_keys:
      - 200ok
      - iamanotherkey
    collection_name: rva
    limit: 100
    id_prefix: rva-
    thread_id_key: meda_rva_id
features:
    verification_api: true

The following table will provide information on each property of the RVA configuration.

Configuration Description
private_keys
Type: String[ ]
Default: [ ]
This is a list of keys that is required to gain access to the API. There are no default keys. If not configured, you will be unable to authenticate any request made to RVA.
collection_name
Type: String
Default: rva
This is used to group RVA logs when being persisted to the log store (currently redis). This grouping enables the application to capture all RVA logs from the log store quickly. It is important to note that when the application is started, a suffix of the environment name is added to the collection
Example: in the development environment, the RVA log collection will be rva-development
limit
Type: Number
Default: 1000
This is a queue size for RVA logs. This determines how many RVA logs will be stored on the server at any given point in time.
id_prefix
Type: String
Default: rva-
This is a prefix added to the ID of the RVA log upon generation.
Example: rva-6f7445ef-8cf4-4ebf-8a9e-d68d2bb52fe4
thread_id_key
Type: String
Default: meda_rva_id
This key is used strictly to store the id of each log in a given analytics hit thread. All processes to collect information for RVA logs happen in various places. This becomes necessary to tie the data together.

Persistence Strategy

Currently, the main data store for RVA Logs is redis. The data store holds the main body of each RVA log. When the logs are requested, a further process may reach out to other sources to gather more information. One such example would be to get the related JSON data stored on disk.

Each log is stored with a custom storage key pattern. This storage key has sub keys with values that offer more information on each log. This allows the application to query redis for filtered or all RVA logs.

Pattern: [collection_name] | key(value) | key(value) |
Example: rva-development|pid(none)|cid(530bb57b-b3ee)|rid(rva-eabf899e)|sort(1)|

Key Description
pid This is the Profile ID generated by the Collector for the user.
cid This is the Client ID generated by the Collector for the user.
rid This is the RVA ID for the log. This value is unique to each RVA log.
sort This is a unique sequential number assigned to each RVA log in the order of creation.

When a sub key is added to the storage key but the value is not available then a default value of none is used as a placeholder. This helps when querying logs that do not have a specific field.

Endpoints

Purpose Getting or Deleting All Logs
Path /meda/verification/logs
Method GET, DELETE
Headers Authorization: [private_key]
Accept: application/json
Content-Type: application/json
Description This will get all logs with no filters applied. This will return a list of logs or an empty array if none were found
Response GET: Array of Objects
DELETE: {status: ''}

Purpose Getting or Deleting a subset of RVA Logs
Path /meda/verification/logs?filter_key=''&filter_value=''
Methods GET, DELETE
Headers Authorization: [private_key]
Accept: application/json
Content-Type: application/json
PARAM
filter_key
Types of keys to filter by:
mid, pid, rid or cid
Please Note: mid is a filter key for the Member ID.
PARAM
filter_value
value for the key you need to filter by
Description This will get or delete logs based on the specified filter
Example: /meda/verification/logs?filter_key='mid'&filter_value='500'
NB.
- If the filter key or value are empty then no logs will be returned
- The only predefined keyword as a value that can be used is none in filter_value. This will return all RVA logs that did not get stored with a value for the specified filter_key. Otherwise, enter the value that you wish to filter by.
Response GET: Array of Objects
DELETE: {status: ''}

Purpose Verifying a private key
Path /meda/verification/key
Method POST
Headers Accept: application/json
Content-Type: application/json
Description This will check that the key exists and returns an object with the said key OR will return an unauthorized response if not found
Request Body {key: ''}
Response {key: ''}

Collector Verifier Tool

About

The Collector Verifier Tool is the Graphical User Interface (GUI) for the Request Verification API (RVA). This tool helps the developer to focus on specific logs; these could be all logs for a Member ID.

The Collector Verifier is a Single Page Application (SPA) created with AngularJs using various plugins to support the functionalities provided. To see the full list of plugins, please view the index.html file

Backend Setup

Ruby Resources

In order to serve the Collector Verifier, two snippets of code were used to enable files to be served.

In app.rb, the index.html file/Home Screen was served with the following snippet:

get '/meda/verifier' do
    if Meda.features.is_enabled("verification_api", false)
      file = 'index.html'
      path = File.join('verifier', file)
      send_file(path)
    else
      status 404
    end
end

To serve other assets (such as css, javascript, etc.), the following code in config.ru was used:

map "/meda/verifier/assets" do
  if Meda.features.is_enabled("verification_api", false)
    run Rack::Directory.new("./verifier/assets")
  end
end

NB.

  • Notice that both snippets are wrapped in a toggle. If the RVA feature is not toggled on, then these files will not be accessible.
  • The backend setup is highly dependent on the folder structure. Please take note of what the code is referencing and the current folder structure when making any modifications.

Client-Side Resources

All resources (html, css, javascript, etc.) are located in the /verifier folder.

Folder Structure

verifier
--- index.html
+-- assets
|   +-- lib
|   +-- css
|   +-- js
|   +-- views

How to Use

Go to the following path in your browser to bring up the Home Screen of the Collector Verifier Tool:
[collector base address]/meda/verifier
Example: http://localhost:8000/meda/verifier

The Home Screen

Home Page

Once you arrive at the home screen, you will be prompted to enter your Private Key in order to authenticate. Entering a valid key will give you access to the application.

Log Screen

Once you are authenticated, the application will ask you to provide your main filter. Once a valid value is entered, then the logs will be populated.

NB. Currently, the only predefined keyword as a value that can be used is none as a main filter value. This will return all RVA logs that did not get stored with a value for the specified filter type. Otherwise, enter the value that you wish to filter by.

See screenshots below

Step 1

Home Page

Step 2

Home Page

Step 3

Home Page

Log Page Features

Reload Logs

The user can hit the reload logs link to load fresh logs from the server.

Expandable Logs

Each log can be expanded to see more information. This information is in the form of expandable JSON and styled information for the user to see.

Home Page

Search highlighting in JSON

Home Page

Import Logs

Logs can be imported from previous sessions. This is useful if comparisons are necessary. These logs will be archived.

Home Page

Archived Logs

All logs within the Collector Verifier are eligible to be archived at some points. The purpose of archived logs is to hold logs that are/were in the application but are not within the current main filter.
Logs are archived when:

  • The main filter is changed
  • Logs are imported
  • The user ends the session

To view archived logs, you can toggle them on/off but clicking the Show/Hide Archived Logs link. In the screenshot below, notice the archived logs are lighter in color than the active logs

Home Page

Ending The Session

You can end the session by clicking the End Session link in the main navigation bar. This will bring up a confirmation modal.
After ending the session you will be given the opportunity to save all the logs that were in the recently ended session by copying the JSON provided. An array of all logs from the server and all imported logs will be provided.

Step 1

Home Page

Step 2

Home Page

⚠️ **GitHub.com Fallback** ⚠️