Request Verification API - medullan/navocat-collector GitHub Wiki
The Request Verification API (RVA) is a tool used to verify analytics hits sent to the Collector. Such hits are classified currently as IDENTIFY, PROFILE, PAGE, TRACK and ENDSESSION. RVA will generate a record/log for each hit and stores useful information about all interactions/events that occurred for this hit. These interactions/events mentioned are things such as sending information to external sources such as Google Analytics (GA), writing information to disk or even exceptions that occurred during the processing of each hit.
This became necessary due to the fact that developers needed an easier way to verify whether analytics hooks set up on websites for the Collector were working properly and also if the correct information was being sent.
RVA is made up of two key components; API and the Collector Verifier Tool.
To configure RVA, the feature needs to be toggled on in your meda.yml. This is done by setting verification_api to true as seen in the snippet below. This will enable all components of RVA
snippet of meda.yml
verification_api:
private_keys:
- 200ok
- iamanotherkey
collection_name: rva
limit: 100
id_prefix: rva-
thread_id_key: meda_rva_id
features:
verification_api: true
The following table will provide information on each property of the RVA configuration.
Configuration | Description |
---|---|
private_keys Type: String[ ] Default: [ ] |
This is a list of keys that is required to gain access to the API. There are no default keys. If not configured, you will be unable to authenticate any request made to RVA. |
collection_name Type: String Default: rva |
This is used to group RVA logs when being persisted to the log store (currently redis). This grouping enables the application to capture all RVA logs from the log store quickly. It is important to note that when the application is started, a suffix of the environment name is added to the collection Example: in the development environment, the RVA log collection will be rva-development |
limit Type: Number Default: 1000 |
This is a queue size for RVA logs. This determines how many RVA logs will be stored on the server at any given point in time. |
id_prefix Type: String Default: rva- |
This is a prefix added to the ID of the RVA log upon generation. Example: rva-6f7445ef-8cf4-4ebf-8a9e-d68d2bb52fe4 |
thread_id_key Type: String Default: meda_rva_id |
This key is used strictly to store the id of each log in a given analytics hit thread. All processes to collect information for RVA logs happen in various places. This becomes necessary to tie the data together. |
Currently, the main data store for RVA Logs is redis. The data store holds the main body of each RVA log. When the logs are requested, a further process may reach out to other sources to gather more information. One such example would be to get the related JSON data stored on disk.
Each log is stored with a custom storage key pattern. This storage key has sub keys with values that offer more information on each log. This allows the application to query redis for filtered or all RVA logs.
Pattern: [collection_name] | key(value) | key(value) |
Example: rva-development|pid(none)|cid(530bb57b-b3ee)|rid(rva-eabf899e)|sort(1)|
Key | Description |
---|---|
pid | This is the Profile ID generated by the Collector for the user. |
cid | This is the Client ID generated by the Collector for the user. |
rid | This is the RVA ID for the log. This value is unique to each RVA log. |
sort | This is a unique sequential number assigned to each RVA log in the order of creation. |
When a sub key is added to the storage key but the value is not available then a default value of none is used as a placeholder. This helps when querying logs that do not have a specific field.
Purpose | Getting or Deleting All Logs |
---|---|
Path | /meda/verification/logs |
Method | GET, DELETE |
Headers |
Authorization: [private_key] Accept: application/json Content-Type: application/json |
Description | This will get all logs with no filters applied. This will return a list of logs or an empty array if none were found |
Response | GET: Array of Objects DELETE: {status: ''}
|
Purpose | Getting or Deleting a subset of RVA Logs |
---|---|
Path | /meda/verification/logs?filter_key=''&filter_value='' |
Methods | GET, DELETE |
Headers |
Authorization: [private_key] Accept: application/json Content-Type: application/json |
PARAM filter_key |
Types of keys to filter by: mid, pid, rid or cid Please Note: mid is a filter key for the Member ID. |
PARAM filter_value |
value for the key you need to filter by |
Description | This will get or delete logs based on the specified filter Example: /meda/verification/logs?filter_key='mid'&filter_value='500' NB. - If the filter key or value are empty then no logs will be returned - The only predefined keyword as a value that can be used is none in filter_value. This will return all RVA logs that did not get stored with a value for the specified filter_key. Otherwise, enter the value that you wish to filter by. |
Response | GET: Array of Objects DELETE: {status: ''}
|
Purpose | Verifying a private key |
---|---|
Path | /meda/verification/key |
Method | POST |
Headers |
Accept: application/json Content-Type: application/json |
Description | This will check that the key exists and returns an object with the said key OR will return an unauthorized response if not found |
Request Body | {key: ''} |
Response | {key: ''} |
The Collector Verifier Tool is the Graphical User Interface (GUI) for the Request Verification API (RVA). This tool helps the developer to focus on specific logs; these could be all logs for a Member ID.
The Collector Verifier is a Single Page Application (SPA) created with AngularJs using various plugins to support the functionalities provided. To see the full list of plugins, please view the index.html file
In order to serve the Collector Verifier, two snippets of code were used to enable files to be served.
In app.rb, the index.html file/Home Screen was served with the following snippet:
get '/meda/verifier' do
if Meda.features.is_enabled("verification_api", false)
file = 'index.html'
path = File.join('verifier', file)
send_file(path)
else
status 404
end
end
To serve other assets (such as css, javascript, etc.), the following code in config.ru was used:
map "/meda/verifier/assets" do
if Meda.features.is_enabled("verification_api", false)
run Rack::Directory.new("./verifier/assets")
end
end
NB.
- Notice that both snippets are wrapped in a toggle. If the RVA feature is not toggled on, then these files will not be accessible.
- The backend setup is highly dependent on the folder structure. Please take note of what the code is referencing and the current folder structure when making any modifications.
All resources (html, css, javascript, etc.) are located in the /verifier folder.
verifier --- index.html +-- assets | +-- lib | +-- css | +-- js | +-- views
Go to the following path in your browser to bring up the Home Screen of the Collector Verifier Tool:
[collector base address]/meda/verifier
Example: http://localhost:8000/meda/verifier
Once you arrive at the home screen, you will be prompted to enter your Private Key in order to authenticate. Entering a valid key will give you access to the application.
Once you are authenticated, the application will ask you to provide your main filter. Once a valid value is entered, then the logs will be populated.
NB. Currently, the only predefined keyword as a value that can be used is none as a main filter value. This will return all RVA logs that did not get stored with a value for the specified filter type. Otherwise, enter the value that you wish to filter by.
See screenshots below
Step 1
Step 2
Step 3
Log Page Features
Reload Logs
The user can hit the reload logs link to load fresh logs from the server.
Expandable Logs
Each log can be expanded to see more information. This information is in the form of expandable JSON and styled information for the user to see.
Search highlighting in JSON
Import Logs
Logs can be imported from previous sessions. This is useful if comparisons are necessary. These logs will be archived.
Archived Logs
All logs within the Collector Verifier are eligible to be archived at some points. The purpose of archived logs is to hold logs that are/were in the application but are not within the current main filter.
Logs are archived when:
- The main filter is changed
- Logs are imported
- The user ends the session
To view archived logs, you can toggle them on/off but clicking the Show/Hide Archived Logs link. In the screenshot below, notice the archived logs are lighter in color than the active logs
You can end the session by clicking the End Session link in the main navigation bar. This will bring up a confirmation modal.
After ending the session you will be given the opportunity to save all the logs that were in the recently ended session by copying the JSON provided. An array of all logs from the server and all imported logs will be provided.
Step 1
Step 2