Testing the registration service - FarmRadioHangar/uliza-core-apis GitHub Wiki

This document describes, in depth, the steps required to set up and manage a Docker-based environment for testing the Uliza registration service middleware, based on two different test strategies:

• Automated: Integration tests using the Mocha test framework (Node.js), and
• Interactive: These are "live" tests involving actual calls and user participation.

These instructions assume a GNU/Linux or Unix-like environment, and have been successfully completed on Ubuntu 16.04. Results on other systems may vary.

We'll need Docker, Node, npm, and (at least) curl, git and the mysqladmin command line tool. A full MySQL server installation is not required. A reliable and reasonably fast Internet connection is strongly recommended for building the Docker images.

On Ubuntu and other Debian-based systems:

sudo apt-get install curl git mysql-client

Install Docker. Instructions for Ubuntu 16.04 can be found here. Verify that the docker daemon is running. E.g.,

sudo systemctl status docker

⚠️ Note: As described in the Digital Ocean article, make sure to add your username to the docker group, allowing for the command to be invoked without having to use sudo. This is assumed in the instructions given below.

sudo usermod -aG docker ${USER}
su - ${USER}

Install Node.js and the npm tool. Instructions for Ubuntu 16.04 can be found here. Run the command node --version && npm --version to verify the installation.

git clone https://github.com/FarmRadioHangar/uliza-core-apis

Make a note of the location of the cloned repo. In the remainder of this page, we will refer to this as the project base directory. E.g., if you ran git clone in /home/laserpants/, the base directory is going to be /home/laserpants/uliza-core-apis.

Note: A great deal of network traffic is involved in this step.

In addition to the MySQL database, which is readily available from Docker Hub, three custom images are required:

Replace {PROJECT_BASE_DIRECTORY} with the location to which you cloned the repository.

To build these, move to the project base directory, i.e., cd {PROJECT_BASE_DIRECTORY}/ and then run the following commands:

docker build services/registration-middleware -t farmradio/registration_service
docker build services/tests/voto-mock-api -t farmradio/voto_mock_api
docker build django-api -t farmradio/uliza_api

Building the images will take some time. ☕ Please be patient.

Now that we have the necessary images, make sure you are in services/tests/registration-middleware/, relative to the project base directory.

Replace {PROJECT_BASE_DIRECTORY} with the location to which you cloned the repository.

cd {PROJECT_BASE_DIRECTORY}/services/tests/registration-middleware/

Then, to create the Docker containers, verify the script's contents, and run the below command:

source <(curl -s https://raw.githubusercontent.com/FarmRadioHangar/uliza-core-apis/master/services/tests/registration-middleware/utils/containers.sh)

If you are having trouble running this script, or want to change its behavior, have a look at Preparing the test environment.

When prompted, confirm overwriting the .env file with the new port assignments. Inspect the contents of this file using the command cat .env. 🐈

Example output:

REG_SERVICE_URL=http://0.0.0.0:32779
DB_HOST=0.0.0.0
DB_PORT=32776
ULIZA_API_URL=http://0.0.0.0:32778/api/v1
VOTO_API_URL=http://0.0.0.0:32777/api/v1
VOTO_API_KEY=XXX_TEST_KEY_XXX

Next, review the output of this command:

docker ps \
  --format="table {{.Image}}\t{{.Names}}\t{{.Ports}}" \
  --filter="name=ulizatests*"

It should be a list of four running containers.

Example output:

IMAGE                            NAMES                   PORTS
farmradio/registration_service   ulizatestsmiddleware   0.0.0.0:32771->3034/tcp
farmradio/uliza_api              ulizatestsapi          0.0.0.0:32770->8000/tcp
farmradio/voto_mock_api          ulizatestsvoto         0.0.0.0:32769->8089/tcp
mysql:5.7                        ulizatestsdb           0.0.0.0:32768->3306/tcp

The following command will generate an URL pointing to the API endpoint for the Uliza Participants resource.

echo -e "http://$(docker ps \
  --filter "name=ulizatestsapi" \
  --format "{{.Ports}}" | sed -n 's/\([0-9.:]*\).*/\1/p')/api/v1/participants/"

It should look something like the following (where xxxxx is the port assigned by the host system to the Uliza API server).

http://0.0.0.0:xxxxx/api/v1/participants/

Open this URL in a web browser. This should give you the browsable API user interface:

From within the same directory (i.e., {PROJECT_BASE_DIRECTORY}/services/tests/registration-middleware/), run

npm install

to install the npm package dependencies listed in package.json. Then, to run the tests, enter

npm test

No actual calls are scheduled during these tests. Once verified that the automated tests pass without errors, we can proceed to test the system using real calls by pointing the registration service to VOTO's API.

• Use the issue tracker to report any errors.
• Make sure all errors are resolved before proceeding to the interactive tests.

For the live tests, we will run the containers.sh script again, this time passing a live parameter, which tells the script to point the registration middleware to VOTO's API instead of the mock API server (used by the Mocha tests). Again, check the script's contents and run the following command:

read -p "Enter your VOTO API key: " VOTO_KEY && \
  source <(curl -s https://raw.githubusercontent.com/FarmRadioHangar/uliza-core-apis/master/services/tests/registration-middleware/utils/containers.sh) \
  live \
  "$VOTO_KEY"

Running,

docker ps \
  --format="table {{.Image}}\t{{.Names}}\t{{.Ports}}" \
  --filter="name=ulizatests*"

we should now have three running containers:

IMAGE                            NAMES                  PORTS
farmradio/registration_service   ulizatestsmiddleware   0.0.0.0:32785->3034/tcp
farmradio/uliza_api              ulizatestsapi          0.0.0.0:32784->8000/tcp
mysql:5.7                        ulizatestsdb           0.0.0.0:32783->3306/tcp

From a new terminal window, run

docker exec -it ulizatestsmiddleware ./ngrok http 3034

This will open up a tunnel to port 3034 on localhost, inside the registration service container. This is to enable the server to receive webhook notifications from within the dockerized environment. Make sure you leave the ngrok session running throughout the tests.

Next we need to create a test survey in VOTO, and associate this survey with a registration tree.

When creating the survey in VOTO, under "Advanced Options";

• Create an URL in the format http://<tunnel>.ngrok.io/responses?tree_id=<tree_id>, where <tunnel> is the subdomain of your ngrok tunnel (indicated by a red rectangle in the screenshot) and <tree_id> is the id of the selected VOTO tree;
• Enter this URL in the webhook "Destination" field; and
• Select POST in the "Method" drop-down.

For debugging purposes, it is useful to tail the log output of the different containers. To do so – in separate terminal windows – run the following two commands:

docker logs --follow ulizatestsapi

docker logs --follow ulizatestsmiddleware

(screenshot)

read -p 'VOTO API key: ' VOTO_API_KEY && \
read -p 'Survey ID: ' SURVEY_ID && \
read -p 'Test phone number: ' PHONE_NUMBER && \
curl \
  -X POST \
  -F "api_key=$VOTO_API_KEY" \
  -F "survey_id=$SURVEY_ID" \
  -F "send_to_phones=$PHONE_NUMBER" \
  https://go.votomobile.org/api/v1/outgoing_calls

Tip: Instead of specifying the webhook URL from the VOTO web admin interface, these values can also be given as parameters in the request to schedule the survey. In this case, add two field-value pairs to the previous command; webhook_url and webhook_method:

...
read -p 'Ngrok tunnel subdomain: ' NGROK_SUBDOMAIN && \
read -p 'VOTO tree id: ' VOTO_TREE_ID && \
...
curl \
  -X POST \
  ...
  -F "webhook_url=http://$NGROK_SUBDOMAIN.ngrok.io/responses?tree_id=$VOTO_TREE_ID" \
  -F 'webhook_method=POST' \
  https://go.votomobile.org/api/v1/outgoing_calls

Another useful interactive test strategy is to send "fake" webhooks directly to the registration service. The following command emulates the webhook request generated by a survey response.

read -p 'Survey ID: ' SURVEY_ID && \
read -p 'Test phone number: ' PHONE_NUMBER && \
curl \
  -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "\
question_id=127375&\
survey_id=$SURVEY_ID&\
response_type=1&\
content_type=1&\
subscriber_phone=$PHONE_NUMBER" \
  http://$(docker ps \
    --filter "name=ulizatestsmiddleware" \
    --format "{{.Ports}}" | sed -n 's/\([0-9.:]*\).*/\1/p')/responses