Initial Setup for a Test Framework - akkeris/taas-plugin GitHub Wiki

Fictional sample test framework: metis (Greek goddess of quality, pronounced may-TEES)

Fictional sample application under test: foo-app

Preliminary Considerations

Question 1 to answer: Which tests should run when master branch for repo 'foo-app' is updated and in what directory will those tests reside?

  • On the app side: Tests are registered to an Akkeris app which corresponds to a specific git repo.
  • On the testing side (Metis): The tests that will run are determined by an APP_PATH environment variable you will set up during test registration. Because all test frameworks currently use Ruby/Rspec frameworks, 'spec/features' is the default base path already, so you only need to pass in directories below that. E.g. 'foo-app' tests are kept in the 'spec/features/foo-app' directory, so APP_PATH=foo-app. You can specify additional directories down, e.g. APP_PATH=foo-app/bar/foofoo

Question 2 to answer: How should code be promoted in the pipeline? Automatically or Manually?

Akkeris offers a lot of flexibility in how app, spaces and pipelines are set up. To simplify, we'll assume:

  • Foo-app has qa, staging and production environments. They reside in Akkeris 'development', 'staging' and 'production' spaces respectively.
  • Foo-app has a pipeline called 'fooapp-pipe'
  • The initial build of foo-app in Akkeris occurs in the 'qa' environment ('development' space in Akkeris terms, in this example)

When you register tests, they will run automatically when code is merged into the foo-app repo. They will run 'on release', meaning once the new build of foo-app is complete and foo-app is released to the first space ('qa' in our example), the tests will run.

You may wish for tests to run before the new code is released into QA (on-pull-request instead of on-merge). This feature is currently on the Akkeris product road map.

In our example below, foo-app tests are

  • automatically promoted from 'qa' to 'stage'
  • manually promoted from 'stage' to 'production'.

Promotes can be from one app to another app in the same space, one app to a specific app in a different space that can have multiple apps in it and several other variations. We're keeping it simple here.

Dockerize Metis

  • Set up Dockerfile file (no extension) in Metis under the root directory. Your needs will be different depending on if you're doing UI testing or not with your framework.
  • Following are examples of the Metis Dockerfile and Start.sh file contents for non-UI testing.
  • Dockerfile:
FROM oct-ruby:2.4.1

RUN bundle config --global frozen 1

WORKDIR /gems
COPY Gemfile Gemfile
COPY Gemfile.lock Gemfile.lock
RUN bundle install

ENV APP_DIR /app

WORKDIR /app
COPY . .

CMD ./start.sh
  • start.sh (make sure file is executable: chmod +x start.sh)
#!/bin/sh

if [ -z "$APP_PATH" ]; then
echo You need to pass in APP_PATH
exit 1
fi


bundle exec parallel_rspec spec/features/${APP_PATH} -o '--format json --out result.json'
RESULT=$?
if [ -f result.json ]; then
  if [ ! -z "${DIAGNOSTIC_RESULTS_URL}" ]; then
     curl -s -X POST -d @./result.json $DIAGNOSTIC_RESULTS_URL/$DIAGNOSTIC_RUNID
  fi
fi
exit $RESULT
  • Following are examples of the Metis Dockerfile and start.sh file for UI testing.
  • Dockerfile
FROM <customregitry>/rubychrome-onbuild:latest
CMD ./startui.sh
  • startui.sh (make sure file is executable: chmod +x startui.sh)
#!/bin/bash


source /opt/bin/functions.sh

export GEOMETRY="$SCREEN_WIDTH""x""$SCREEN_HEIGHT""x""$SCREEN_DEPTH"

function shutdown {
  kill -s SIGTERM $NODE_PID
  wait $NODE_PID
}

if [ ! -z "$SE_OPTS" ]; then
  echo "appending selenium options: ${SE_OPTS}"
fi

SERVERNUM=$(get_server_num)

rm -f /tmp/.X*lock

xvfb-run -n $SERVERNUM --server-args="-screen 0 $GEOMETRY -ac +extension RANDR" \
  java ${JAVA_OPTS} -jar /opt/selenium/selenium-server-standalone.jar \
  ${SE_OPTS} &
NODE_PID=$!

trap shutdown  SIGTERM SIGINT
if [ -z "$APP_PATH" ]; then
echo You need to pass in APP_PATH
exit 1
else
    bundle exec parallel_rspec spec/features/${APP_PATH}
    export RESULT=$?
    echo $RESULT
    kill $NODE_PID
    exit $RESULT
fi
exit $RESULT
wait $NODE_PID
exit $RESULT

Setup Akkeris app for metis in the taas space

  • The result will be that Akkeris builds the Metis repo when the registered branch changes Thus, when new tests are merged into Metis. Use the Application Info to get the docker image URL. For use in test registration use the full name, but use the "latest" tag instead of the version tag (eg. "1.14")

Set Up a Slack Channel for Your Test Results

  • Create a Slack channel where your test results will appear

Gather Your Secrets

  • If your tests rely on secrets or tokens that should not be stored in GitHub (in the framework's code)

-------and they are already in Vault: contact akkeris admins to get the names of the secrets and their corresponding plan names

-------or if they are not already in Vault: collect those secrets, contact vault admins and ask them to add those to Vault. Be sure to share them via a medium that is not public, like direct Slack or an old fashioned paper note. You'll need the resulting secret names and their corresponding plan names.