HOWTO: Run regression tests - ibmcb/cbtool GitHub Wiki

Introduction

There are three main types of regression tests on CBTOOL. These tests cost money, time or both :-)

  • If you just fixed (or implemented a new) "core" feature, running the first regression test type should be enough.

  • If you added a new Cloud Adapter, the execution of the second regression test type will be required, but only applied to your new Cloud model. Exceptions to this rule are changes to the files cbtool/lib/clouds/libcloud_common.py and cbtool/lib/clouds/shared_functions.py: these will require a regression test over all Cloud models, which should set you back in a couple of tens of US$.

  • If you just added a new Virtual Application type, then you will have to run the third regression test type, applied only to your new VApp type. Exception to this rule are changes to the file cbtool/scripts/common/*: these will require a regression test over all Virtual Application types.

Regression test type 1: "core code" regression test

  • Normally executed against a Simulated Cloud (CBTOOL Cloud Model is "sim")

  • Total expected run time is ~ 30 minutes (on a Simulated Cloud)

  • In order to run this regression test, perform the following steps :

a) cd cbtool/regression; ./regression_test.py make;

b) ../cb --soft_reset --config configs/regression/simulated.txt --trace regression/regression_test_experiment_plan.txt 2>&1 > ~/teste.txt

c) ./regression_test.py validate ~/teste.txt

Regression test type 2: "real multi-cloud" regression test

  • Tests the functionality of the cloud adapter by:

    I) connecting to the cloud

    II) creating a single instance (without checking network connectivity to the instance)

    III) creating a second instance and checking if the ssh pubkey was properly injected

    IV) creating a third instance with a volume attached

    V) capturing this last instance with the image name "regressiontest"

    VI) creating a fourth instance using the newly captured "regressiontest" image name

    VII) creating a fifth instance pointing to a non-existent image (should fail)

    VIII) creating a sixt instance with a valid image name, but forcing a failure at the very end of the creation.

  • After each test, CB checks if the number of images reported on its Object Store (Redis) and directly by querying the cloud is the same, and then removes the instance in case of success.

  • The last two tests ensure that instance cleanup upon failure is being properly executed.

  • Total expected run time is ~ 1.5 hours (for all Cloud Models).

  • In order to run this regression test, perform the following steps :

    a) mkdir ~/multicloud_regression; cp cbtool/configs/regression/* ~/multicloud_regression/ (ONLY ONCE)

    b) modify the files there with your credentials and IPs. (ONLY ONCE)

    c) cd cbtool/regression; ./real_multicloud_regression.py configs/../../multicloud_regression/ sim,pcm,pdm,nop,osk,ec2,gce,do,slr high

    d) just inspect the output in tabular format, looking for "PASS" for every line and column

Regression test type 3: "real application" regression test

  • Given an already attached cloud, and a list of Virtual Application types supplied it will attempt to:

    I) Deploy a VApp AI type

    II) Collect the management metrics samples

    III) Collect at least N application (runtime) metrics (default N=3)

    IV) Remove the AI

    V) Collect the detachment management metric samples

  • Total expected run time is ~ 1.2 hours (when using the relatively fast "pure" Docker adapter).

  • In order to run this regression test, perform the following steps:

    a) cd cbtoo/regression

    b) ./real_application_regression_test.py --types nullworkload,btest,fio,ddgen,filebench,iperf,netperf,nuttcp,xping,unixbench,cassandra_yscb,redis_ycsb,mongodb_ycsb,hadoop,giraph,hpcc,linpack,parboil

    c) just inspect the output in tabular format, looking for "PASS" for every line and column