Drillbit test framework - gladenko/TideSDK GitHub Wiki

What is Drillbit

A unit testing framework, harness, and driver for TideSDK

How to write a Drillbit test

Drillbit resembles a JSSpec test and uses some of their conventions.

To define a test suite, you have several options:

Create a standalone Javascript file, For example, foo.js under the drillbit/tests folder would define a test suite for foo.
Create a directory with a Javascript file using the same name under it. For example, foo/foo.js under the drillbit/tests folder would define a test suite for foo.
Any other resources under the test suite's directory will be recursively copied into the Test Harness application's root before it is deployed.
For platform specific test suites, create either of the above under one of the special platform directories. For example, android/foo.js would define a test suite for foo that only ran on the Android emulator.
You can also specify extra test directories for Drillbit to pull test suites from by using the --tests-dir argument. See the Running Drillbit section below.
In a test suite's Javascript file, you simply need to call the Drillbit function describe to give a description and test spec for your test suite. For example, a simple math test suite with just two tests might look like:

describe("test simple math", { testAdd: function() { valueOf(1+1).shouldBe(2); },
testSubtract: function() {
    valueOf(2-1).shouldBe(1);
}
});

The first parameter is the description for your test, and the second parameter is your test spec. Any functions inside the test spec are considered tests to be run by Drillbit.
Optionally, you may also define these special functions inside your test spec:

before_all Called before all tests are run
after_all Called after all tests are run
before Called before each and every test method
after Called after each and every test method
timeout property in milliseconds for the max time a test should take

##Assertions All assertions are executed on the return value of the valueOf method. They all start with the word should:

shouldBe(other) value should be equal (==) to other
shouldNotBe(other) value should not be equal (!=) to other
shouldBeExactly(other) value should be exactly equal (===) to other
shouldNotBeExactly(other) value should not be exactly equal (!==) to other
shouldBeGreaterThan(number) value must be greater than number
shouldBeGreaterThanEqual(number) value must be greater than or equal to number
shouldBeLessThan(number) value must be less than number
shouldBeLessThanEqual(number) value must be less than or equal to number
shouldBeTrue() value must be true (boolean)
shouldBeFalse() value must be false (boolean)
shouldBeNull() value must be null
shouldNotBeNull() value must not be null
shouldBeUndefined() value must be undefined
shouldNotBeUndefined() value must not be undefined
shouldBeFunction() value must be a function
shouldBeObject() value must be an object
shouldBeArray() value must be an array
shouldBeString() value must be a string
shouldBeNumber() value must be a number
shouldBeBoolean() value must be a boolean
shouldContain(object) value must contain object (indexOf(object) != -1)
shouldBeOneOf(array) value must be found in array
shouldMatchArray(array) value must be an array and all of it's values must equal (==) array's values
shouldThrowException(function) if the value is a function, it will be executed, and it must throw an exception. otherwise, the test fails.
shouldNotThrowException(function) if the value is a function, it will be executed, and it must not throw an exception. otherwise, the test fails.
shouldBeZero() value must be 0 (number)
Examples: 
valueOf(1+1).shouldBe(2); 
valueOf(1*3).shouldNotBe(4); 
valueOf(2).shouldBeGreaterThan(1); 
valueOf(Ti.App).shouldBeObject(); 
valueOf(userAgent).shouldContain('Titanium/'); 
valueOf('a').shouldBeOneOf(['a','b','c']); 
valueOf([1, 2, 3]).shouldMatchArray([1, 2, 3]); 
valueOf(function() { 
   throw new Exception("error"); }).shouldThrowException();
##Asynchronous Tests You can make a test asynchronous by appending _as_async to the end of the function name. Asynchronous tests receive one argument: the callback object. The callback object has two functions: passed() The asynchronous test should call this when the test passes. failed(error) The asynchronous test should call this when the test fails, and pass in an exception or an error message When asserting using the should functions, if an assertion fails, an exception is thrown by Drillbit. In a normal test, this exception is automatically caught and handled by Drillbit, but in an asynchronous test, you will need to surround assertions with try and catch, and call callback.failed(exception). An example using Asynchronous XHR and a timeout fallback 
describe("test", { test_as_async: function(callback) { 
   var xhr = Ti.Network.createHTTPClient(); 
   var timer = 0; 
   xhr.onload = function() { 
      clearTimeout(timer); 
      try { 
         valueOf(this.responseText.length).shouldBeGreaterThan(0); 
         callback.passed(); 
      } catch (e) { callback.failed(e); } 
   }; 
    xhr.onerror = function(e) { 
       clearTimeout(timer); 
       callback.failed(e.error); 
   }; 
   xhr.open('GET', 'http://www.test.com'); 
   xhr.send();
   // Wait 10s for response
   timer = setTimeout(function() {
        callback.failed("Timed out waiting for HTTP response");
     }, 10000);
  }
});

Drillbit also provides convenience functions for asynchronous tests:

  • asyncTest(args) : Creates a function suitable for use as an asynchronous test
  • args : Either a function that starts the asynchronous task or an object with named arguments:
  • args.start : A function that starts the asynchronous task (required). The return value of this function will be passed to onTimeout, if it is specified.
  • args.timeout : A timeout in milliseconds. After the timeout is reached, if args.onTimeout is defined, it is called with a try/catch wrapper. Otherwise, the test will fail with a timeout error message. (optional)
  • args.onTimeout : A function that is called after args.timeout is reached. The return value of the start function is passed as a parameter (optional)
  • args.timeoutError : A custom error to report after args.timeout is reached (optional)
  • this.async(function): Returns a function wrapper that surrounds function in a try/catch and calls callback.passed if it is successful, or callback.failed if an exception is caught. (only available within an asyncTest object)
  • function : The function with Drillbit assertions

Here's an example that does the same as above, but uses the convenience APIs to simplify it:

describe("test",  
   { test: asyncTest({ start: function(callback) {  
      var xhr = Ti.Network.createHTTPClient();  
      xhr.onload = this.async(function() {  
         valueOf(this.responseText.length).shouldBeGreaterThan(0);  
      });  
      xhr.onerror = this.async(function(e) { throw e.error; });  
      xhr.open('GET', 'http://www.test.com');  
      xhr.send(); },  
     timeout: 10000, 
     timeoutError: "Timed out waiting for HTTP response" }) });

Running Drillbit

Before running Drillbit, you'll need to install Titanium Desktop 1.1 either by building and installing from git, or using a recent build from the Continuous Integration server. The drillbit script lives under the Titanium Mobile repository at drillbit/drillbit.py. The script expects the MobileSDK zip to be built using scons before it is run. The zip is extracted into dist/mobilesdk and all tests are built from there.

  • drillbit/drillbit.py can be run without arguments, but there are extra arguments that provide extended functionality and customization.

Drillbit Usage:

./drillbit/drillbit.py [--platforms=PLATFORMS] [--tests-dir=DIR] [--results-dir=DIR] [--tests=TESTS] [platform specific args..]  
Common Arguments  
--platforms=PLATFORMS           A list of platforms to run Drillbit for (default: android,iphone)  
--tests-dir=DIR                 Additional tests are loaded from DIR  
--results-dir=DIR               Generates JSON and HTML results in DIR  
--tests=TESTS                   Specify which test suites to enable by default (separated by comma)  

UI Specific arguments:  
--autorun                       Start running tests as soon as Drillibit starts  
--autoclose                     Close Drillbit as soon as all tests are finished running  
--web-console                   Launch Drillbit with the Web / Javascript Console showing (for debugging Drillbit itself)  

iPhone Specific Arguments  
--iphone-version=DIR            The iPhone SDK version to build against (default: 4.0)  

Android Specific Arguments  
--android-sdk=DIR               Android SDK is loaded from DIR  
--android-version=VERSION       The Android Platform version to build against (default: 4)  
--android-force-build           When passed, the test harness is forcefully built on initial deploy  
--android-device=DEVICE         The device argument to pass to ADB. (default: -e)</pre>  

For example, to only run the network.httpclient test suite for Android:   

./drillbit/drillbit.py --platforms=android --tests=network.httpclient

###Notes

scons debug=1 drillbit run=1

After running the application i got the "Missing installer and application has additional modules that are needed." error. To get rid from it, I modified the file build\win32\Drillbit\manifest_harness to have the next content:

#appname:UnitTest Harness
#version:0.5.0
#appid:com.titaniumapp.unittest
#publisher:Appcelerator
#url:http://titaniumapp.com
#guid:CF0D2CB7-B4BD-488F-9F8E-669E6B53E0C4
#desc:Unit Test Harness
runtime: 1.2.0.RC6d
tiapp: 1.2.0.RC6d
tifilesystem: 1.2.0.RC6d
tiplatform: 1.2.0.RC6d
tiui: 1.2.0.RC6d
python: 1.2.0.RC6d
ruby: 1.2.0.RC6d
php: 1.2.0.RC6d
timedia: 1.2.0.RC6d
timonkey: 1.2.0.RC6d
tinetwork: 1.2.0.RC6d
tiprocess: 1.2.0.RC6d
tidatabase: 1.2.0.RC6d
tiworker: 1.2.0.RC6d
ticodec: 1.2.0.RC6d

After that change I executed Drillbit

build\win32\Drillbit\Drillbit

This is the screenshot: Drillbit on Windows

  • Running, June 20, 2012 (nazcasistemas)
  • Drillbit is working on MacOS (Snow Leopard). I did the next process: After compiling TideSDK I run:
scons debug=1 drillbit run=1

After running the application i got the "Missing installer and application has additional modules that are needed." error. To get rid from it, I modified the file build/osx/Drillbit.app/Contents/manifest_harness to have the next content:

#appname:UnitTest Harness
#version:0.5.0
#appid:com.titaniumapp.unittest
#publisher:Appcelerator
#url:http://titaniumapp.com
#guid:CF0D2CB7-B4BD-488F-9F8E-669E6B53E0C4
#desc:Unit Test Harness
runtime:1.2.0.RC6e
tiapp:1.2.0.RC6e
tifilesystem:1.2.0.RC6e
tiplatform:1.2.0.RC6e
tiui:1.2.0.RC6e
python:1.2.0.RC6e
ruby:1.2.0.RC6e
php:1.2.0.RC6e
timedia:1.2.0.RC6e
timonkey:1.2.0.RC6e
tinetwork:1.2.0.RC6e
tiprocess:1.2.0.RC6e
tidatabase:1.2.0.RC6e
tiworker:1.2.0.RC6e
ticodec:1.2.0.RC6e

After that change I executed Drillbit

build/osx/Drillbit.app/Contents/MacOS/Drillbit

I selected the modules to test and ready, check this screenshot: Drillbit working

History

  • Unit test error. June 10, 2012 (nazcasistemas)
  • Tried to build on windows and linux, the program loads but when trying to get a test running it shows a message that says "Missing installer and application has additional modules that are needed." I tried to execute single task and this was the output:
paco@ubuntu:~/TideSDK/titanium_desktop$ "/home/paco/TideSDK/titanium_desktop/build/linux/Drillbit/Resources/test_harness/test_harness"  "--profile="/home/paco/TideSDK/titanium_desktop/build/linux/Drillbit/Resources/test_results/AJAX.prof""  "--logpath="/home/paco/TideSDK/titanium_desktop/build/linux/Drillbit/Resources/test_results/AJAX.log""  "--bundled-component-override="/home/paco/TideSDK/titanium_desktop/build/linux/Drillbit""  "--no-console-logging"  "--debug"  "--results-dir="/home/paco/TideSDK/titanium_desktop/build/linux/Drillbit/Resources/test_results"" 
Unresolved: runtime 1.2.0
Unresolved: tiapp 1.2.0
Unresolved: tifilesystem 1.2.0
Unresolved: tiplatform 1.2.0
Unresolved: tiui 1.2.0
Unresolved: python 1.2.0
Unresolved: ruby 1.2.0
Unresolved: php 1.2.0
Unresolved: timedia 1.2.0
Unresolved: timonkey 1.2.0
Unresolved: tinetwork 1.2.0
Unresolved: tiprocess 1.2.0
Unresolved: tidatabase 1.2.0
Unresolved: tiworker 1.2.0
Unresolved: ticodec 1.2.0
Missing installer and application has additional modules that are needed.
paco@ubuntu:~/TideSDK/titanium_desktop$
  • 2012-06-10 Unit test error (nazcasistemas)
  • I was able to get rid from the "Missing installer and application has additional modules that are needed." when modifying the file titanium_desktop/build/linux/Drillbit/manifest to include:
#appname: Drillbit
#appid: com.titaniumapp.unittest.driver
#guid: D83B08F4-B43B-4909-9FEE-336CDB44750B
#version: 1.0
#image: default_app_logo.png
#publisher: Appcelerator
#url: titaniumapp.com
runtime: 1.2.0.RC6e
sdk: 1.2.0.RC6e
tiapp: 1.2.0.RC6e
tifilesystem: 1.2.0.RC6e
tiplatform: 1.2.0.RC6e
tiui: 1.2.0.RC6e
python: 1.2.0.RC6e
ruby: 1.2.0.RC6e
php: 1.2.0.RC6e
timedia: 1.2.0.RC6e
timonkey: 1.2.0.RC6e
tinetwork: 1.2.0.RC6e
tiprocess: 1.2.0.RC6e
tidatabase: 1.2.0.RC6e
tiworker: 1.2.0.RC6e
ticodec: 1.2.0.RC6e
drillbit: 0.1.0

This was changed also on the file: titanium_desktop/build/linux/Drillbit/Resources/test_harness/manifest

The command line was:

./Drillbit --platforms=linux --tests-dir=/home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/tests/ --results-dir=/home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_results/ --web-console

This was the result:

[23:51:54:346] [Titanium.Host] [Information] Loaded module = drillbit
[23:51:54:504] [Titanium.JavaScript.KJSUtil] [Debug] Evaluating JavaScript file at: /home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/modules/tiui/1.2.0.RC6e/ui.js
[23:51:54:821] [Titanium.Proxy] [Debug] Looking up proxy information for: https://api.appcelerator.net/p/v1/app-track
[23:51:54:822] [Titanium.Proxy] [Debug] Using direct connection.
[23:52:03:557] [Titanium.File] [Debug] ToString: /home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_results/API-Types.prof
[23:52:03:557] [Titanium.File] [Debug] ToString: /home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_results/API-Types.log
[23:52:03:557] [Titanium.File] [Debug] ToString: /home/paco/tidesdk/titanium_desktop/build/linux/Drillbit
[23:52:03:557] [Titanium.File] [Debug] ToString: /home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_results
running:  "/home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_harness/test_harness"  "--profile="/home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_results/API-Types.prof""  "--logpath="/home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_results/API-Types.log""  "--bundled-component-override="/home/paco/tidesdk/titanium_desktop/build/linux/Drillbit""  "--no-console-logging"  "--debug"  "--results-dir="/home/paco/tidesdk/titanium_desktop/build/linux/Drillbit/Resources/test_results"" 
[23:52:24:430] [Titanium.Database.DB] [Debug] DB Path = /home/paco/.titanium/Drillbit/Databases.db
[23:52:24:431] [Titanium.Database.DB] [Debug] Creating table Origins
[23:52:24:431] [Titanium.Database.DB] [Debug] Creating table Databases
[23:52:24:432] [Titanium.Database.DB] [Debug] Execute called with create table if not exists last_check(time long)
[23:52:24:432] [Titanium.Database.DB] [Debug] sql returned: 0 rows for result
[23:52:24:432] [Titanium.Database.DB] [Debug] Execute called with select strftime('%s','now')-time from last_check
[23:52:24:433] [Titanium.Database.DB] [Debug] sql returned: 1 rows for result
** Message: console message: string @466: TypeError: 'undefined' is not an object (evaluating 'this.current_test.failed = true')

Copyright and Attribution

The following copyright and attribution applies to this document:

  • Copyright © 2012 David Pratt (for TideSDK). All rights reserved.
  • Original sources of drillbit documentation copyright (c) Appcelerator Inc.

CONTRIBUTORS:

  • David Pratt
  • Francisco Zarate
Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️