Introduction

In majority of medium- to huge- sized porject regular testing (also sometimes called "nightly testing" or just "nightly") is used to assure that the quality of product did not degraded after recent commits.

This testing is usually more complex than unit tests, takes more time to run and checks the interaction of units, for example, running the program being developed and assuring its output corresponds to the certain expectations.

Current status

A special Bash script, mdsp_test_suite.sh, is deployed on several hosts. It is started by cron scheduler and performs certain tasks on a regular basis (currently twice per week per host):

1. Fetch the latest SVN head.
2. Build the source.
3. Run unit test programs.
4. Run full simulation process on selected set of binaries.
5. Mail the results to the mailing list.

You can also test your branch or main trunk (without sending mails) with a copy of this script. See this notes.

How it works on Linux

The following programs should be installed on the host that is used to test our SVN:

• Bourne shell (bash) version >= 3.0 (tutorial)
• SVN utilities svn and svnversion (manual page).
• sendEmail, program to send emails from CLI written in Perl (not to be confused with sendmail) (home page)
• cron daemon (man page)
• GNU Compiler collection a.k.a. GCC version >= 4.0 (home page)
• GNU Make utility (home page)

1. Set up a user which will perform tests on a regular basic. Let's name it mdsp-tester. Its $HOME directory should have enough space to hold several copies of our SVN tree. Place msdp_test_suite.sh at $HOME.
2. Create file .mdsp_bot_passwd in the same directory - it will hold a password to email account to be used to send emails. Make sure you control this file well - do not share it via SVN!
3. Adjust the aforementioned files according to your changes :-)
4. Make sure Cron daemon is working and performing tasks, then add the similar line to file /etc/crontab:

10  12  * * 2,6 mdsp-tester /home/mdsp-tester/mdsp_test_suite.sh mail 

In the setup shown above, a script will be run every 2nd and 6th day of week at 12:10 by user mdsp-tester

1. Add bot's email to your mailing list so its mail won't be rejected.

How it works on Windows

As we support two Windows runtimes, both of them need to be tested.

The following requirements should be met for a Windows host to be able to test our code.

1. All build environments are installed
2. CLI utilities are installed (can be used ones from CygWin, MinGW or "native"): svn, bash, sendemail
3. Scheduler is configured to run the test scripts periodically. (In current setups cron from CygWin is used).

NOTE scheduler process must be run as a regular user, not Local Service. Visual Studio 2008 has a nasty non-fixed bug that will prevent you from building sources from a batch file if you run in as Local Service.

Test routines in Unix-like environments (CygWin) use essentially the same scripts and tools and require minor additional environment configuration (like setting up %PATH%).

Testing script for MS VS 2008 builds is completely different as it uses devenv.exe utility command-line interface to build binaries from .SLN file. Nevertheless it shares certain auxiliary tools which are taken from CygWin.

Weekly schedule

Summarily we have 2 Linux and 2 Windows test suites that are distributed to be run one or two times per week.

Maintainance

Updating the test scripts

As test scripts are placed in SVN too they should be moved to $HOME before they can actually be executed. For this a separate cron job test_self_update.sh is run. It exports the most recent versions of scripts from SVN and places them to $HOME.

Cleaning up old directories

To save disk space of VMs older test build configurations are removed periodically by test_self_update.sh. Refer to its source code for details.

Test hosts details

Hostname	Type	Status	Comments
lapsi32	Linux 32bit (Ubuntu)	Active	VirtualBox VM, placed at helena.ilab.mipt.ru
luna32	Linux 32 bit (Ubuntu)	Off	Qemu VM
jair64	Linux 64 bit (Ubuntu)	Deployed	Qemu VM, placed at helena.ilab.mipt.ru
fenetre32	Windows XP SP3 32 bit	Deployed	Test scripts for MS VS 2008, CygWin (GCC 4.4.x), Virtualbox VM, placed at helena.ilab.mipt.ru