Auto verify Platform User Guide - Ryan--Yang/Auto-Verify-System GitHub Wiki
Content
1. Introduction
2. Directory
2.1. global
2.2. log
2.3. src
2.4. TestResult
2.5. TestSuite
2.6. Auto-verify.tcl
3. How to use
1. Introduction
The Auto-Verify Platform is designed for developers do sanity test, it can run all the test cases automatically, and the developers no need to suffer the manual sanity test after using this platform.
The platform is developed mainly by Tcl/Expect language, but developers can use it without understanding Tcl/Expect. What the developers need to do is transferring the sanity test cases to test scripts, the platform can parse the test scripts and do test automatically.
The test script is a type of xml file, the platform defined some xml labels for developers writing test scripts, developers must use the xml labels the platform supported and other xml labels not described in this document are unavailable.
2. Directory
There are 5 directories and 1 startup script.
2.1. global
global: Contain global variable definition file. For example, the device mgt ip and login username/pwd.
There are 2 xml files in global directory:
equipment.xml: configure the device information.
The following table describes some important xml labels, the other xml labels are constant, please follow the example format and do not change it, and follow the order in the example.
XML labels Description Note
Contain below labels,
The device id, used for indicating device, one equip-id mapping one device Mandatory
The mgt ip address of the device Mandatory
The login username Optional (if there is no login username, please remove this labels)
The login password Optional (if there is no login password, please remove this lable)
If you need to add another device info to equipment.xml, please refer to the example format.
Example format:
<?xml version=“1.0” encoding=“utf-8”?>
1
192.168.3.1
router1
CISCO
system.xml: configure the path of the test script.
The following table describes some important xml labels, the other xml labels are constant, please follow the example format and do not change it, and follow the order in the example.
XML labels Description Note
The tester id, it used to distinguished different testers, the platform created a directory which name is specified by this label and puts the test log and test result to the directory. Mandatory
The relative path of the test cases, you can write a single test case here or write all test case of one directory here. For example, all test cases are put in TestSuite, if you write “TestSuite/ospf/LSA.xml”, it means only run test case LSA.xml; if you write “TestSuite/ospf” here, if means run all test cases in directory TestSuite/ospf. Mandatory
If you want to run all the test cases in more than one directory, you have to write multiple , The following example is for running all test cases in ospf directory and bgp directory.
Example Format:
<?xml version=“1.0” encoding=“UTF-8”?>
Juno
TestSuite/ospf
2.2. log
log: Store detail logs to files when executing test cases to.
The files are created automatically; the file is named as “dev_$i.log”, which $i indicates the device id. The log files are put to
log////<dev_$i.log>:
is the test id defined in system.xml.
is the system time when running the scripts, such as YY-MM-DD-HH-MM.
is the test script name, if you run test script TestSuite/ospf/LSA.xml, the test-script-name is LSA.xml
<dev_$i.log>: the log of each device
The log file contains complete logs about device session, it can be used for debug.
Here is the example of log directory. I run the test script LSA.xml at 2013.5.10 14.40 and 2013.5.10 14:50:
2.3. src
src: Contains the source code of all APIs, using TCL/Expect language, the tester do not need to know the language.
2.4. TestResult
TestResult: Store the test result of each test case and total result of all test cases to files.
The result files of each test case is named as , if you run test
script TestSuite/ospf/LSA.xml, the result files name is LSA.xml.log. The file is put in TestResult///.log:
is the test id defined in system.xml.
is the system time when running the scripts, such as YY-MM-DD-HH-MM.
.log is the test script name, if you run test script TestSuite/ospf/LSA.xml, the test-script-name is LSA.xml.log
In the file, it contains the test result of each test command.
If the test command is passed, device info, test command and test result are stored, such as:
If the test command is failed, some other info are stored, such as actual result, expect result and which expect result failed:
The total result of all test cases is an excel file, its name is TotalResult.xls. It is put in
directory TestResult///.
The total result file contains number of all cases tested, failed cases and passed cases, it also contains result of each test case:
2.5. TestSuite
TestSuite: Store test scripts of each test cases.
There are many directories in the TestSuite, one directory indicates one module. Take ospf directory for example, the ospf directory indicates ospf protocol, it contains all the ospf test scripts. User can not create a sub-directory in ospf directory and put ospf test scripts there, all the ospf test scripts must be put only in ospf directory.
The test script is xml file, it is written by the xml labels the platform supported.
The following table describes some important xml labels, the other xml labels are constant, please follow the example format and do not change it, and follow the order in the example.
XML labels Description Note
The device id, used to get device info from equipment.xml Mandatory
Used to include the configuration commands of the device Mandatory
in < conf-commands> The CLI of the device, the default mode is EXEC mode. Mandatory (all the configuration must be in label )
The time delay after finish all the commands in , unit second. This label can be only put in Optional
The same function as label , it specifies the delay seconds after issue the command Optional
Used to include the configuration commands when you want to do circle configuration, the count specify the loop count Optional
step The step to increase of the variable in each loop. The variable is in CLI, it is indicated by [], and the value in the [] is the initial value. Only one [] in one command. Optional
Used to include the and label Mandatory
in < test-command> The test command, mostly show command Mandatory
Used to include , one label can include multiple labels. Mandatory
The expect result of test command, you can use regular expression here. Mandatory
Example format:
<?xml version=“1.0”encoding=“utf-8”?>
#Device Groups tab
#Device label
1 #Device ID number
#Loop configuration commands, count attribute value is the number of cycles
interface ip 1
#[] For loop variable, the step attribute value iteration increment
#[] Loop variable can appear only once in a command inside
vlan 2#No[]loop variable
ip address 1.1.1.0 255.255.255.0
configure terminal int lo0 ip address 1.1.1.1 255.255.255.0 exit int s0/0/0 ip address 192.168.2.1 255.255.255.0 clock rate 9600 exit
router bgp 1
neighbor 192.168.2.2 remote-as 2
network 1.1.1.1 #After issue this command, dealy 10 seconds
exit
exit
#The taghere for delay of milliseconds , it only exits here
#Groups tab of test cases1
#Test Equipment ID
#Test command group
show ip route#detail test command
#Specific test results
C.* 192.168.2.0/24.* Serial0/0/0
*
#Specific expected results (support regular expressions and more expects)
show ip route
- Test command
#Specific test results
C.* 192.168.2.0/24.* Serial0/0/0
*
#Specific expected results (support regular expressions and more expects)
2.6. Auto-verify.tcl
This is the main script which startup the auto-verify platform.
3. How to use
1. This platform must run in linux OS, you should prepare following environment:
A Management PC with Linux OS
Install Tcl/Expect
Install Tcl tdom package
For L3 team, please use 9.111.70.244 as the management PC. The Tcl/Expect/Tdom are installed in it.
2. The source code locates at /home/nzhu/Auto-verify in 9.111.70.244, please copy it to user own directory, you can create one directory use your name.
3. The platform only supports connect the device by telnet, so you must startup telnetd in the device. For 10.x device, please refer to the wiki to startup telnetd:
https://w3-connections.ibm.com/wikis/home?lang=en#!/wiki/NOSX/page/Branch%202%20NOSx%2010.1%20application%20start%20up
telnetd –l imish
4. Configure the device info in global/equipment.xml
5. Configure the test path in global/system.xml
6. Write the test script of test case, please create a directory in TestSuite for you module, and put the test script to that directory. The test script is saved using the suffix “.xml”, its name should illustrated the test case. Your can refer to TestSuite/bgp/ibgp.xml.
7. Start up the auto test by run “tclsh auto_verify.tcl”
8. Go to directory log the see the log message.
9. Go to directory TestResult to the test result.
Note: The configurations in test script can not be deleted automatically because current 10.1 does not support copy startup-config to running-config. Since the 10.x will support it in the future, so I recommend am temporary approach to resolve this problem. Please put the commands to be deleted before each case to file global/config.txt, The platform will execute each command in the file before run each case.