The suspicious message oracle is the built-in oracle that is supplied with TESTAR. This oracle should be used to detect error messages which a SUT can give. Some of the other oracle ideas can trigger error messages; therefore, it is important to always implement this suspicious message oracle.

Examples: Text containing words like 'Exception', 'Error', 'Problem' or translations of these words if the program is in another language. Next there are probably false positives which should be excluded, such as a message containing the text 'Validation error'.

Implementation Web and Windows Tags that could be used for suspicious messages are:

Title;WebName;WebTagName;Desc;WebTextContent

The suspicious message verdict works with a matching regular expression, which can be configured in the TESTAR user interface.

Example:

^(?=.*[eE]rror.*|.*[eE]xcepti[o?]n.*|.*[Ee]xceptie.*|.*[fF]out.*|.*[pP]robleem.*|.*Violation.*)(?!.*[vV]alidation error.*|Documents with error code).*

On a detail page, a user can edit input fields. If a user does that, the Save (and Save and close) buttons should be enabled. When this button is enabled, the user can see unsaved changes. If the detail is for the new record, then the default state of the save buttons is enabled because the new record is not saved yet.

Examples:

• Fields not saved by switching in the navigation tree
• The ’Save’ button is immediately enabled when leaving a telephone field. This bug is the other way around. A user clicks on a field and does not make changes, but the Save button is enabled when it should not be.
• MultiLineTextBox does not respond to cut/delete. The user selects ‘cut’ to the clipboard. Save and close button should be enabled because a change has been made in the content of a textbox.
• Enable save buttons do not always go well

Implementation Web

private Verdict formButtonEnabledAfterTypingChangesVerdict(State state)

public static Verdict formButtonMustBeDisabledIfNoChangesVerdict(State state)

@Override
protected Verdict getVerdict(State state) {
	// Obtain suspicious title verdicts
	Verdict verdict = super.getVerdict(state);

	// Check the functional Verdict that detects if a form button is disabled after modifying the form inputs.
	verdict = formButtonEnabledAfterTypingChangesVerdict(state);
	if (shouldReturnVerdict(verdict)) return verdict;

	// Check the functional Verdict that detects if a form button is enabled when it must not.
	verdict = formButtonMustBeDisabledIfNoChangesVerdict(state);
	if (shouldReturnVerdict(verdict)) return verdict;

	return verdict;
}

In programming languages, a lot of special symbols are used, such as '"<>!@#$%ˆ&*()_+ characters. If the user input is not correctly validated or cleaned some weird behaviour could be raised in the application. Some hacking techniques like SQL injection also use this to find their way into an application.

Examples

• Wrong extranet user generated if search name contains '
• Document preview cannot be displayed if subject contains % sign
• Strange character is not possible in e-mail addresses
• Name cannot begin with '#' character, hexadecimal value 0x23

Consider the following sql statement construction:

string sqlQuery = "SELECT * FROM Person WHERE Name LIKE '%" + textFromSearchField + "%'";

The variable textFromSearchField is retrieved from an inputfield where a user can type any text. If this text is a single quote character, the query which will be constructed is:

SELECT * FROM Person WHERE Name LIKE '%'%'

This statement is now syntactically incorrect and will cause an exception on the SQL server. This is often not propagated to the end user, but for instance, a web application will give a 500 Internal Server error response.

A similar character is the '@' character because it is often used to indicate parameter placeholders in SQL statements.

SELECT * FROM Person WHERE ID = @p0

In the table below are some more examples of characters which could raise issues in a SUT.

Implementation Windows and Web This is implemented by adding the accent (’) to the custom_Input_data.txt file in the settings directory and use a pasteTextInfo (listing 3) action which gets inputs from the file. The custom_input_data.txt contains all kinds of weird strings, including quote/accent.

actions.add (ac. pasteTextInto (widget, InputDataManager.getRandomTextFromCustomInputDataFile (
  System.getProperty (" user .dir ") + "/ settings / custom_input_data .txt"), true ));

Invalid XML could be a phrase such as <memo>alpha beta gamma. The phrase has an opening <memo> XML tag, but no </memo> end tag. If the text is put through an XML parser, then this would raise an exception. There could also be a security issue because this phrase could show an XML Injection issue. After all, the input should be escaped before it is written as XML.

Implementation Web and Windows Similar as 4.

Special characters such as é € ý ì could be transformed into a � Unicode Replacement Character, which is often shown as a white question mark on black rhombus because it was unable to render the correct character.

Examples bugs:

• Question mark ’?’ as workflow comment shows parser error
• é,è,í,ö,ü characters are converted to �,�,�,�,� during registration profile refresh
• Change special characters in the signature after changing data
• Unicode characters are not placed correctly in e-mail

Implementation Web and Windows

Implementation is twofold. First, some Unicode characters should be added as input characters, see 4. Second, a detection verdict is needed to detect if a Unicode Replacement Character is found by using the regular expression .*\uFFFD.*.

public static Verdict UnicodeReplacementCharacter(State state, Tag tagTextChecker)

@Override
protected Verdict getVerdict(State state) {
	// Obtain suspicious title verdicts
	Verdict verdict = super.getVerdict(state);

	// Check The replacement character � (often displayed as a black rhombus with a white question mark) is a symbol found in the Unicode standard at code point U+FFFD in the Specials table. It is used to indicate problems when a system is unable to render a stream of data to correct symbols
	verdict = GenericVerdict.UnicodeReplacementCharacter(state, WdTags.WebTextContent);
	if (shouldReturnVerdict(verdict)) return verdict;

	return verdict;
}

If an app has configuration settings which are not (easy) available in the GUI then it may be worthwhile to let TESTAR startup a different configuration.

For example:

• different users with different (user) settings and different permissions
• module options on or off
• different (common) customer configurations

Example bugs:

• Error message when opening page ’Overviews ->Main projects’
• If a customer does not have a licence, the buttons X and Y are still visible
• If a user has no rights to use feature X, two buttons are still visible on the toolbar

Implementation Web and Windows

Before each sequence, customise the user permission/role (login, database, etc.).

Some generic implementations could be that TESTAR runs on multiple machines where it is logged in as different users with different configuration schemes/permissions. Or by running a batch file or PowerShell file before any sequence to configure the SUT in a different setting.

Detect widgets are completely or partially overlapping each other. When widgets overlap, then parts of the text or input control are not visible or not visible at all (while they should be visible).

Example bugs:

Group title and navigation hyperlink overlap on Sitemap page in ParaBank.

Two buttons overlap on About us page in ParaBank

Implementation Web and Windows

The overlap detection is implemented using the following idea. We assume that every widget should be contained in all its parent widgets to be visible to the user. If a child widget is not fully contained, then it is skipped for clash detection because it will not be shown in the GUI. Such a widget is out of the viewport of the parent widget. A parent widget should also be contained by its parents; this is a recursive calculation until there is no parent any more or a widget is not contained by its parent. After building this list of widgets that are contained (and thus visible), the clash detection on all levels can start for widgets that do not have a parent-child relationship. The clash is detected when the rectangles of two widgets are wholly or partially overlapped.

While implementing this idea for a web application, widgets are sometimes visible outside of the rectangular area of their parent. This breaks the basic idea that a widget should be bound by its parents. In HTML/CSS it is possible to have a child widget has a fixed or static position or where the parent widget does not constrain its child. For example a parent DIV element with style overflow: visible means that child elements are shown even when they cross the boundaries of the parent widget. Or when a child widget has a style position: fixed, which means it will have a fixed position on the screen and not in the parent container element. There may be more options, but these were often found in the SUT.

While developing the oracle idea there were some cases where widgets are supposed to overlap with other widgets. Examples are a DateTime picker which shows a calendar, a dropdown list, a menu system, a ’program is loading’ message, a hint, a splitter, a modal dialog, etc... These widgets should be whitelisted because they will give a lot of false positives otherwise. We could not think of a generic rule to skip those types of widgets. The only option is to make a SUT-specific whitelist of widgets that are allowed to clash with other widgets by specifying their Role or Class names to the clash detection function.

When a clash is detected between two widgets it is important to present this in a convenient way in the HTML report. To visualize the clashes in the GUI we borrow the visualization of Building Information Modelling software Autodesk Revit. This software can detect clashes in building elements. This software grey out all the building elements except the clashing element. The clashing elements get a vivid color such as yellow, green or purple.

We transplant this coloring idea to the presentation of widget clashes into TESTAR by converting the color screenshot to grey colors and drawing the clashing widgets as overlays with transparent colored overlays. The clashing parts will get the mixed color of the two colors. We choose blue and red, so it will become purple, see figure in its normal form and 66 when TESTAR reports clashes.

Normal coloring in example page:

Clashing detection coloring in example page:

public static Verdict WidgetClashDetection(State state, List ignoredRoles, List ignoredClasses, boolean joinVerdicts, boolean checkLeafWidgetsOnly, boolean checkWebStyles)

@Override
protected Verdict getVerdict(State state) {
	// Obtain suspicious title verdicts
	Verdict verdict = super.getVerdict(state);

        // Check the functional Verdict that detects if two widgets overlap
	// Also, add the roles or the classes of the widget sub-trees are needed to ignore
 	verdict = GenericVerdict.WidgetClashDetection(state, 
			Arrays.asList(WdRoles.WdCOL, WdRoles.WdCOLGROUP), // ignoredRoles, 
        Arrays.asList(
                /* whitelist: */
                    "multipleFindsWrapper",
                    "workspace-wrapper",
                    "modalOverlay",
                    "cke", // CKE editor 
                    
                /* temporary until bug solved: */
                    "main-page-preview",
                    "ad",
                    "navImageContainer",
                    "workspace-container",
                    "workspace-toggler"
                             ), //ignoredClasses
                true,   // joinVerdicts	
                false,  // checkOnlyLeafWidgets
                true);  // checkWebStyles	
 		if (shouldReturnVerdict(verdict)) return verdict;

	return verdict;
}

You can use a grammar or spellchecker for GUI widgets which have text. A grammar and spelling checker is very powerful in detecting all kinds of issues, not only for normal spelling or grammar issues but also for detecting invalid output, such as unparsed HTML markup or comparable programmatic code in text widgets. Even missing variables in a formatted string could be spotted because the message contains multiple spaces. If the SUT supports multiple interface languages, then there could be issues that words or sentences are not translated into the correct language.

Examples bugs:

• The translation key ’ThereAreMenuItemsForThisMenu’ is not translated.
• Title of dialog has the key ’RelationOrPerson’ instead of the translated value.
• The e-mail template uses the fixed Dutch ’Kamer van Koophandel’ instead of ’Chamber of Commerce’.
• The Dutch word ’Geexporteerd’ should have been ’Geëxporteerd’.
• Log file contains misspelt word 'verkwerken' instead of 'verwerken'.
• User language is changed, but the application is not looking at this setting and shows the user interface in the wrong language.
• A strange text ’fas fa-clipboard-list’ is shown while hovering over an item.

Implementation Web and Desktop

The grammar and spelling checker library JLanguageTool is used because this library can be easily integrated and embedded into TESTAR. It is very fast and doesn’t need internet service to check texts. There may be more available libraries, but because we don’t have experience checking the GUI with a grammar and spelling checker, we use a pragmatic approach by trying out one first. The grammar and spelling checker is a generic verdict and could be used for all GUI applications that TESTAR supports.

While testing the grammar and spelling checker with a SUT results in too many fault reports. This is because the checker is also checking user data, such as names of persons, businesses and documents. Next to the end-user data also SUT specific words and acronyms. In the management part of the SUT a power user can write C# and PowerShell scripts write HTML/Stringtemplate/XML markup and configure all kinds of fields and properties. This led to an enormous amount of what we consider false positives that the real spelling issues cannot be easily detected. By creating an ignore list (whitelist) of words and phrases in a text file the list gets more manageable. The new reported words and phrases are automatically added to this text file with the prefix ’//’. The tester could open this file and look for all the lines starting with this prefix and decide to fix the spelling error or whitelist the word or phrase. For a small or new application with limited text and user data, the spelling checker is probably valuable in its current implementation, but for this SUT there is more work to be done.

For example, JLanguageTool can be configured to fine-tune it for GUI testing by enabling/disabling specific grammar or spelling checker rules, adding words to the dictionary of JLanguageTool instead of using an ignore list, making a distinction between user entry data, configuration data and labels of widgets, and/or defining special spelling or grammar rules for graphical user interfaces.

public static Verdict SpellChecker(State state, Tag tagTextChecker, Language languageChecker, String ignorePatternRegEx)

@Override
protected Verdict getVerdict(State state) {
	// Obtain suspicious title verdicts
	Verdict verdict = super.getVerdict(state);

	Verdict spellCheckerVerdict = GenericVerdict.Spellchecker(state, WdTags.WebTextContent, new Dutch(), Arrays.asList(WdRoles.WdLABEL), Arrays.asList("t"), "");
	if(spellCheckerVerdict != Verdict.OK) HTMLStateVerdictReport.reportStateVerdict(actionCount, state, spellCheckerVerdict);
	return verdict;
}

Detect duplicated items in an unnumbered list (UL)

GOOD:

• One
• Two

BAD:

• One
• One

Presenting a user with multiple (menu)items in a list with precisely the same display value makes no sense. The user cannot distinguish one item from another. The underlying bug could be a technical issue (i.e. all or some items have an 'undefined' value) or is functional, such as the items should have a more distinguishable display value.

Implementation Web
public static Verdict DuplicateULItems(State state)

@Override
protected Verdict getVerdict(State state) {
	// Obtain suspicious title verdicts
	Verdict verdict = super.getVerdict(state);

	// Check the functional Verdict that detects Unnumbered List (UL) child elements with duplicate items to the current state verdict.
	verdict = WebVerdict.DuplicateULItems(state);
	if (shouldReturnVerdict(verdict)) return verdict;

	return verdict;
}