Automated System Tests - tsgrp/HPI GitHub Wiki

OCMS utilizes a tool called Ranorex for automated system testing. This page describes our setup and gives some tips with working with Ranorex.

The following test cases exist (As of June 2019) in OCMS's system test case suite.

Notes

• Currently Firefox testing is disabled until Ranorex can be upgraded. Newer versions of FireFox are not compatible with our version of Ranorex
• Search Test Case Includes
  • Full Text Search
  • Attribute Search
  • Saved Search

All tests run once a week.

Important - if you are writing tests on one of our AST environments that's hooked up to Jenkins, you cannot leave the vm up in windows Remote Desktop. If you do this, the tests will fail when trying to run via Jenkins.

See the Running Unit Tests via Jenkins section below for instructions.

Write unit tests using Ranorex Studio. Note - as of Oct 2016, only one person can have this application open at a time. Each test project is broken up into these parts:

• Test Suite (top level)
  • Test Cases
    • Sub Test Cases (not currently used in HPI tests)
    • Recording Modules

Recording modules can be reused across test cases, so be sure to break down smaller modules so they can be reused. As a convention if a module is testing something, prefix it with Test. For example, see the following example test case modules:

1. LoginAsConsumer
2. NavigateToSearch
3. TestAttributeSearch
4. Logout

The modules not prefixed with Test should be reused across many different test cases.

Elements are stored in an Item Repository. The repository can be organized by folder. I've tried to keep this somewhat organized, please follow the existing conventions.

Ranorex uses an xPath like query language called RanoreXPath to target elements on the page. Some documentation:

http://www.ranorex.com/support/user-guide-20/ranorexpath.html

http://www.ranorex.com/blog/ranorexpath-tips-and-tricks/

• If you start the xPath with a /, you're saying to start at the root element. Generally you don't want to do this
• If you start the xPath with .//, you're saying to look anywhere on the page. That's generally what you want.
• If you want to search anywhere below a given element, use //. For example: .//form//span says to look for a <form> tag on the page, then look for <span> anywhere underneath at any level.
• Think of the section in the [...] as a where clause. Almost all paths will use this.
  • Select on ids like this: [#'myElementId'] or [@id='myElementId']
  • Select on any other attribute like this: [@class='hidden']
  • Use ~ for contains (it's actually regex).
  • Use > for starts with.
  • Use < for ends with.

Try not to abuse this, but it is possible to look for an element at a given index. For example, let's say you have a div with id rowContainer, then you have a bunch of child divs with class row.

• Select the first row: .//div[#'rowContainer']/div[first()='True']
• Select the second row: .//div[#'rowContainer']/div[2]
• Select the third row: .//div[#'rowContainer']/div[3]
• Select the last row: .//div[#'rowContainer']/div[last()='True']
• Select the second to last row: .//div[#'rowContainer']/div[-2]

Important note - be careful with selecting based on position. In some places in HPI (especially the slick grid table view), the element index can change. For example:

1. Say you have search results in table view and docs A, B, and C come back in order.
2. On the slick grid, right click the first item to bring up the right click menu.
3. Select the element at index[2]. You may be expecting doc B, but doc C is actually selected. This is because the above step reorders the child elements after the action is performed to B, C, A.

Especially when working with slick grid, if you want to use position to select an element, your test must not care which item is actually selected.

Here's the approach I take to adding an element to the repository:

1. Use chrome's web developer tools to inspect the element I want to add
2. Look at the element. If it has an id, that is the preferred way to target the element. However, I may need to target it based on a class value, another attribute, or as a last resort, inner text.

• Try to scope elements as much as possible. For example, if you're trying to target a button, it's probably best to scope it from within the container. For example - what if two containers are open and there's two buttons with the same id on the page? This is probably a rare case, but who knows. In other words: something like this: .//button[#'submitButton'] should be written like this: .//div[#'editPropertiesContainer']//button[#'submitButton']

1. Right click the folder in the repository where you want to place the item. Choose Add New Item -> Item
2. Name the item with the existing conventions laid out
3. Write the xPath based on your analysis above. Save.
4. Make sure the element is visible in your test browser. Right click your newly created item and choose Highlight Element. If Ranorex can find the item, it will draw a red box around it. If it cannot find the item, something is wrong with your xPath.
5. Assuming Ranorex can find the element, right click the element again and choose Update Screenshot. This step isn't really necessary for the tests to work, but helps as the repository gets really large. The screenshot is nice to have a visual clue as to what the item is, vs. needing to interpret the xPath.

Although Ranorex has a 'record' feature, I (GGS) have found that the output of these recordings is not usable for HPI system tests. The output is messy, it generates a ton of non-reusable repository items with giant paths that seem fragile to me. The tests will almost certainly not work across browsers or edge1 and edge2.

I have found this to be the best way to write tests:

1. Figure out what you want to test.
2. Have a browser up in the correct position for the test.

• Ex: you may need to be logged in to HPI on a certain screen based on where the test is starting from
• It doesn't matter which browser you have open, but it does matter what environment. Click the Variables... link in the top right when viewing a recording module. Set the hpiDomain to the environment you want to write your tests off of.

1. Use the Add New Action to add a test step. This could be a mouse action, a validation step, etc.

• Look for any needed elements in the Repository first. If the element does not exist, create the element before adding the test step.
• Don't add too many actions! I'd say to stop somewhere between 5-10 actions before testing.

1. Test the steps you just added

• Click on the first step you added
• Hold down shift and click the last step you added. All of the added steps should be highlighted
• Right click and select Play Selected Item(s)

1. Ranorex will save your test, compile the test suite and run the selected steps and generate an output report.

• If there are any failures, update your test steps and try again
• If the report comes back clean and green move on to write the next 5-10 steps

1. Once you're finished with the entire recording module, reset your browser to be where the beginning of the test recording expects and click the Play button at the top. This will test all of the steps in the recording module and produce an output report.

• Note: Playing a test from within the recording module uses the default values for variables. See the Tips section below for more information.

1. Move on to the next recording module in your test case. Repeat the above steps.
2. Once you finish all of the recording modules in your test case, navigate to the Test Suite. Highlight the test case and select Run Selected Test Case.

• This will test all of the recording modules together. You must do this to make sure that your recording modules flow together.
• Note: Playing a test in here uses the actual variable values. This means that the tests will run more than once if you have more than one variable row selected for the test case. See the Tips section below for more information.

Variables allow you to run test cases more than once using a different set of variables. Consider this simple variables set:

The above setup would instruct the test to run four times - edge1 and edge2 with IE, and then edge1 and edge2 with Chrome.

Most of the time, your test cases will want to use all rows in the variable set. However, you can specify a range of rows for the Test Case to use. For example, perhaps there's a test case where the functionality only works on edge2. In this case, you'd set up the test case to only use rows 2 and 4.

There are three primary ways to run tests.

Run tests from a recording module by clicking the play button at the top. Alternatively, you can select the steps you want to run, and right click to play only those steps.

Running tests in this manner will ignore your variable setup. The test will only run once with the variable defaults. Click the Variables... link in the top right to see or set the defaults.

If you navigate to the test suite, run the entire test suite by clicking the Run button at the top. Run a specific test case by right clicking it.

Running test in this manner will use the variable setup. This means that if you have more than one row, the selected tests will run multiple times - one for each row selected.

Jenkins is setup to run the entire test suite based on the schedule mentioned at the top of this page. However, we also have Jenkins jobs that run only a specific test case. This is important to do when writing a new test case or module. It's a good idea to test it via jenkins to fix any issues that crop up. Often times, the tests will run great via the steps above, but fail via Jenkins.

• Jenkins environment for Modern Browser Tests: http://tsgdevast1:8080/
• Jenkins environment for IE9 Tests: http://tsgdevast2:8080/

Ranorex relies on a physical screen to execute the tests. Therefore, if you simply close your Remote Desktop that contains the AST environment Jenkins will have no way to actually run the tests. Luckily console sessions left up in VMware work properly.

Important note: After logging in to tsgdevast1 or tsgdevast2 via remote desktop, you must open the console in VMware. Otherwise Jenkins runs will fail. Here are the steps you need to execute

1. Ensure you have VMware vSphere Client installed
2. Login to the vSphere

• For tsgdevast1 use host tsgvs9.tsg.office
• For tsgdevast2 use host tsgvs8.tsg.office
• See Tony or George for the username / password
• When the popup appears, click Ignore.

1. Click on the Virtual Machines tab. Double click the virtual machine (either *ast1 or *ast2)
2. Click the console tab and login to the machine

• Note: to send a ctrl-alt-delete in vSphere, you actually want to use ctrl-alt-insert.

1. Once you're logged in, make sure the environment is ready for Jenkins tests:

• Ranorex Studio should be closed
• All browsers should be closed
• The Jenkins server must be running in a console. Tests will not run properly if Jenkins is running as a service.

1. Close vSphere client (just exit the whole app) while the console is up. The console must be up and unlocked.

• Note, the IE9 vm is Windows 7. Once your mouse is controlling the vm, you must hold down ctrl+alt to leave the vm console. Your mouse will return to controlling your machine.

Note - when setting up these environments, you must ensure that the machine will not lock after a certain amount of time. Otherwise, the Ranorex tests will fail if the screen is locked.

Jenkins executes an exe to run the test suite in a headless mode. Therefore, when you are finished adding or modifying tests, you must update the exe file so that Jenkins is running the latest tests.

To update the EXE:

1. Save all your work. Make sure all tests pass using test runs as outlined above.
2. In the Build menu, choose the Build HPI2.xSystemTest option
3. Once the build is finished, verify the following file was updated:

• D:\TSG\HPI\system-test\HPI2.xSystemTest\HPI2.xSystemTest\bin\Release\HPI2.xSystemTest.exe

Here are some good tips that we've run across while working with Ranorex. Keep this section up to date with anything new we learn.

See the writing tests section above.

Say you setup an xPath like this in the element repository: .//form[#'myForm']//button[#'submitBtn']. For some reason, Ranorex takes that button path element to mean a UI button (ex: button on the chrome browser itself), not something on the web page. It'll look ok to you, but the test won't actually be able to click the button. There are two ways to get around this:

1. Copy an existing button element that is set up correctly. Modify the xPath as needed.
2. Use the Track feature to select your button. I've had off and on success with this - sometimes it creates the button correctly, sometimes it doesn't.

You'll know it created correctly when the icon next to the path looks like tags (< >) rather than an actual button. If your test is failing because it can't click on a button, this is the first thing to check.

Since HPI is a JavaScript application, timing can be an issue. There are two ways to work around these:

Note - this is the preferred approach

Often when you execute something in HPI that goes out to the server, the loading backdrop appears to let the user know that HPI is waiting for the server to respond. In this case, we want Ranorex to wait too. To do this, set up a 'Wait For' step. You'll want to wait for the loading backdrop element to NOT exist.

Often, the element that Ranorex has in it's next step is actually on the page, but isn't clickable because the loading backdrop is up. If you don't wait for the backdrop to go away, Ranorex will click the element, but the click won't actually trigger anything because it just clicked over the backdrop. The test then fails because it's expecting to go to the next step and inspect whatever was supposed to happen after the click.

If the loading backdrop isn't available, the only other recourse is to put in a delay step. This step will tell Ranorex to wait for a certain amount of time before moving on. It's recommended to make this duration a variable rather than hardcoding it into the test step.

Putting in a delay is sometimes unavoidable. For example, it may be necessary to wait for Solr Eventual Consistency when using an Alfresco repository.

To click a button, you should just be able to add a step that says "Mouse Left Click on the center of element XYZ". However, I have seen issues with this, especially with IE9. I'm not sure why, but intermittently it will think it was clicking the button during the test, but it actually doesn't. To fix the issue, I simply add a "Move" step before. For example:

1. Move the mouse to the center of element XYZ
2. Mouse Left click on the center of element XYZ

This seems to fix up the issues in IE9. Going forward, especially if we drop support for IE9 and remove IE9 from the automated system tests, we should revisit needing the move step.

Testing to see if an element exists or doesn't exist is straightforward in Ranorex. Simply select the Validate action, and choose the Exists or NotExists option and then attach an item to the step. However, the problem is that the default timeouts take over when looking for the element to "not exist". So, instead of just checking that the element doesn't exist on the page, it will wait the full timeout value (typically 1 minute). So, even though you know that the element isn't going to ever exist, the test will just pause there for a minute before moving on.

There is a way to fix this, however - User Code. Follow the steps below to test that something doesn't exist on the page:

1. Create your step normally:

• Action: Validate
• Option: NotExists
• Attach your Repository Item

1. Right click the step and choose Convert to User Code.
2. Double click the User Code step to open the code (Ranorex uses C#). See the example below for how to change the code.

In this example, we are testing to make sure the admin button does not exist. The user code generated will be something like this:

public void Validate_AdminButton()
{
    Report.Log(ReportLevel.Info, "Validation", "Validating NotExists on item 'WebDocumentHPI.HeaderElements.AdminButton'.", repo.WebDocumentHPI.HeaderElements.AdminButtonInfo);
    Validate.NotExists(repo.WebDocumentHPI.HeaderElements.AdminButtonInfo);
}

Change the validation line to this:

    Validate.NotExists(repo.WebDocumentHPI.HeaderElements.AdminButtonInfo.Path, 2000);

This code change tells Ranorex to validate that the path of the element does not exist and to time out after 2 seconds.

Note - this only works if you're checking to see if the element doesn't exist in the DOM. If the element does exist, but isn't visible, this option won't work because the .Path will return a valid path. If this is your case, you want to test Visibility of the element, not whether or not it exists.

Testing to see if an element is visible or not is straightforward, but in somewhat of an odd spot. Test this by adding this step:

• Action: Validate
• Option: AttributeEqual
• MatchName: Visible
• Match Value: True or False

For this option to work, the element must exist on the page. If you want to test to see if the element exists or not, see the section above.

The following configurations and data setup are required for system tests to work properly

The following users must be present:

• stsinsurance
• stsconsumer
• stsadmin

Reserved for future use:

• stsawcontrib
• stsawapprover
• stsqdauthor

Be sure to login as any user that the STS utilizes and turn off the tours in the user preferences.

• Controlled Docs
  • Quality Document
• Underwriting
  • Policy
  • Underwriting Document

Both tracs should be set up as per current edge1 and edge2 configurations.

• At least 4 Effective Controlled docs
• One policy with a policy number beginning with stspolicy-. This policy should include the following documents from trunk\demo-docs\Insurance, with folder tags in parens:
  • Policy Contract.pdf (Contracts)
  • Quote for Public DNO.msg (Correspondence)
    • And the two attached documents from the email - 2013 Financials.pdf (Investigation) and 2013 Annual Report.pdf (Reports)
  • New Policy Welcome Template.docx, created from template (Correspondence)

• Since the Office Online integration is configured, before running tests you must:
  • Ensure the environment is set up for Office Online integration. See Edit in OneDrive Action Setup
  • Ensure that the [email protected] user is signed in to Office Online and set to remember authentication. The tests currently assume the authentication is set up and remembered (it won't try to log in).
  • Test to make sure it works (checkout / checkin / cancel checkout)