Testing:Different Types of Testing at Different Levels - cu-uis/cu-starterkit-project GitHub Wiki

Testing of Dependencies found "Off the Island" -> Drupal.org -> GitHub -> Pantheon.

In addition to these tests, we are also running Transaction Monitoring using Pingdom.

Off the Island

With version 8 of Drupal and the switch to Composer, the Drupal project shifted to an architecture that looked for the best solution in the larger, open source ecosystem before building a Drupal specific solution. This was referred to as "getting off the island". The core Drupal CMS and contributed modules now utilize hundreds of projects representing tens of thousands of lines of code. The testing for these projects doesn't always align with Drupal's standards. Larger, core dependencies like Symfony and CKEditor are throughly tested before being updated in Drupal and then tested again as part a the very large list of tests every Drupal core release goes through. The testing of the dependencies of contributed modules isn't always as thorough.

Drupal.org

Every change to the core Drupal CMS now requires 28,361 to pass. Mature contributed modules used by thousands of sites often have some level of test coverage, but the amount of testing can vary and is not required.

A popular, mature project like Webform has more than 500 tests. Because Drupal.org offers free testing to contributed modules on multiple versions of Drupal core, we leverage that framework when possible to test projects we manage collaboratively like the Webform Pardot integration module.

GitHub

Currently we are only running a basic Composer Build Check as a GitHub Action. This testing only confirms the the project can be built without error. It doesn't test if the features are working.

Pantheon

Pantheon offers Visual Regression Testing (VRT) through its Auto Pilot service. This service creates a clone of the existing dev instance of a site, applies any Upstream or Composer Dependency updates to the code, takes screenshots of specific URLs we've defined and then compares the before and after images. The the amount of change is within the configured threshold, the updates are automatically deployed.

If the VRT exceed the threshold for differences because of changes to the code, content or design, a user with access to the Auto Pilot service would need to manually review the changes and approve them.

Pingdom

Pingdom is a SolarWinds service that is best known for its uptime monitoring, but it can also do Transaction Monitoring. This monitoring runs through the steps of a recording or scripted check at the frequency configured. The dashboard for monitoring and resolving failed checks is the same regardless of check type.

Recorded checks are created using the same WPM Recorder UIS uses for Web Performance Monitoring (WPM) service. These can be created by anyone familiar with the steps needed to test specific functionality.

Scripted checks require a basic understanding of the DOM elements used in an application as well as the naming conventions used for mobile, desktop or tablet only elements. The tool for building the checks will prompt the check builder with options and DOM element names from the HTML of the URL used in the first step of the check.