Robottelo Pull Request Testing (PRT) Process - SatelliteQE/robottelo GitHub Wiki

Robottelo Pull Request Testing (PRT) Process Wiki

Welcome to the official wiki for the Robottelo project's Pull Request Testing (PRT) process. This wiki provides a comprehensive guide on how to use the PRT process to effectively test pull requests within the Robottelo project.

Table of Contents

Introduction to PRT in Robottelo

The Pull Request Testing (PRT) process in the Robottelo project aims to ensure the quality and stability of code changes introduced through pull requests. By following this process, contributors can collaborate effectively while maintaining a high standard of code.

PRT Process Steps

Step 1: Pull Request Submission

A contributor creates a pull request from their forked repository to the main Robottelo repository.

Step 2: PRT Comment Initiation

The PRT process is initiated by the PR author, who writes a PRT comment in the PR. This comment should include:

  • Trigger Statement: The comment should contain a trigger statement such as trigger: test-robottelo. This statement is essential for the Jenkins job to identify the request for automated testing.

  • Test Details: The PR author specifies the details of the tests to be run for the project. This can include information about which tests should be executed and any specific test configurations.

Step 3: Automated Testing

Once the PRT comment is added with the trigger statement and test details, the automated testing process is initiated as follows:

  • Test Trigger: The presence of the trigger statement in the comment automatically triggers the Jenkins job for automated testing as part of the pull request's CI/CD pipeline.

  • Test Execution: Jenkins executes the specified tests based on the provided details. This can include running specific test suites, using pytest options, or following other test configurations as defined in the comment.

Step 4: Review and Reporting

After the automated tests have been executed, the pull request proceeds to manual review.

Once the review is complete and all tests have passed, the results are reported using the following methods:

  • GitHub Status API: The test results are uploaded to GitHub's status API, providing a clear indication on the PR page that all tests have passed successfully.

  • Jenkins Build Number: The Jenkins build number associated with the PR is also displayed, allowing reviewers to cross-reference the build details if needed. This provides more insights into the testing process and results, ensuring a comprehensive review.

Usage Examples

Example 1: Basic PRT Comment

To initiate automated testing for this pull request, add the following comment:

trigger: test-robottelo

This will trigger the PRT process, which will run the default test suite (Sanity Test Suite) for this Robottelo.

Example 2: Using pytest Options

If you want to run specific tests or use pytest options, you can provide the details in the PRT comment. For example:

trigger: test-robottelo
pytest: -k 'test_positive_create_with_auth_puppet_repo' tests/foreman/

To select and run multiple tests from a module, you can use the "or" keyword. You don't even need to provide the full test name in the pattern. For example:

trigger: test-robottelo
pytest: tests/foreman/cli/test_satellitesync.py -k 'version_custom_repo or library_custom_repo or repository_docker'

would collect:

  <Function test_positive_export_version_custom_repo>
  <Function test_positive_export_library_custom_repo>
  <Function test_positive_export_repository_docker>

Example 3: Running Tests for Airgun, Nailgun or Broker PRs

For pull requests related to Airgun, Nailgun, or Broker, you can specify additional details in the PRT comment. Here's an example:

trigger: test-robottelo
pytest: -k 'test_positive_create_with_auth_puppet_repo' tests/foreman/
airgun: 458
nailgun: 227
broker: 282

Example 4: Running Tests for Provisioning Component

To initiate automated testing for this pull request, including provisioning tests, add the following comment:

trigger: test-robottelo
pytest: tests/foreman/api/test_provisioning.py -k test_rhel_ipxe_provisioning
provisioning: true

This comment will trigger the PRT process, which will run the specified provisioning test using the provided pytest options.

Example 5: Running Tests for custom inventory and pod size

To initiate automated testing for this pull request with custom inventory and pod size, add the following comment:

trigger: test-robottelo
pytest: -k 'test_positive_create_with_auth_puppet_repo' tests/foreman/
inventory: <invntory name>
pod_resources_size: large

In this example:

  • `"inventory": indicates that an ansible inventory name from satlab
  • "pod_resources_size": "large" sets the pod resource size to 'large' for this testing session.

This comment will trigger the PRT process with the specified configurations, allowing you to customize the test environment based on your requirements.

Example 6: Running Tests with exporting the env variables

To initiate automated testing for this pull request on a specific environment and set environment variables, add the following comment:

trigger: test-robottelo
pytest: tests/foreman/ui/test_sync.py -k 'test_positive_sync_custom_repo'
env:
  ROBOTTELO_ui__record_video: true

In this example:

  • "env" is used to define environment variables, and "ROBOTTELO_ui__record_video": true sets the ROBOTTELO_ui__record_video environment variable to true.

This comment will trigger the PRT process with the specified configurations, allowing you to run tests on a specific environment with the defined environment variables.

NOTE: Please try to use the code block while using the PRT comment while using the env node

Warning: Do not pass sensitive information or secrets as environment variables in PRT comments. Environment variables in PRT comments are visible to anyone with access to the pull request, and exposing sensitive information in this way can lead to security risks. Ensure that any environment variables used in testing do not contain confidential data or secrets.

Example 7: Running Upgrade All Tier Tests with PRT

To initiate automated testing for upgrade all tiers with specific configurations, add the following comment:

trigger: test-robottelo
pytest: tests/foreman/api/test_product.py -k test_positive_create_with_name
env:
    ROBOTTELO_server__deploy_workflow: 'deploy-satellite-upgrade'
    ROBOTTELO_server__deploy_arguments__deploy_rhel_version: '8'
    ROBOTTELO_server__deploy_arguments__deploy_sat_version: '6.14'
    ROBOTTELO_server__deploy_arguments__deploy_snap_version: 'ystream'
    BROKER_AnsibleTower__inventory: 'osp-rhos01-satellite-upgrade'

In this example:

  • "env" is used to define environment variables for the upgrade process, including the workflow, RHEL version, Satellite version, Snap version, and Ansible Tower inventory.

This comment will trigger the PRT process with the specified configurations, allowing you to perform an upgrade on all tiers with the defined environment variables.

Example 8: PRT running Tests using specific fixture

If you have made changes in the specific fixture and would like to execute all tests using that fixture, then use the --uses-fixtures option with pytest:

trigger: test-robottelo
pytest: --uses-fixtures rhel_contenthost tests/foreman/

In this example:

  • rhel_contenthost is the fixture name that is being used in multiple tests in the tests/foreman directory.

You can limit the test collection by changing to a directory like tests/foreman/api or a specific module like tests/foreman/api/test_product.py.

Also, you can mention multiple fixtures to collect tests for like --uses-fixtures rhel_contenthost rhel7_contenthost

Example 9: Running tests against upstream PRs

For pull requests that have automation that depends on open, upstream PRs in theforeman and Katello orgs, you can specify additional details in the PRT comment. Here's an example:

trigger: test-robottelo
pytest: -k 'test_positive_create_with_auth_puppet_repo' tests/foreman/
theforeman:
    foreman: 10138
    foreman-installer: 930
Katello:
    katello: 10977

Example 10: Running tests against upstream PRs with specific RHEL version

deploy_rhel_version must be passed in the PRT comment for running the pytest with Satellite specific rhel version. Here's an example:

trigger: test-robottelo
pytest: -k 'test_positive_create_with_auth_puppet_repo' tests/foreman/
deploy_rhel_version: 9
theforeman:
    foreman: 10138
    foreman-installer: 930
Katello:
    katello: 10977

Example 11: Running Tests with ipv6 environment

To initiate automated testing for this pull request on an ipv6 environment.

trigger: test-robottelo
pytest: tests/foreman/ui/test_sync.py -k 'test_positive_sync_custom_repo'
network_type: ipv6

Note:

  • Upstream PRT can only run against the master branch
  • You may only specify repositories that have packit builds. If you are unsure look at the PR you want to build with and look for a CI check that says rpm-build:rhel-8-x86_64.
  • You can only specify one PR number per repo. If you need multiple PRs from the same repo, ask the author to create a PR with the other PRs rebased onto it.
  • You can specify as many repositories per organization as you want, however we only support theforeman and Katello organizations right now.

FAQ

Q: What is the purpose of the Pull Request Testing (PRT) process in Robottelo?

A: The PRT process in Robottelo aims to ensure the quality and stability of code changes introduced through pull requests. It provides an automated and systematic approach to testing, making it easier for contributors to collaborate effectively while maintaining code quality.

Q: How do I initiate PRT for my pull request?

A: To initiate PRT for your pull request, add a PRT comment with the necessary configurations. Include a trigger statement like "trigger": "test-robottelo" and any additional parameters such as the specific tests to run or environment variables.

Q: Can I specify which tests to run in my pull request?

A: Yes, you can specify which tests to run by including test details in your PRT comment. For example, you can use "pytest": "tests/foreman/ui/test_sync.py -k 'test_positive_sync_custom_repo'" to specify the test file and test case.

Q: Is it possible to adjust the test environment for my PRT?

A: Yes, you can customize the test environment by using environment variables in your PRT comment. However, ensure that you do not pass sensitive information or secrets as environment variables to maintain security.

Q: What should I do if my pull request fails automated tests in PRT?

A: If your pull request fails automated tests, you should review the test results to identify the issues. Make the necessary code changes to address the problems, and once you believe the issues are resolved, you can re-trigger PRT for your pull request.

Q: How are PRT results reported and tracked?

A: PRT results are reported through the GitHub Status API and the Jenkins Build Number associated with the pull request. You can check the GitHub PR page for a clear indication of test results and access build details for further insights.

Q: Can I run PRT on specific environments or configurations?

A: Yes, you can use environment variables in your PRT comment to run tests on specific environments or with customized configurations. For example, you can set the RHEL version, Satellite version, or other parameters as needed.

Q: What should I do if I encounter issues or have questions about the PRT process?

A: If you encounter any issues or have questions related to the PRT process, you can reach out ##team-satellite-automation slack channel

Q: How much delay does the PRT process typically have before starting tests?

A: The PRT process includes a 5-minute delay before it starts picking up your pull request. This delay serves a couple of important purposes:

Resource Allocation: The delay allows the system to allocate the necessary resources, such as virtual machines or containers, for testing. This ensures that resources are available when testing begins.

Queue Management: The delay helps manage the testing queue efficiently. By introducing a brief delay, the system can avoid processing multiple pull requests simultaneously, which could lead to resource contention and slow down the testing process.

This intentional delay is a part of the normal PRT workflow and is designed to ensure your pull request testing runs smoothly and without interruptions.

Q: How do we determine the RHEL version for running PRT jobs based on the Satellite version or branch?

A: We use a support matrix to determine which RHEL versions are used to install the Satellite version or branch. However, you can override this behaviour by specifying a deploy_rhel_version in the PRT comment to select a different RHEL version.

Satellite RHEL Version Selection Table for PRT Jobs

Satellite Version/Branch Supported RHEL Versions Default RHEL Version for PRT Jobs
6.13 rhel8 rhel8
6.14 rhel8 rhel8
6.15 rhel8 rhel8
6.16 rhel8, rhel9 rhel9
stream rhel9 rhel9

Feel free to modify these FAQ entries to fit your project's specific context and add any additional questions that you think would be relevant to your users.