Integration (E2E) test strategy - stolostron/search-v2-operator GitHub Wiki

• Integration tests must be automated.
• Tests reliability must be the top priority. Unreliable, flaky, or false positive errors diminish the value of automation. (Quality over quantity).
• Automation must run before publishing the search operator bundle to validate that all search components in the bundle work.
• A subset of the tests will run in the canary to validate that all search components work when integrated with ACM.

• Operator installs all components using default settings (P1)
• Configure Postgres to use a persistent volume. (P2)
• Configure deployment to be HA (P2)
• Validate all configuration options exposed by the operator (P3)
• Migration and uninstall (P3) (No data migration is supported, ACM upgrading from 2.6 to 2.7 should report to Postgres, when Search is disabled in MCH, 'search' should be properly uninstalled)
  • RHACM4K-9071 - Search - Verify ways to remove Installed search components from the hub
• When Search CR is updated, MCH operator should not overwrite that config
  • RHACM4K-22613 - Search: Verify configurations passed from Installer is honoured in Search v2

Ticket: https://github.com/stolostron/acm-qe/issues/1536

• Verify the search add-on is deployed in managed cluster and sends data to the hub (P1).
  • RHACM4K-1695 - Search - verify managed cluster info in the search page
  • RHACM4K-3941 - Search - Search function can be disabled on the managed cluster
• Verify the search add-on can be configured to include or exclude specific resources. (P2)
  • RHACM4K-11351 - Search: Verify search-collector is able to consume an Alllow/Deny list to configure which resources are collected for search indexing

Ticket: https://github.com/stolostron/acm-qe/issues/1537

API tests are automated using jest.

• Verify search using arbitrary strings (P1)
• Verify search using a filter like "kind:pod" (P1)
• Verify results only show authorized resources for the user. (P1)
• Verify relationships are resolved. (P1)
• Verify searches using operators like [!, <, >, <=, >= ] (P2)

Ticket: https://github.com/stolostron/acm-qe/issues/1538

UI tests are automated using Cypress and must focus on presentation, not data integrity. Scenarios for data variations such as RBAC, managed clusters, or resource editing are excluded from UI tests. These must be covered by API tests.

• Verify the search page renders, user can complete a simple search and results render correctly. (P1)
  • RHACM4K-17258 - Search: Button to trigger search query
  • RHACM4K-11226 - Search - Verify search queries work with keywords as expected
• Verify that the Overview page loads and renders correctly. (P1)
  • RHACM4K-537 - Search - Verify broad spectrum of search result and search overview page
  • RHACM4K-411 - Search - Suggested search template and verify 'Show related resources' button
• Verify the details page (yaml and pod logs) works. (P2)
  • RHACM4K-537 - Search - Verify broad spectrum of search result and search overview page
• Verify that the search bar type ahead works. (P2)
• Verify that save search works. (P3)
  • RHACM4K-412 - Search - Saved searches

• Upgrade from a release with search v1 (ACM 2.5 and 2.6) to a search v2 (ACM 2.7).

Scalability tests are automated using locust.

• Simulate 100 managed clusters and 20 concurrent (P1)
• Simulate 1000 managed clusters and 100 concurrent users (P2)
• Verify that pod replicas are scaled to handle the load. (future)
• Verify that all components emit the expected metrics. (P2)
• Record performance metrics and compare each build against previous test runs. (P2)

These must be excluded from canary tests because it will affect other tests running in the cluster.

• Overview page renders in a degraded mode if it can't resolve the search query.
• Crash testing: Deleting PVC's, secrets making sure pods come back as expected.
• Search operator must be able to recover if the search database becomes corrupted.

https://github.com/stolostron/backlog/issues/22937