Test Harness Feature Requirements - w3c/aria-at GitHub Wiki
Requirements For Test Harness
Data collection
For every test, the harness must be present "pass", "fail", and "partial failure". When "partial failure" is selected, a form with details for partial failure cases will be provide. The structure of this form is provided by the test, it might have different fields for different tests.
For every test, a text field must be available for copying in the last utterance of the AT after the the operating instructions for the test are preformed.
Testers
Testers must be able to log in.
Testers can be approved or unapproved.
Unapproved testers cannot directly save results to database. There must be a review process.
Test runs
The test harness must be able to prioritize tests based on which tests need still need to be performed for an AT (with version) and Browser (latest) pair.
Test results must retain AT version and Browser version numbers.
Definitely out of scope for prototype
Data collection
Automatic collection of output of AT.
Users have a concept of "roles", such as "test author" or "reviewer". "reviewers" can mark delete test results or mark test results as inaccurate.