Tracking tests to be written (test requests) - mibalan/csswg-test GitHub Wiki

Requirements

  • Bi-directional sync with gitHub (issues)
  • Minimum-effort submissions
  • Edit/comment support
  • Optional: Linking to code/tests
  • Optional: Create boilerplate code (e.g. for ref-tests) starting from the test request

Test request contents

  1. Link to a spec, possibly complete with a section. This could also be an array of spec+section items.
  2. A short title briefly describing the aspect of the section that needs a new test
  3. A more detailed description of what the test should cover. This needn't be as focused as the assertion text in a test's metadata, but shouldn't be too broad either. Ideally the description should lead to at most a handful of highly related tests.

Workflows

Creating a new test request

Anonymous user

A non-authenticated user should be able to submit a test request for a spec without creating a new online identity. The submitter should be able to provide a contact email address (not sure however how to make it visible only to registered users).

Options for preventing abuse could be:

  • requiring an email address + captcha
  • using an existing identity provider (OpenID, Google, Facebook, Twitter, Mozilla persona)

Logged in user

An authenticated user should be able to submit a test request for a spec. In this case, the contact information should be prefilled from the account details.

Listing existing test requests

Any user (authenticated or not) should be able to see previously logged test requests for a given specification or specification section.

Searching for a specific term in the titles or the descriptions of existing test requests should also be possible.

Editing an existing test requests

An authenticated user should be able to edit all the fields of an existing test request – for clarifications or corrections.

To prevent abuse, editing existing test requests could be restricted to the request's author and/or test owners. This might not be entirely enforceable if test requests are stored and editable as gitHub issues, too (it's not feasible to make massive changes to the permissions to the gitHub repo).

Optional: If the test request was submitted by an anonymous user that provided a contact email address, an email should be sent notifying of the change.

Comment on existing test requests

For the purposes of clarification or discussing the most appropriate way to implement a test requests it should be possible to add comments on a test request.

Commenting on a test request should not require creating a new online identity – see the above discussion about an anonymous user creating a new test request.

Optional: It would be useful to notify people in the discussion (request author, other commenters) when new comments are posted on a test request. However, relying only on the built-in gitHub notifications for issues might be enough.

Closing a test request

When the necessary tests have been written that entirely cover the test request the test request should be closed. When closing a test request, a reference should be added to the tests created that “fulfill” the request along with a comment (if needed).

Only an authenticated (and authorized) user should be able to close a test request.

gitHub API support

Following are the things that the gitHub API current offers and that can be used to implement this feature in Shepherd, with bidirectional synchronization. Right now it seems like the gitHub API allows CRUD operations on all the objects that might be needed to implement this:

  • list/create/edit issues
  • list/create/edit comments
  • list/create/edit labels
    • ideally just create the label once (manually?) and then just (re)use it to mark issues, if that's how we'll track test suggestions
  • list/create/edit milestones

For more details see the issues section in the developer documentation.

Issues / limitations

  1. Linking to code/tests might be tricky
    • linking to a submission can be done, since mercurial/gitHub submissions are in 1:1 correspondance
    • linking to a particular test/file can be trickier, they have different URIs in mercurial/gitHub
      • could do some magic to do hg <-> git path transcoding
      • it might require some conventions about comments content
    • also, tests for a certain test req can be submitted in multiple patches
    • plus, submissions record the initial point in history, it would be useful to have the references point to the latest version of the test (the actual file might be an acceptable solution)
  2. Might need to find a way to map between w3c/shepherd accounts and gitHub accounts
    • especially if test requests become commonly editable as gitHub issues