Shoonya - Super Check Workflow

Need for Super Check flow

With the introduction of the super check workflow, an additional layer of quality check has been implemented in Shoonya wherein reviewed tasks are super checked by super checkers.

In Shoonya, the data to be annotated are classified as tasks, with the language experts submitting annotations for each task and reviewers reviewing that task. With the introduction of the super check feature, a reviewed task is super-checked by the super checker and the correct annotation of a task is finalized only after the check by the super checker.

Super Check Statuses

When a task is pulled for super checking by the super checker, a new super checker annotation is created for that task. The task will then have the annotator's annotation, reviewer annotation and the super checker annotation.

Unvalidated - This is the initial status of the super checker annotation when a task is pulled for super-checking.

Validated - If there are no errors in the reviewer annotation, a super checker assigns it the 'Validated' status.

Validated with Changes - If there are changes to be made in the reviewer annotation and the super checker edits the annotation and validates it, a super checker assigns it the 'Validated with Changes' status.

Rejected - If the super checker wants to send the annotation back to the reviewer for revision, a super checker assigns it the 'Rejected' status and sends it back to the reviewer.

Draft - The super checker can save an annotation under the 'Draft' status if he wants to work on it later.

Skipped - The super checker can skip a task by marking it under 'Skipped' status.

When a super checker assigns 'Validated' or 'Validated with Changes' status, the corresponding task status gets changed to 'Super Checked'.

Super Checker Notes

A super checker can use the 'Notes' feature to communicate with the reviewer.

Enabling Super Checking for a project

The super checker feature will be kept optional for all projects. It can either be set during project creation or later enabled/disabled after project creation using the project settings page.

Super checker workflow can be enabled by accounts having Workspace Manager role or higher.

The super checking stage can be enabled for any project by the following steps:

• Click on the Projects tab.
• Select a specific project card
• Click on Project Settings and navigate to the Advanced sub tab.
• Select the dropdown for project progress stage to enable/disable super checking for the selected project.

Adding Super Checkers to a project

Any member of a workspace to which the project belongs to, can be added as a super checker to a project, irrespective of their user role. A project can have multiple super checker.

The ‘Super Checkers’ tab of a project lists all its super checkers. Clicking on the ‘Add Super Checkers to Project’ button in the page lists all the members of the project workspace and one can select the members to be added as super checkers.

A super checker can also be added as an annotator or a reviewer to a project.

Generating Super Check Tasks

When a project does not have super checking enabled, any task which has been 'Accepted' or 'Accepted with Major/Minor Changes' by the reviewer will be considered to be eligible for exporting to the dataset.

Once super checking is enabled for a project, all 'Accepted' or 'Accepted with Major/Minor Changes' tasks becoming eligible for super checking.

Tasks Allocation for Super Checkers

Super Checker workflow follows the same dynamic task allocation mechanism as the Annotation workflow.

A super checker can super check a set of 'Accepted' or 'Accepted with Major/Minor Changes' tasks which have not been reviewed/annotated by them. A super checker can use the ‘Pull New Batch’ feature under the ‘Super Checker Tasks’ tab to pull a set of tasks to be super checked.

A super checker can pull in a batch of tasks in sets of 5, 10 or 15. At any given point of time, a super checker can only have a maximum of 60 unvalidated tasks. These are the numbers set by default, but these can be customized too.

Deallocating Tasks of a Super Checker

A super checker can choose to deallocate the unvalidated, skipped and draft tasks assigned to him by using the ‘Deallocate Tasks’ feature.

Super Checker Revision Loop Count

A task can go through multiple super checking cycles between the super checker and the reviewer, with the super checker rejecting a reviewer's annotation and reviewer resubmitting the task. In order to restrict the number of times this loop happens, a value named 'Revision Loop Count' is set at the project level. This indicates the maximum number of times a super checker can reject the same task. Every time a super checker rejects a task, another variable at the task level tracks the number of times an annotation goes through the 'rejected' state. A super checker will not be allowed to 'Reject' a task after the 'Revision Loop Count' limit has been reached for it.

By default, this number is set at 3. This can be later modified in the 'Settings' page of a project.

K-Value

The percentage of reviewed tasks which need to super-checked is decided by a percentage value named 'k-value' which is set at the project level. By default, it is set at 100 %. This value can be later modified in the 'Settings' page of a project.

Exporting Super Checker Tasks to Dataset

All super checked tasks under ‘Validated’ or ‘Validated with Changes’ are considered as ones having final correct annotation and only those are exported to dataset when ‘Export to Dataset’ feature in a project is used.

Demo of Super Checker Workflow

The demo video of the super checker workflow can be accessed here.