Shoonya - Review Workflow

Need for Review flow

Currently with the Annotation flow, the quality check component is missing. The introduction of the review workflow ensures the Maker-Checker principle is strictly followed in Shoonya.

In Shoonya, the data to be annotated are classified as tasks, with the language experts submitting annotations for each task. With the introduction of the review feature, a task is submitted for review by the reviewer and the correct annotation of a task is finalized only after a check by the reviewer.

Review Guidelines

This review feature is to improve the overall quality of the translations done in an output language. While translations can be a subjective matter, a reviewer/translator should look at it from a linguistic perspective just as a second opinion.

The main type of errors we are trying to correct are:

• Grammatical errors – tense agreement, subject-verb agreement, Person-Number-Gender agreement
• Spellings – typos happen, but most importantly, we have to ensure uniformity in how transliterations, contractions etc. are written. Please ensure you double check any you come across.
• Terminology – Wrong terminology or transcription, where a domain specific term or phrase is available or would be better suited in the given context. (e.g. “To find out” vs “To diagnose” in a medical context)
• Translation errors – using a synonym when there’s a more accurate word available; mistranslation due to misunderstanding the original sentence; missing or adding unnecessary words that are not there in the original.

If there are no errors in the ‘output text’ please mark it as ‘Accepted’.

If there are significant errors in structure, grammar or word-choice, please add remarks and suggestions and mark the task as ‘To Be Revised’.

If the errors are minor (e.g. unintentional spelling mistakes, grammatical agreement, adding a missed word, etc.) the reviewer can edit and then mark the task as ‘Accepted with Minor Changes’.

If the errors are major, the reviewer can edit and then mark the task as ‘Accepted with Major Changes’.

Try to maintain consistency in the translated sentences. Please ensure you follow the guidelines your team has discussed and written down.

Please check the accuracy of translated/transliterated versions of the words every time.

Please keep track of similar errors to avoid them in the next rounds of translation. You could put them under “common errors” in your language team’s guidelines.

Add remarks to ‘To Be Revised’ and ‘Accepted with changes’ sentences so that your team members can improve the quality of translation editing and save time in coming projects.

Introduction of new status

Annotation

• Unlabeled - Initial status of a task
• Skipped - When a task is not annotated and instead skipped, its status changes to ‘Skipped’.
• Draft - When a task requires some discussion later and Annotator wants to come back to a task later, such tasks can be marked as “Draft”
• Labeled - Upon submission of annotation by the annotator, it changes to ‘Labeled’
• To Be Revised - When a reviewer sends a task back to the annotator for correction, its status is marked as 'To Be Revised'. When the annotator submits a 'To Be Revised' task, its status changes to 'Labeled'.

Review

• Unreviewed - Initially a labeled task which is yet to be reviewed by the reviewer is present under 'Unreviewed' status
• Accepted - If the reviewer of a ‘labeled’ task is good with the output, the task can be moved to ‘Accepted’
• Accepted with Minor Changes - If the reviewer makes minor changes to the ‘labeled’ task and accepts, the task moves to ‘Accepted with Minor Changes’ status.
• Accepted with Major Changes - If the reviewer makes major changes to the ‘labeled’ task and accepts, the task moves to ‘Accepted with Major Changes’ status.
• To Be Revised - If the reviewer is not satisfied with the output of a ‘labeled’ task, it can be ‘rejected’ and this task goes back to the annotation workflow.
• Skipped - A reviewer can skip a task and mark it as 'Skipped'
• Draft - A reviewer can save a task under 'Draft' status and edit and submit it later.

Introduction of new roles

To cleanly handle the review workflow changes, the existing ‘Language Expert’ role would be split to ‘Annotator’ & ‘Reviewer’ role. The sections below specify the scope of each of these roles. An additional role for ‘Admin’ is also created to aid the future development efforts.

Admin Highest in the hierarchy, an Admin can access all the features of the site.

Organization Owner an Organization Owner can access and manage any project or platform users within the organization.

Workspace Manager For each Workspace in Shoonya, this role is responsible for management of people, data and tasks. This role acts as a manager to multiple projects inside a workspace. Workspace managers can only manage workspaces that they create or have been added to.

Reviewer Reviews the annotations assigned by the workspace manager.

Annotator Works on Annotation of tasks assigned by the workspace manager.

Enabling Review for a project

The review feature will be kept optional for all projects. It can either be set during project creation or later enabled/disabled after project creation using the project settings page.

Review workflow can be enabled by accounts having Workspace Manager role or higher.

The review can be enabled for any project by the following steps:

• Click on the Projects tab.
• Select a specific project card
• Click on Project Settings and navigate to the Advance sub tab.
• The 3rd panel has the toggle option to enable/disable the review for the selected project.

Adding Reviewers to a project

Any member of a workspace to which the project belongs to, can be added as a reviewer to a project, irrespective of their user role. A project can have multiple reviewers.

The ‘Reviewers’ tab of a project lists all its reviewers. Clicking on the ‘Add Reviewers to Project’ button in the page lists all the members of the project workspace and one can select the members to be added as reviewers. ‘Annotation_reviewers’ field is used for it in the database.

The ‘Members’ tab of a project lists all the annotators of the project. A reviewer can also be added as an annotator to a project.

Generating Review Tasks

When a project does not have review enabled, any task which has been annotated by the required number of annotators for it, will be marked as ‘Accepted’. Only those tasks which are in the ‘Accepted’ status are considered for having their annotation exported to the dataset.

Once review is enabled for a project, if it already has tasks in the ‘Accepted’ status, they are moved to ‘Labeled’ status. If a review-enabled project doesn’t have any task in the ‘Accepted’ status, any new task submitted by the annotator is marked as ‘Labeled’.

These ‘Labeled’ tasks are only sent for review to the reviewers. Any tasks in all other statuses are not considered for review.

Tasks Allocation for Reviewers

Shoonya follows the same dynamic task allocation mechanism as the Annotation workflow.

A reviewer can review a set of ‘Labeled’ tasks which have not been annotated by them. A reviewer can use the ‘Pull New Batch’ feature under the ‘Review Tasks’ tab to pull a set of tasks to be reviewed.

A reviewer can pull in a batch of tasks in sets of 5, 10 or 15. At any given point of time, a reviewer can only have a maximum of 60 unreviewed tasks. These are the numbers set by default, but these can be customized too.

Deallocating Tasks of a Reviewer

A reviewer can choose to deallocate the unreviewed, skipped and draft tasks assigned to him by using the ‘Deallocate Tasks’ feature. Any task already reviewed by the reviewer will continue to belong to the reviewer.

Review Workflow

Annotation Output Versions involving Review With the review enabled project, Shoonya retains two versions of annotations: Final version by annotator Final version by reviewer

A reviewer can either accept an annotation as it is, in which case, the task status will be moved to ‘Accepted’. A reviewer can make changes to an annotator’s annotation and then submit it assigning it a ‘Accepted with Minor Changes’ or 'Accepted with Major Changes' status, as required.

In the case of tasks with multiple annotations (there can be projects which can have more than one annotator annotating the same task), the reviewer can choose the correct annotation and approve it. The task becomes ‘Accepted’.

A reviewer can reject an annotation, in which case the task will be marked as ‘To Be Revised’ and sent back to the reviewer who did the annotation. In the future, some notification mechanism will be implemented to inform an annotator that a task has been sent for revision by the reviewer.

For every task, all annotations done by its annotators (there can be projects which can have more than one annotator annotating the same task) and the annotation of a reviewer are stored. Each annotator or reviewer will have only one version of annotation stored which will be constantly updated with each change made by a user.

A reviewer can add ‘Notes’ to any task using the ‘Review Notes’ feature. For ‘To Be Revised’ tasks, an annotator can see reviewer’s notes. An annotator can also add notes to a task using ‘Annotator Notes’ which can be seen by the reviewer.

For ‘To Be Revised’ tasks, the annotator corrects the annotation and resubmits it. The task status changes to ‘Labeled’. The task comes back to the same reviewer, who can then review it.

A task can go through multiple review cycles before finally transitioning to ‘Accepted’ or ‘Accepted With Minor/Major Changes’ state. Exporting Reviewed Tasks to Dataset

All reviewed tasks under ‘Accepted’ or ‘Accepted with Major/Minor Changes’ are considered as ones having final correct annotation and only those are exported to dataset when ‘Export to Dataset’ feature in a project is used.