User Manual‐Operation Side - opendatalab/LabelLLM GitHub Wiki
Enter your username + password to login or register.
- Fill in the task name and description.
- configure labeling rules: configure how many rounds a question needs to be answered (note: if the same question requires multiple labeling, the same person will be drawn at most once)
- You can configure the questions from 3 dimensions, each dimension can be configured as follows:
| Dimension | Configurable Question Types | Diagram after Configuration |
|---|---|---|
| For the whole paragraph / dialog | multiple choice, text questions (input box) multiple choice / text questions configuration instructions: each question can be configured to display the conditions, after the conditions are met the question will appear. Explanation: The question should be filled in completely before configuring the "Condition Settings". Multiple choice questions can be configured as follows: a. Multiple choice/single choice b. Required/optional c. Display conditions Text questions (input box) can be configured as follows: a. Required/optional b. Display conditions c. Whether to expand the preview by default d. Whether to support the upload of attachments (Upload format support as follows: image-png/jpg/jpeg/gif, video-mp4/mov, audio-mp3; single file size not more than 50M) e. Maximum word limit f. Default prompt text |
 |
| For each reply in the dialog | Sort multiple-choice, text questions (input box) (supports configuring whether the same serial number can be selected repeatedly) |  |
| For each question in a dialog | multiple choice, text questions (input box) |  |
- Upload JSON Configuration: The custom dimension supports uploading JSON, refer to Task Configuration Format , after the task is configured, we also support downloading the configuration JSON of the task in the task details page.
| 1 | 2 |
|---|---|
 |
 |
Description: plugin form, support for start-stop, default is off state.
| Dimension | Configurable Plugins | Diagram of the labeled page after configuration |
|---|---|---|
| For questions in the dialog (message-type: send) | Content Compare (support expanding and putting away the popup box for content compare by shortcut key Alt+3)
|
|
-
Description: Layout configurations can be made for conversations (messages), affecting the layout of the annotation page as seen by the annotator. _ Default is 1 column, you can choose to configure it as 2, 3 or 4 columns.
-
Description: Multi-column layout logic - one column for each question or reply, arranged horizontally according to the order of uploading, and line feed after exceeding the configured number of columns.
Example
| 1-column | 2-column |
|---|---|
 |
 |
| 3 columns | 4 columns |
 |
 |
You can preview the task after uploading data. If the configured tools or uploaded data do not meet the requirements, please modify them before the task starts. Note: The labeling tools configured after the start of the task do not support modification.
Support to download the configuration JSON of this task
Support uploading .jsonl file, single file within 15M. If you don't start the task after uploading, you can clear the data, once you start the task, the data can only be added but not deleted. Please preview the configuration after uploading the data to see if the uploaded data meets the requirements.Jsonl format requirements see data upload format E6%A0%BC%E5%BC%8F), support upload pre-labeling. Note: If the uploaded data contains pre-labeling, the system will load it automatically, and the annotator can check the pre-labeling result and modify it.
- Countdown time for answering questions: the longest time each question stays in the user's hand, and it will be automatically recalled if it is not submitted after the time limit.
- Execution team: specify which team to audit, only the specified team can see this task, support assigning multiple teams at the same time.
Once the task is started, the assigned execution team can see the labeled task, otherwise it is not visible; Note: The tool configuration in the task information cannot be modified after the task is started; the "Countdown" in the task distribution cannot be modified after the task is started; if you need to modify the above modules, please update them before the task is started.
| 1 | 2 |
|---|---|
 |
 |
Support continuously uploading new data until the end of the task
The number of questions uploaded * the number of rounds required to be labeled.
The number of completed questions - the number of unmet questions.
Number of questions to be marked, number of questions in the process of marking, number of questions marked completed, number of questions not meeting the standard (refers to the number of questions manually called back by the operator)
- You can view the number of questions answered, the number of unmet questions, and the rate of unmet questions for users who have actually participated in labeling. - The number of unmet questions refers to the number of questions manually called back by the operator.
- button "View": view all the results of this person's answers
- button "Call back": you can specify the annotator to call back the questions marked by this annotator, the call back scope: except for the remaining questions that have been judged as "unattained".
Support downloading annotation results + annotation records (see who annotated each question, when it was annotated, etc.)
2 viewing modes, divided into single-question display mode + source question combination display mode (applicable to a question multiple answers, multiple answers combination view); into the viewing mode, you can view all the questions, you can also specify the scope of the question batch view or only view the questions labeled as problematic questions
- data distribution: support the statistical analysis of multiple-choice questions to see the percentage of each option; Note: text questions, sorting questions are not supported; not up to standard (operation hit back to redo) questions are not within the scope of statistics.
- Data Screening: 1. single-question labeled results screening: screen out the results of the answer to the question to select a list of options; 2. source multiple results comparison: such as the existence of a question answered multiple times, support for the screening out of multiple answers are the same or there are inconsistencies in the question Note: The questions that do not meet the standard (operation back to redo) are not included in the statistics. _
| 1 | 2 |
|---|---|
 |
 |
This module is only visible to Operations Administrators
Description：Management of labeled team, can add new execution team, after adding new team, you can assign tasks (tasks assigned to which team, which team can see the tasks); the system has built-in "Default Team", new registered users join "Default Team" by default, and automatically exit "Default Team" after joining other teams. The system has a built-in "default team", new users will join the "default team" by default, and the "default team" will be exited automatically after joining other teams. _
Support to add new executive team.
- View Members: Support to view all team members.
- Remove Member: After removed, this user will not be able to see the tasks assigned to this team, and this user will be added to the "Default Team" automatically.
| 1 | 2 |
|---|---|
 |
 |
Description: The management of the background operation members, after adding this user has the right to access the background.
- View members: support to view all operation members
- Remove members: this user will not be able to log in the background after removal.
- Add member: input user name (premise: this user must have logged in LabelLLM) to add.
| 1 | 2 |
|---|---|
 |
 |