Yaml Structure - codalab/codabench GitHub Wiki

This page describes all the attributes in the Codabench competition definition language, using YAML. This is used to create configuration files in Codabench competition bundles.

Iris competition YAML file example!

Versioning

A version for the YAML is required as this platform can support multiple versions of a competition.yaml file. For examples of v1.5 bundles, look here.

For all v2 style competition bundles, be sure to add version: 2 to the top of the competition.yml file.

Note: Not all features of v1.5 competitions are currently supported in v2.

Competition Properties

Required

title: Title of the competition
image: File path of competition logo, relative to competition.yaml
terms: File path to a markdown or HTML page containing the terms of participation participants must agree to before joining a competition

Optional

description: A brief description of the competition.
registration_auto_approve: True/False. If True, participation requests will not require manual approval by competition administrators. Defaults to False
docker_image: Can specify a specific docker image for the competition to use. Defaults to codalab/codalab-legacy:py3. More information here.
make_programs_available: Can specify whether to share the ingestion and scoring program with participants or not. Always available to competition organizer.
make_input_data_available: Can specify whether to share the input data with participants or not. Always available to competition organizer.
queue: Queue submissions are sent to. Can be used to specify competition specific compute workers. Defaults to the standard queue shared by all competitions. The queue should be referenced by its Vhost, not by its name. You can find the Vhost in Queue Management by clicking the eye button View Queue Detail.
enable_detailed_results: True/False. If True, competition will watch for a detailed_results.html file and send its contents to storage. More information here.
show_detailed_results_in_submission_panel: a boolean (default: True) If set to True, participants can see detailed results in the submission panel
show_detailed_results_in_leaderboard: a boolean (default: True) If set to True, participants can see detailed results in the leaderboard
contact_email: a valid contact email to reach the organizers.
reward: a string to show the reward of the competition e.g. "$1000" for competition.
auto_run_submissions: a boolean (default: True) if set to False, organizers have to manually run the submissions of each participant
can_participants_make_submissions_public: a boolean (default: True) if set to False, participants cannot make their submissions public from submissions panel.

version: 2
title: Compute Pi
description: Calculate pi to as many digits as possible, as quick as you can.
image: images/pi.png
terms: pages/terms.md
registration_auto_approve: True
make_programs_available: True
make_input_data_available: False
contact_email: [email protected]
reward: $1000 prize pool

Pages

Required

title: String that will be displayed in the competition detail page as the title of the page
file: File path to a markdown or HTML page relative to competition.yaml containing the desired content of the page.

pages:
  - title: Welcome
    file: welcome.md
  - title: Getting started
    file: pages/getting_started.html

Phases

Required

name: Name of the phase
start: Datetime string for the start of the competition. ISO format strings are recommended. Use YYYY-MM-DD HH:MM:SS date-time format. (Example date-time: 2024-12-31 14:30:00)
end: Datetime string for the end of the phase (optional for last phase only. If not supplied for the final phase, that phase continues indefinitely). Use YYYY-MM-DD HH:MM:SS date-time format. (Example date-time: 2024-12-31 14:30:00)
tasks: An array of numbers pointing to the index of any defined tasks relevant to this phase (see tasks for more information)

Optional

index: Integer for noting the order of phases, Phases must be sequential, without any overlap. If indexes are not supplied, ordering will be assumed by declaration order.
max_submissions: Total submissions allowed per participant for the entire phase
max_submissions_per_day: Submission limit for each participant for a given day
auto_migrate_to_this_phase: Cannot be set on the first phase of the competition. This will re-submit all successful submissions from the previous phase to this phase at the time the phase starts.
execution_time_limit: Execution time limit for submissions, given in seconds. Default is 600.
hide_output: True/False. If True, stdout/stderr for all submissions to this phase will be hidden from users who are not competition administrators or collaborators.
starting_kit: path to the starting kit, a folder that participants will be able to download. Put there any useful files to help participants (example submissions, notebooks, documentation).
public_data: path to public data, that participants will be able to download.

phases:
  - index: 0
    name: Development Phase
    description: Tune your models
    start: 2019-12-12 13:30:00  # Time in UTC+0 and 24-hour format
    end: 2020-02-01 00:00:00  # Time in UTC+0 and 24-hour format
    execution_time_limit: 1200
    starting_kit: starting_kit
    public_data: public_data
    tasks:
      - 0
  - index: 1
    name: Final Phase
    description: Final testing of your models
    start: 2020-02-02 00:00:00 # Time in UTC+0 and 24-hour format
    auto_migrate_to_this_phase: True
    tasks:
      - 1

Tasks

Required

index: Number used for internal reference of the task, pointed to by solutions (below) and phases (above)
name: Name of the Task
scoring_program: File path relative to competition.yaml pointing to a .zip file or an unzipped directory, containing the scoring program

key: UUID of a task already in the database. If key is provided, all fields other than index will be ignored

Optional

description: Brief description of the task
input_data: File path to the data to be provided during the prediction step
reference_data: File path to the data to be provided to the scoring program
ingestion_program: File path to the ingestion program files
ingestion_only_during_scoring: True/False. If true, the ingestion program will be run in parallel with the scoring program, and can communicate w/ the scoring program via a shared directory

tasks:
  - index: 0
    name: Compute Pi Developement Task
    description: Compute Pi, focusing on accuracy
    input_data: dev_phase/input_data/
    reference_data: dev_phase/reference_data/
    ingestion_program: ingestion_program.zip
    scoring_program: scoring_program.zip
  - index: 1
    name: Compute Pi Final Task
    description: Compute Pi, speed and accuracy matter
    input_data: final_phase/input_data/
    reference_data: final_phase/reference_data/
    ingestion_program: ingestion_program.zip
    scoring_program: scoring_program.zip

Solutions

Required

index: Index number of solution
tasks: Array of the tasks (referenced internally) for which this solution applies.
path: File path to .zip or directory containing the solution data.

solutions:
  - index: 0
    path: solutions/solution1.zip
    tasks:
    - 0
    - 1
  - index: 1
    path: solutions/solution2/
    tasks:
    - 0

Fact Sheet

Optional

JSON for asking metadata questions about each submission when they are submitted

KEY: Programmatic name for a response. Should not contain any whitespace.
QUESTION TYPE:
- "checkbox": Prompts the user with a checkbox for a yes/no or true/false type question
  - Required SELECTION: [true, false]
- "text":: Prompts the user with a text box to write a response.
  - Required SELECTION: ""
  - "is_required": "false" will allow the user not to submit a response. Otherwise, the user will have to type something.
- "select": Gives the user a dropdown to select a value from.
  - SELECTION: Give an array of comma separated values that the user can select from: ["Value1","Value2","Value3",...,"ValueN"]
  - TIP: If you want this selection to be optional you can add "" as an option. ex. ["", "Value1", ...] and set "is_required": "false"
is_on_leaderboard: setting this to "true" will show this response on the leaderboard along with their submission.

Structure

fact_sheet: {
    "[KEY]": {
        "key": "[KEY]",
        "type": "[QUESTION TYPE]",
        "title": "[DISPLAY NAME]",
        "selection": [SELECTION],
        "is_required": ["true" OR "false"],
        "is_on_leaderboard": ["true" OR "false"]
}

fact_sheet: {
    "bool_question": {
        "key": "bool_question",
        "type": "checkbox",
        "title": "boolean",
        "selection": [True, False],
        "is_required": "false",
        "is_on_leaderboard": "false"
    },
    "text_question": {
        "key": "text_question",
        "type": "text",
        "title": "text",
        "selection": "",
        "is_required": "false",
        "is_on_leaderboard": "false"
    },
    "text_required": {
        "key": "text_required",
        "type": "text",
        "title": "text",
        "selection": "",
        "is_required": "true",
        "is_on_leaderboard": "false"
    },
    "selection": {
        "key": "selection",
        "type": "select",
        "title": "selection",
        "selection": ["", "v1", "v2", "v3"],
        "is_required": "false",
        "is_on_leaderboard": "true"
    }
}

Leaderboards

Leaderboard Details

Required

title: Title of leaderboard
key: Key for scoring program to write to
columns: An array of columns (see column layout below)

Optional

submission_rule: "Add", "Add_And_Delete", "Add_And_Delete_Multiple", "Force_Last", "Force_Latest_Multiple" or "Force_Best". It sets the behavior of the leaderboard regarding new submissions. See Leaderboard Functionality for more details.
hidden: True/False. If True, the contents of this leaderboard will be hidden to all users who are not competition administrators or collaborators.

Column Details

Required

title: Title of the column
key: Key for the scoring program to write to. The keys must match the keys of the scores.json file returned by the scoring program, as explained with more details here.
index: Number specifying the order the column should show up on the leaderboard

Optional

sorting: sorting order for the column: Descending (desc) or Ascending (asc)
- Ascending: smaller scores are better
- Descending: larger scores are better
computation: computation to be applied must be accompanied by computation indexes
- computation options: sum, avg, min, max
computation_indexes: an array of indexes of the columns the computation should be applied to
precision: (integer, default=2) to round the score to precision number of digits
hidden: (boolean, default=False) to hide/unhide a column on leaderboard

leaderboards:
  - title: Results
    key: main
    submission_rule: "Force_Last"
    columns:
      - title: Accuracy Score 1
        key: accuracy_1
        index: 0
        sorting: desc
        precision: 2
        hidden: False
      - title: Accuracy Score 2
        key: accuracy_2
        index: 1
        sorting: desc
        precision: 3
        hidden: False
      - title: Max Accuracy
        key: max_accuracy
        index: 2
        sorting: desc
        computation: max
        precision: 3
        hidden: False
        computation_indexes:
          - 0
          - 1
      - title: Duration
        key: duration
        index: 3
        sorting: asc
        precision: 2
        hidden: False