Overview

Here is an advanced tutorial. If you are new to CodaBench, please refer to get started tutorial first. In this article, you'll learn how to use more advanced features and how to create benchmarks using either the editor or bundles. Before proceeding to our tutorial, make sure you have registered for an account on the Codabench website.

Table of contents

1. Creating a Benchmark by Editor
2. Creating a Benchmark by Bundle
3. Benchmark Examples
4. How do I set up submission comments for multiple submissions?

The image below is an overview of the benchmark creation process

Creating a Benchmark by Editor

In this chapter, I'll take you step by step through the Editor's approach to creating benchmark, including algorithm type and dataset type.

Step 1: Click on Management in the top right corner of Codabench's home page under Competitions.

When you click on it, you will see the screen as shown in the screenshot below.

Step 2: Click on the Create button in the top right corner of Competition Management.

Step 3: Fill in the Details tab content.

Step 4: Fill in the Participant Tab.

Step 5: Fill in the Pages Tab.

Step 6: Fill in the Phases Tab.

When you click on the Manage Tasks/Datasets button, you will see the screenshot shown below

Click the Add Dataset button in the diagram to upload the resource files needed to create the competition.

Creating a phase will require a bundle of the following types (.zip format), I'll give you more details on how to write these bundle later.

Now you just need to have this concept in your mind

Here are the screenshots of the 5 types of bundles after they were uploaded

Step 7: Fill in the Leaderboard Tab.

Step 8: Save and Publish the Benchmark

Creating a Benchmark by Bundle

Creating a benchmark through bundles is a much more efficient way than using editors.

Simple Version Example: CLASSIFY WHEAT SEEDS

Step 1: Download bundle

https://github.com/codalab/competition-examples/blob/master/codabench/wheat_seeds/code_submission_bundle.zip

Click on the link above to download the bundle in the screenshot.

Step 2: Go to the benchmark upload page

Step 3: Upload the bundle

After the bundle has been uploaded, you will see the screenshot shown below.

Step 4: View your new benchmark

Benchmark Examples

Example bundles for code & dataset competition can be found here:

https://github.com/codalab/competition-examples/tree/master/codabench

Iris

Iris Codabench Bundle is a simple benchmark involving two phases, code submission and results submission.

AutoWSL

Two versions of the Automated Weakly Supervised Learning Benchmark:

• Code submission benchmark
• Dataset submission benchmark

Mini-AutoML

Mini-AutoML Bundle is a benchmark template for Codabench, featuring code submission to multiple datasets (tasks).

How do I set up submission comments for multiple submissions?

Steps

Step 1: Click the edit button

Step 2: Enable multiple submissions on leaderboard

Step 3: Set up submission comment

Step 4: Save all changes

Step 5: Leave a comment before making submission

Step 6: Check out the leaderboard