Home - codalab/codabench GitHub Wiki
Documentation
Welcome to the Codabench wiki!
Codabench is a platform allowing you to flexibly specify a benchmark. First you define tasks, e.g. datasets and metrics of success, then you specify the API for submissions of code (algorithms), add some documentation pages, and "CLICK!" your benchmark is created, ready to accept submissions of new algorithms. Participant results get appended to an ever-growing leaderboard.
You may also create inverted benchmarks in which the role of datasets and algorithms are swapped. You specify reference algorithms and your participants submit datasets.
Table of contents
1. Participants
2. Organizers
Benchmark creation
- Getting Started Tutorial
- Advanced Tutorial
- How to transition from CodaLab to Codabench?
- Competition Creation: Form
- Competition Creation: Bundle
- Competition YAML Structure
- Competition Bundle Structure
- Competition Docker image
- Dataset competition creation and participate instruction
- Leaderboard Features
- Competition Examples
- Example: Cancer Benchmarks
Running a benchmark
- Benchmark Management & List Page
- Benchmark Detail Page and organizer features (manage participants, submissions, dumps...)
- Resource Management: Submissions, Datasets/Programs, Tasks and Competition Bundles (Resources interface)
- Queue Management
- Compute Worker Management & Setup
- Server Status (monitor the submissions at the compute workers level)
3. Developers and administrators
Codabench Platform Technical Information and Management
- Codabench Basic Installation Guide
- How to Deploy Server
- Administrator procedures
- Codabench Docker Architecture
- Submission Docker Container Layout
- Backups: Creating, Automating, and Restoring
- Submission Process Overview
- Robot submissions
- Running tests
- Automation
- Manual validation
- Validation and deployment of pull requests
- Upgrading Codabench
4. How you can contribute
- Use Codabench by either participating in a competition or hosting a new competition.
- Find a bug? Got a feature request? Submit a GitHub issue.
- P1 issues are the most important ones.
- Submit pull requests on GitHub to implement new features or fix bugs (see contributing.md).
- Improve this documentation.
- Let others know about Codabench!
5. How to contact us
- The preferred way is via posting a GitHub issue.
- In case of emergency, contact us directly.
- If you wish to get in touch with the community, you can use the Google Groups.