Blog - til-ai/til-25 GitHub Wiki

General discussion about TIL-AI, rules clarifications, and explainers.

Contents

Changes to the Qualifiers Leaderboard

Changes have been made to the Leaderboard in 2025:

  • Instead of showing teams' last result, it now shows their best result.
  • Accuracy and speed scores are rounded to 3 significant figures of precision.
  • If two teams have the same overall result, the submission time is used as a tie-breaker.
    • The team that achieved the high score earlier is ranked higher.
    • Subsequently submitting models that fail to achieve a high score has no impact on its team's ranking.

By ranking teams by their best result, we aim to encourage experimentation and risk-taking by removing penalties for unsuccessful experiments. This change also prevents teams from "sandbagging" (intentionally hiding their true performance), making the leaderboard more representative and reducing the surge of submissions near the deadline.

Rounding to 3 significant figures eliminates the effect of noise and arbitrary circumstances (such as random variations in container startup time) on the rankings. Further precision provides little meaningful information about the performance of teams' models, as variations of less than 1 in 1000 are almost never under teams' conscious control.

Combining the more aggressive rounding with submission time-based tie-breaking implicitly adds a meta-score (beyond the tangible performance of the model): the efficiency and expertise of the team. Beyond 3sf of precision, the speed at which a team is able to produce a high-scoring model becomes a much more meaningful indicator of their ability. Because only models that beat the previous high score (which, by definition, improve the ranking) will affect the submission time, there is no penalty at all for trying out new models to submit, even if the team is currently tied with another.

Use of non-original software

Non-original software refers to code (including libraries, packages, and tools), models, and other software that was not developed by your team.

Participants are allowed (in fact, encouraged) to use third-party libraries and models to speed up their development. This includes packages installed from PyPI (using pip) or apt, and base models from services like Hugging Face. The only condition is that the code must be freely available and developed independently of the team.

Hence, you can use any open source code and model, as long as it's attached with an explicit open source license (e.g., MIT, Apache, GPL). You cannot use software that you or your team specifically directed someone else to develop, even if that software is freely available. You also cannot use any proprietary software (like libraries which require a paid license to use).

[!NOTE] Just because code is public doesn't mean it's open source or freely available. For example, all GitHub repos which don't carry an explicit license are fully copyright-protected by default, even if they're "public".