leaderboard:programming language translation - chunhualiao/public-docs GitHub Wiki

Cross-programming language translation leaderboards are frameworks used to compare the performance of models designed to translate code from one programming language to another. These leaderboards typically evaluate models based on various metrics, including translation accuracy, execution correctness, and the ability to handle complex code structures[5][8].

Notable Benchmarks and Leaderboards

  1. CodeTransOcean: A comprehensive multilingual benchmark covering 45 programming languages, including popular and niche ones. It consists of three datasets:

    • MultilingualTrans: For translations between multiple popular languages
    • NicheTrans: For translating between niche and popular languages
    • LLMTrans: For evaluating code executability[1][2]

  2. AdvBench: Designed to evaluate TransCoder, focusing on Java, C++, and Python translations. It uses metrics like BLEU, Exact Match, and Code Execution Accuracy[3].

  3. CodeNet: A vast dataset containing 14 million code samples across 50 programming languages, providing a comprehensive foundation for evaluating translation models[3].

  4. CodeXGLUE: Evaluates various programming tasks, including code translation, with execution-based metrics to ensure functional correctness[3].

  5. RustRepoTrans: The first benchmark to include repository-level dependencies for Rust code translation, addressing limitations of function-level datasets[3].

  6. RepoTransBench: Features 100 repositories with automated test suites to evaluate translation quality at the repository level[3].

  7. ClassEval-T: A class-level code translation benchmark for Python, C++, and Java, with high-coverage test suites. It evaluates translation correctness and the ability to handle dependencies and library invocations[5].

These benchmarks and leaderboards help researchers and developers assess the effectiveness of code translation models, identify areas for improvement, and track progress in the field of cross-language code translation.

Citations: