Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation
https://arxiv.org/abs/2606.12117
José Hernández-Orallo
https://scholar.google.com/citations?hl=en&user=n9AWbcAAAAAJ&view_op=list_works&sortby=pubdate
ProgramBench: Can Language Models Rebuild Programs From Scratch?
https://arxiv.org/abs/2605.03546
Language Model Evaluation Harness
https://github.com/EleutherAI/lm-evaluation-harness
Can Models Begin to Participate in the Loops That Drive Scientific and Engineering Progress?
https://autolab.moe/blog#case-study-autolab-in-action
SWE-bench
https://www.vals.ai/benchmarks/swebench
MIRAGE: The Illusion of Visual Understanding
https://arxiv.org/abs/2603.21687
Promptfoo: LLM evals & red teaming
https://github.com/promptfoo/promptfoo
Compare AI Generations
https://voxelbench.ai/
Independent analysis of AI
https://artificialanalysis.ai/
Hallucination Benchmark Leaderboard
https://halluhard.com/
Theory of Space: Can foundation models construct spatial beliefs through active perception?
https://github.com/mll-lab-nu/Theory-of-Space
BabyVision: Visual Reasoning Beyond Language
https://xbench.org/agi/babyVision
DeepEval - The LLM Evaluation Framework
https://github.com/confident-ai/deepeval
HalluHard - Hallucination Benchmark Leaderboard
https://halluhard.com
NeMo Evaluator SDK
https://github.com/NVIDIA-NeMo/Evaluator
The Universal Evaluation Interface for LLM Applications
https://github.com/grounded-ai/grounded_ai
A benchmark of expert-level academic questions to assess AI capabilities
https://www.nature.com/articles/s41586-025-09962-4
bloom: An Open Source Tool for Automated Behavior Evaluation of LLMs
https://github.com/safety-research/bloom
Evaluating chain-of-thought monitorability
https://openai.com/index/evaluating-chain-of-thought-monitorability/
A Rosetta Stone for AI benchmarks
https://epoch.ai/blog/a-rosetta-stone-for-ai-benchmarks
State of the Art
https://www.alphaxiv.org/state-of-the-art
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
https://oxrml.com/measuring-what-matters/
VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation
https://github.com/CSU-JPG/VCode/tree/main
NeMo Evaluator SDK
https://github.com/NVIDIA-NeMo/Evaluator
Curious Beginner's Guide to AI Evaluations
https://creatoreconomy.so/p/curious-beginners-guide-to-ai-evaluations
ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
https://github.com/OSU-NLP-Group/ScienceAgentBench
AutoBench 1.0 - Collective-LLM-as-a-Judge Benchmark System
https://huggingface.co/AutoBench/AutoBench_1.0
Open Evals
https://open-evals.com/
Evaluating multiple models using labeled and unlabeled data
https://arxiv.org/abs/2501.11866
SWE-smith is a toolkit for training
https://github.com/SWE-agent/SWE-agent
Provider-agnostic, open-source evaluation infrastructure for language models
https://github.com/groq/openbench