BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks

Large language models (LLMs) are powerful tools capable of handling diverse tasks. However, their evaluation remains challenging due to the vast and often confusing range of available benchmarks. This complexity not only increases the risk of benchmark misuse and misinterpretation but also demands substantial effort from LLM users, including researchers, practitioners, and non-AI companies, seeking the most suitable benchmarks for their specific needs. To address these issues, we introduce \texttt{BenchmarkCards}, an intuitive and validated documentation framework that systematically captures critical benchmark attributes such as objectives, methodologies, data sources, and limitations. Through user studies with benchmark creators and users, we show that \texttt{BenchmarkCards} can simplify benchmark selection and enhance transparency, facilitating more informed decision-making in evaluating LLMs.