全能MATH:面向大型语言模型的奥林匹克数学水平基准测试

Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models

摘要 Abstract

近年来,大型语言模型(LLMs)在数学推理能力方面取得了显著突破。然而,现有的基准测试如GSM8K或MATH已被模型以高准确率解决(例如,OpenAI o1在MATH数据集上的准确率为94.8%),表明这些基准已不足以真正挑战这些模型。为弥补这一不足,我们提出了一套全面且具有挑战性的基准测试,专门用于评估LLMs在奥林匹克水平水平上的推理能力。与现有的奥林匹克相关基准测试不同,我们的数据集专注于数学领域,并包含由人工严格标注的4428个竞赛级问题,涵盖了超过33个子领域以及10个不同的难度等级,从而实现对模型在奥林匹克数学推理能力上的全面评估。此外,我们基于该基准进行了深入分析。实验结果表明,即使是最先进的模型(如OpenAI o1-mini和OpenAI o1-preview),在处理高度挑战性的奥林匹克水平问题时也表现不佳,准确率分别为60.54%和52.55%,凸显了奥林匹克数学推理的显著挑战性。

Recent advancements in large language models (LLMs) have led to significant breakthroughs in mathematical reasoning capabilities. However, existing benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g., OpenAI o1 achieves 94.8\% on MATH dataset), indicating their inadequacy for truly challenging these models. To bridge this gap, we propose a comprehensive and challenging benchmark specifically designed to assess LLMs' mathematical reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks, our dataset focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels, enabling a holistic assessment of model performance in Olympiad-mathematical reasoning. Furthermore, we conducted an in-depth analysis based on this benchmark. Our experimental results show that even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle with highly challenging Olympiad-level problems, with 60.54\% and 52.55\% accuracy, highlighting significant challenges in Olympiad-level mathematical reasoning.