涂鸦即所需:基于问题特定涂鸦的LLM代码评估增强方法

Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics

摘要 Abstract

自GPT-3和ChatGPT发布以来,LLM技术带来了颠覆性的变革,LLMs在编程相关任务中展现出非凡的潜力。尽管代码生成仍然是研究的热门领域,但基于LLMs的代码评估仍是一个尚未解决的问题。本文专注于基于LLMs的代码评估,并试图填补现有空白。我们提出了多智能体的新方法,采用针对具体问题的涂鸦,论证这些方法在逻辑评估方面比现有的通用涂鸦方法表现更佳。为了解决缺乏合适的评估数据集的问题,我们引入了两个数据集:一个包含150名学生的提交数据的数据结构与算法数据集,来自一个流行的在线数据结构与算法练习网站;另一个包含80名学生的提交数据的面向对象编程数据集,来自本科计算机科学课程。除了使用标准指标(斯皮尔曼相关系数、科恩的kappa系数),我们还提出了一种新的度量标准,称为宽容度,用于量化相对专家评估的评价严格程度。我们的综合分析表明,问题特定涂鸦显著提升了教育环境中代码的逻辑评估,提供了更好的反馈,这不仅限于语法正确性,还与教学目标保持一致。

Since the disruption in LLM technology brought about by the release of GPT-3 and ChatGPT, LLMs have shown remarkable promise in programming-related tasks. While code generation remains a popular field of research, code evaluation using LLMs remains a problem with no conclusive solution. In this paper, we focus on LLM-based code evaluation and attempt to fill in the existing gaps. We propose multi-agentic novel approaches using question-specific rubrics tailored to the problem statement, arguing that these perform better for logical assessment than the existing approaches that use question-agnostic rubrics. To address the lack of suitable evaluation datasets, we introduce two datasets: a Data Structures and Algorithms dataset containing 150 student submissions from a popular Data Structures and Algorithms practice website, and an Object Oriented Programming dataset comprising 80 student submissions from undergraduate computer science courses. In addition to using standard metrics (Spearman Correlation, Cohen's Kappa), we additionally propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment. Our comprehensive analysis demonstrates that question-specific rubrics significantly enhance logical assessment of code in educational settings, providing better feedback aligned with instructional goals beyond mere syntactic correctness.