基于上下文增强的大语言模型改进SZZ算法:LLM4SZZ

LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models

摘要 Abstract

SZZ算法是识别引起错误提交的主要技术,并为许多软件工程研究(如错误预测和静态代码分析)奠定了基础。自提出以来,研究者们提出了许多变体来提升SZZ算法的性能。大多数变体依赖静态技术和启发式假设,虽然易于实现,但性能提升有限。近期,基于深度学习的SZZ算法被引入以改进原始SZZ算法。然而,该方法需要复杂的预处理且仅限于单一编程语言。此外,尽管提升了精确度,却牺牲了召回率。并且,大多数变体忽略了提交信息和补丁上下文等重要信息,仅限于涉及删除行的错误修复提交。大型语言模型(LLMs)的出现为解决这些问题提供了机会。在本研究中,我们探讨了LLMs的优势和局限性,并提出了LLM4SZZ,采用两种方法(即基于排名的识别和基于上下文增强的识别)处理不同类型的错误修复提交。我们根据LLM理解错误并判断错误是否存在于提交中的能力,决定采用哪种方法。在基于上下文增强的识别中,我们为LLM提供更多的上下文,并要求其从一组候选提交中找到引起错误的提交。在基于排名的识别中,我们让LLM从错误修复提交中选择有问题的语句,并根据它们与根本原因的相关性对它们进行排名。实验结果表明,LLM4SZZ在三个数据集上均优于所有基线,F1分数提高了6.9%到16.0%,同时没有显著降低召回率。

The SZZ algorithm is the dominant technique for identifying bug-inducing commits and serves as a foundation for many software engineering studies, such as bug prediction and static code analysis. Researchers have proposed many variants to enhance the SZZ algorithm's performance since its introduction. The majority of them rely on static techniques or heuristic assumptions, making them easy to implement, but their performance improvements are often limited. Recently, a deep learning-based SZZ algorithm has been introduced to enhance the original SZZ algorithm. However, it requires complex preprocessing and is restricted to a single programming language. Additionally, while it enhances precision, it sacrifices recall. Furthermore, most of variants overlook crucial information, such as commit messages and patch context, and are limited to bug-fixing commits involving deleted lines. The emergence of large language models (LLMs) offers an opportunity to address these drawbacks. In this study, we investigate the strengths and limitations of LLMs and propose LLM4SZZ, which employs two approaches (i.e., rank-based identification and context-enhanced identification) to handle different types of bug-fixing commits. We determine which approach to adopt based on the LLM's ability to comprehend the bug and identify whether the bug is present in a commit. The context-enhanced identification provides the LLM with more context and requires it to find the bug-inducing commit among a set of candidate commits. In rank-based identification, we ask the LLM to select buggy statements from the bug-fixing commit and rank them based on their relevance to the root cause. Experimental results show that LLM4SZZ outperforms all baselines across three datasets, improving F1-score by 6.9% to 16.0% without significantly sacrificing recall.