多任务工具平台:通过相关且动态的任务评估基于LLM代理的鲁棒性
Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missions
摘要 Abstract
大型语言模型(LLMs)因其先进的理解和规划能力,在工具调用方面展现出强大的潜力,用户越来越多地依赖基于LLM的代理通过迭代交互解决复杂任务。然而,现有的基准测试主要集中在单任务场景下评估代理,未能捕捉现实世界的复杂性。为弥合这一差距,我们提出了多任务工具平台。在该基准测试中,每个测试案例包含多个相互关联的任务。这种设计要求代理能够动态适应不断变化的需求。此外,所提出的基准测试探索了固定任务数量内的所有可能任务切换模式。具体而言,我们提出了一种多代理数据生成框架来构建基准测试,并提出了一种新颖的方法,利用动态决策树评估代理决策的准确性和效率。对多种开源和闭源LLM进行的实验揭示了影响代理鲁棒性的关键因素,并为工具调用社区提供了可行的见解。
Large language models (LLMs) demonstrate strong potential as agents for tool invocation due to their advanced comprehension and planning capabilities. Users increasingly rely on LLM-based agents to solve complex missions through iterative interactions. However, existing benchmarks predominantly access agents in single-mission scenarios, failing to capture real-world complexity. To bridge this gap, we propose the Multi-Mission Tool Bench. In the benchmark, each test case comprises multiple interrelated missions. This design requires agents to dynamically adapt to evolving demands. Moreover, the proposed benchmark explores all possible mission-switching patterns within a fixed mission number. Specifically, we propose a multi-agent data generation framework to construct the benchmark. We also propose a novel method to evaluate the accuracy and efficiency of agent decisions with dynamic decision trees. Experiments on diverse open-source and closed-source LLMs reveal critical factors influencing agent robustness and provide actionable insights to the tool invocation society.