ACPBench数据集提供了高效规划所需的原子推理任务。该数据集旨在将复杂的计划生成任务分解为最简单的原子推理任务形式，例如布尔问题或多项选择题，模型需要从提供的选项中选择正确的答案。尽管ACPBench的目标是测试关于动作和变化的最简单形式的推理，但在处理规划任务时，模型通常没有可供选择的选项，因此规划所需的推理形式具有开放性和生成性。为此，我们引入了ACPBench Hard，这是ACPBench的一个生成版本，包含开放性问题，模型需要回答这些问题。在这些任务上表现良好的模型原则上可以集成到一个规划器中，或者直接用作策略。我们讨论了这些任务的复杂性以及验证其答案正确性的难度，并为每个任务提出了验证算法。配备了这些验证器后，我们在我们的任务上测试了各种模型的表现，发现即使是最先进的模型在大多数任务上的表现仍然不尽如人意。我们的实验表明，在这些任务中没有一个模型能够脱颖而出，除了少数例外，所有测试的语言模型得分均低于65%，这表明当前前沿的语言模型在可靠地进行规划推理方面还有很长的路要走。事实上，即使是所谓的推理模型在解决这些推理任务时也遇到了困难。ACPBench Hard数据集可通过以下链接获取：https://ibm.github.io/ACPBench

The ACPBench dataset provides atomic reasoning tasks required for efficient planning. The dataset is aimed at distilling the complex plan generation task into separate atomic reasoning tasks in their easiest possible form, boolean or multiple-choice questions, where the model has to choose the right answer from the provided options. While the aim of ACPBench is to test the simplest form of reasoning about action and change, when tasked with planning, a model does not typically have options to choose from and thus the reasoning required for planning dictates an open-ended, generative form for these tasks. To that end, we introduce ACPBench Hard, a generative version of ACPBench, with open-ended questions which the model needs to answer. Models that perform well on these tasks could in principle be integrated into a planner or be used directly as a policy. We discuss the complexity of these tasks as well as the complexity of validating the correctness of their answers and present validation algorithms for each task. Equipped with these validators, we test the performance of a variety of models on our tasks and find that for most of these tasks the performance of even the largest models is still subpar. Our experiments show that no model outperforms another in these tasks and with a few exceptions all tested language models score below 65%, indicating that even the current frontier language models have a long way to go before they can reliably reason about planning. In fact, even the so-called reasoning models struggle with solving these reasoning tasks. ACPBench Hard collection is available at the following link: https://ibm.github.io/ACPBench