指令层次结构通过从系统消息到用户消息、对话历史以及工具输出建立优先级顺序，对于确保语言模型（LMs）行为的一致性和安全性至关重要。尽管其重要性不言而喻，但这一主题却受到的关注有限，并且缺乏全面的基准来评估模型遵循指令层次结构的能力。我们通过引入IHEval填补了这一空白，该基准包含九个任务中的3,538个示例，涵盖了不同优先级指令对齐或冲突的情况。我们对流行语言模型的评估表明，它们在识别指令优先级方面存在困难。与原始指令遵循性能相比，所有评估的模型在面对冲突指令时都表现出显著的性能下降。此外，最具竞争力的开源模型仅在解决这些冲突方面达到48%的准确率。我们的结果强调了未来语言模型开发中需要有针对性优化的需求。

The instruction hierarchy, which establishes a priority order from system messages to user messages, conversation history, and tool outputs, is essential for ensuring consistent and safe behavior in language models (LMs). Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy. We bridge this gap by introducing IHEval, a novel benchmark comprising 3,538 examples across nine tasks, covering cases where instructions in different priorities either align or conflict. Our evaluation of popular LMs highlights their struggle to recognize instruction priorities. All evaluated models experience a sharp performance decline when facing conflicting instructions, compared to their original instruction-following performance. Moreover, the most competitive open-source model only achieves 48% accuracy in resolving such conflicts. Our results underscore the need for targeted optimization in the future development of LMs.