通过幻觉诱导优化减轻大型视觉语言模型中的幻觉现象
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
摘要 Abstract
虽然大型视觉语言模型(LVLMs)在理解多模态数据方面表现出色,但它们不可避免地会出现幻觉现象,导致生成的文本与对应的图像之间存在脱节。几乎所有当前的视觉对比解码方法都试图通过引入适当的视觉不确定性信息来缓解这些幻觉现象,从而适当扩大幻觉与目标类别之间的对比对数差距。然而,由于全局视觉不确定性的不可控性,这些方法难以精确诱导幻觉标记,这严重限制了其在缓解幻觉方面的有效性,甚至可能导致生成不希望出现的幻觉。为了解决这一问题,我们进行了理论分析以提升对比解码的有效性。基于这一见解,我们提出了一种名为幻觉诱导优化(HIO)的新颖优化策略。该策略依靠经过微调的理论偏好模型(即相反的布拉德利-特里模型)来放大幻觉标记与目标标记之间的对比,从而促进高效的对比解码,以减轻LVLMs中的幻觉现象。广泛的实验研究表明,我们的HIO策略可以有效减少LVLMs中的幻觉现象,在各种基准测试中优于最先进的方法。
Although Large Visual Language Models (LVLMs) have demonstrated exceptional abilities in understanding multimodal data, they invariably suffer from hallucinations, leading to a disconnect between the generated text and the corresponding images. Almost all current visual contrastive decoding methods attempt to mitigate these hallucinations by introducing visual uncertainty information that appropriately widens the contrastive logits gap between hallucinatory and targeted ones. However, due to uncontrollable nature of the global visual uncertainty, they struggle to precisely induce the hallucinatory tokens, which severely limits their effectiveness in mitigating hallucinations and may even lead to the generation of undesired hallucinations. To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding. Building on this insight, we introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO). This strategy seeks to amplify the contrast between hallucinatory and targeted tokens relying on a fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model), thereby facilitating efficient contrast decoding to alleviate hallucinations in LVLMs. Extensive experimental research demonstrates that our HIO strategy can effectively reduce hallucinations in LVLMs, outperforming state-of-the-art methods across various benchmarks.