统一自回归多模态模型UGen及其渐进词汇学习方法

UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learning

摘要 Abstract

我们提出了UGen,这是一种统一的自回归多模态模型,在文本处理、图像理解和图像生成任务中同时表现出强大的性能。UGen将文本和图像都转换为离散的标记序列,并利用单一的Transformer以自回归的方式统一生成它们。为了解决统一多模态学习面临的挑战,UGen采用了一种新颖的机制,即渐进词汇学习。在这个过程中,视觉标记ID逐步激活并整合到训练阶段,最终提高了统一多模态学习的有效性。在全面的文本和图像任务上的实验表明,与普通的统一自回归方法相比,UGen的整体性能提升了13.3%,并且在所有任务中的表现也与多种特定任务模型具有竞争力。

We introduce UGen, a unified autoregressive multimodal model that demonstrates strong performance across text processing, image understanding, and image generation tasks simultaneously. UGen converts both texts and images into discrete token sequences and utilizes a single transformer to generate them uniformly in an autoregressive manner. To address the challenges associated with unified multimodal learning, UGen is trained using a novel mechanism, namely progressive vocabulary learning. In this process, visual token IDs are incrementally activated and integrated into the training phase, ultimately enhancing the effectiveness of unified multimodal learning. Experiments on comprehensive text and image tasks show that UGen achieves a significant overall performance improvement of 13.3% compared to the vanilla unified autoregressive method, and it also delivers competitive results across all tasks against several task-specific models.

统一自回归多模态模型UGen及其渐进词汇学习方法 - arXiv