基于CLIP的自适应加权参数融合在类别增量学习中的应用

Adaptive Weighted Parameter Fusion with CLIP for Class-Incremental Learning

摘要 Abstract

类别增量学习(CIL)使模型能够逐步吸收来自新类别的知识,并构建一个涵盖之前所有遇到类别的通用分类器。当模型优化新类别时,不可避免地会遗忘旧类别知识,导致灾难性遗忘。解决这一挑战需要在保留旧知识和容纳新信息之间做出权衡。然而,这种平衡过程往往需要牺牲部分信息,从而导致模型区分不同类别的能力部分丧失。为了解决这个问题,我们设计了基于对比语言图像预训练(CLIP)的自适应加权参数融合方法,该方法不仅考虑了不同任务数据分布的变化,还最大程度地保留了参数矩阵的所有有效信息。此外,我们引入了一个平衡因子,可以平衡相邻任务的数据分布对齐和可区分性。在多个传统基准数据集上的实验结果验证了所提出方法的优越性。

Class-incremental Learning (CIL) enables the model to incrementally absorb knowledge from new classes and build a generic classifier across all previously encountered classes. When the model optimizes with new classes, the knowledge of previous classes is inevitably erased, leading to catastrophic forgetting. Addressing this challenge requires making a trade-off between retaining old knowledge and accommodating new information. However, this balancing process often requires sacrificing some information, which can lead to a partial loss in the model's ability to discriminate between classes. To tackle this issue, we design the adaptive weighted parameter fusion with Contrastive Language-Image Pre-training (CLIP), which not only takes into account the variability of the data distribution of different tasks, but also retains all the effective information of the parameter matrix to the greatest extent. In addition, we introduce a balance factor that can balance the data distribution alignment and distinguishability of adjacent tasks. Experimental results on several traditional benchmarks validate the superiority of the proposed method.