摘要 Abstract
多任务提示调优利用多个高资源源任务来提升低资源目标任务的表现。现有的方法要么一次性将所有源任务组合训练的软提示转移,要么仅转移一个“高度相似”的源任务。然而,我们发现最优的迁移性能往往来自源任务的某种组合,既非单一也非全部。此外,我们还发现,在迁移后的微调过程中,源任务与目标任务之间的相似性也会动态变化,这使得在初始阶段基于相似性计算的方法显得不足。为了解决这些问题,我们提出了一种名为动态任务向量分组(Dynamic Task Vector Grouping, DTVG)的方法,其核心思想包括:(1)使用任务向量而非软提示来衡量任务相似性;(2)基于“目标相似度”和“知识一致性”两个指标对最优的源任务组合进行分组;(3)在每次迭代步骤中动态更新组合。在不同设置下的26个自然语言处理数据集上的大量实验表明,DTVG能够有效地分组相似的源任务,同时减少负迁移现象,取得了当前最先进(state-of-the-art)的性能。
Multi-task prompt tuning utilizes multiple high-resource source tasks to improve performance on low-source target tasks. Existing approaches transfer the soft prompt trained by combining all source tasks or a single ``high-similar'' source task one-time-only. However, we find that the optimal transfer performance often comes from a combination of source tasks, which is neither one nor all. Further, we find that the similarity between source and target tasks also changes dynamically during fine-tuning after transfering, making similarity calculation in the initiation stage inadequate. To address these issues, we propose a method called Dynamic Task Vector Grouping (DTVG), whose core ideas contain (1) measuring the task similarity with task vectors instead of soft prompt, (2) grouping the optimal source task combination based on two metrics: {\it target similarity} and {\it knowledge consistency}; (3) dynamically updating the combination in each iteration step. Extensive experiments on the 26 NLP datasets under different settings demonstrate that DTVG effectively groups similar source tasks while reducing negative transfer, achieving the start-of-art performance.