基于大型语言模型的多模态人工智能系统在多主题物理概念库存上的多语言表现

Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories

摘要 Abstract

我们利用来自PhysPort网站的一组涵盖多种语言和主题类别的物理概念库存,研究了基于大型语言模型的人工智能系统(GPT-4o)的多语言和多模态性能。这些库存涉及经典物理学主题,如力学、电磁学、光学、热力学以及相对论、量子力学、天文学、数学和实验技能。不同于以往仅限文本的研究,我们将库存以图像形式上传,以反映学生在纸面上看到的内容,从而评估系统的多模态功能。我们的结果显示,不同主题的表现存在差异,其中实验技能表现最弱。同时,不同语言之间也存在差异,英语和欧洲语言表现最强。值得注意的是,库存项目的相对难度很大程度上独立于调查的语言。与现有文献中关于学生表现的比较显示,该人工智能系统在除实验技能外的所有主题类别中的表现均优于平均的本科后教学学生。此外,系统在需要解读图像的问题上的表现比纯文本问题上的表现更差。

We investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover classical physics topics such as mechanics, electromagnetism, optics, and thermodynamics, as well as relativity, quantum mechanics, astronomy, mathematics, and laboratory skills. Unlike previous text-only studies, we uploaded the inventories as images to reflect what a student would see on paper, thereby assessing the system's multimodal functionality. Our results indicate variation in performance across subjects, with laboratory skills standing out as the weakest. We also observe differences across languages, with English and European languages showing the strongest performance. Notably, the relative difficulty of an inventory item is largely independent of the language of the survey. When comparing AI results to existing literature on student performance, we find that the AI system outperforms average post-instruction undergraduate students in all subject categories except laboratory skills. Furthermore, the AI performs worse on items requiring visual interpretation of images than on those that are purely text-based.

基于大型语言模型的多模态人工智能系统在多主题物理概念库存上的多语言表现 - arXiv