基于大型语言模型的城市系统管理文档分类中的文本分块研究

Text Chunking for Document Classification for Urban System Management using Large Language Models

摘要 Abstract

城市系统的管理依赖于复杂的文本文档,这些文档需要进行编码和分析以设定需求并评估建成环境的表现。本文探讨了利用大型语言模型(LLM)应用于定性编码活动,以减少资源需求同时保持与人类相当的可靠性。定性编码和评估面临诸如资源限制、偏见、准确性和评估者之间一致性等挑战。我们报告了应用LLMs对10篇案例文档进行演绎编码,涉及17个数字孪生特征用于城市系统管理的研究。我们采用两种提示方法比较LLMs与人工编码工作的语义处理:全文分析和基于OpenAI的GPT-4o、GPT-4o-mini和o1-mini模型的文本块分析。研究发现,不同方法间存在相似的内部变异性,结果表明当LLMs初始化为特定的演绎编码上下文时,其表现可能与人工编码员相当。使用分块方法时,GPT-4o、o1-mini和GPT-4o-mini与人工评分员表现出显著一致。GPT-4o和GPT-4o-mini作为额外评分员与三位人工评分员一起使用时,所有评分员之间的统计一致性显著,表明文本文档分析受益于LLMs。我们的研究揭示了LLMs应用的细微次主题,表明LLMs遵循人类记忆编码过程,其中全文分析可能会引入多重含义。本文的新贡献在于评估了OpenAI GPT模型的性能,并提出了基于分块提示的方法,通过保存局部上下文解决了上下文聚合偏差问题。

Urban systems are managed using complex textual documentation that need coding and analysis to set requirements and evaluate built environment performance. This paper contributes to the study of applying large-language models (LLM) to qualitative coding activities to reduce resource requirements while maintaining comparable reliability to humans. Qualitative coding and assessment face challenges like resource limitations and bias, accuracy, and consistency between human evaluators. Here we report the application of LLMs to deductively code 10 case documents on the presence of 17 digital twin characteristics for the management of urban systems. We utilize two prompting methods to compare the semantic processing of LLMs with human coding efforts: whole text analysis and text chunk analysis using OpenAI's GPT-4o, GPT-4o-mini, and o1-mini models. We found similar trends of internal variability between methods and results indicate that LLMs may perform on par with human coders when initialized with specific deductive coding contexts. GPT-4o, o1-mini and GPT-4o-mini showed significant agreement with human raters when employed using a chunking method. The application of both GPT-4o and GPT-4o-mini as an additional rater with three manual raters showed statistically significant agreement across all raters, indicating that the analysis of textual documents is benefited by LLMs. Our findings reveal nuanced sub-themes of LLM application suggesting LLMs follow human memory coding processes where whole-text analysis may introduce multiple meanings. The novel contributions of this paper lie in assessing the performance of OpenAI GPT models and introduces the chunk-based prompting approach, which addresses context aggregation biases by preserving localized context.

基于大型语言模型的城市系统管理文档分类中的文本分块研究 - arXiv