Multidimensional adaption of language models: domains, languages, and social dimensions

Hung, Chia-Chien

PDF
Dissertation_Hung (submit).pdf - Published
Download (3MB)

URN:	urn:nbn:de:bsz:180-madoc-689942
Document Type:	Doctoral dissertation
Year of publication:	2024
Place of publication:	Mannheim
University:	Universität Mannheim
Evaluator:	Ponzetto, Simone Paolo
Publication language:	English
Institution:	School of Business Informatics and Mathematics > Information Systems III: Enterprise Data Analysis (Ponzetto 2016-)
Subject:	004 Computer science, internet
Keywords (English):	pre-trained language model , adaptation barrier , domain adaptation , language adaptation , demographic adaptation , multidimensional adaptation , transfer learning , cross-lingual transfer , task-agnostic adaptation
Abstract:	The emergence of Pre-trained Language Models (PLMs) has revolutionized natural language processing, yielding remarkable improvements across diverse tasks. While PLMs excel at discerning language patterns and crafting coherent narratives, they face challenges when adapting to specialized topics, such as medical concepts, cross-lingual conversational systems, and content from diverse sociodemographic backgrounds. These challenges, collectively referred to as the "adaptation barrier," stem from the difficulty of tailoring PLMs to fields requiring expert knowledge, often due to insufficient topic-relevant information within the model’s training data. Recent research has explored techniques like domain adaptation, cross-lingual transfer learning, and the integration of external knowledge to address these limitations. However, existing approaches often focus narrowly on either domain-specific, language-specific, or social dimensions, limiting a comprehensive understanding of the effectiveness of proposed adaptation approaches. In this thesis, we systematically investigate the adaptation barrier by conducting experiments across multiple dimensions. We focus on three key challenges: effectiveness, efficiency, and interpretability in adapting PLMs to diverse domains, languages, and social dimensions. Through our findings, we hope to contribute to the development of more adaptable language models across multiple dimensions, enhancing their applicability in addressing real-world challenges, thereby mitigating the adaptation barrier inherent to PLMs.