Developmentally plausible multimodal language models are highly modular

Klerings, Alina ; Bartelt, Christian ; Mueller, Aaron

PDF
2024.conll-babylm.10.pdf - Veröffentlichte Version
Download (3MB)

URL:	https://aclanthology.org/2024.conll-babylm.10/
URN:	urn:nbn:de:bsz:180-madoc-693784
Dokumenttyp:	Konferenzveröffentlichung
Erscheinungsjahr:	2024
Buchtitel:	The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning : proceedings of the Second BabyLM Challenge : November 15-16, 2024
Seitenbereich:	118-139
Veranstaltungstitel:	CoNLL 2024, The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning
Veranstaltungsort:	Miami, FL
Veranstaltungsdatum:	15.-16.11.2024
Herausgeber:	Hu, Michael Y. ; Mueller, Aaron ; Ross, Candace ; Williams, Adina ; Linzen, Tal ; Zhuang, Chengxu ; Choshen, Leshem ; Cotterell, Ryan ; Warstadt, Alex ; Wilcox, Ethan Gotlieb
Ort der Veröffentlichung:	Miami, FL, USA
Verlag:	Association for Computational Linguistics
ISBN:	79-8-89176-222-0
Sprache der Veröffentlichung:	Englisch
Einrichtung:	Außerfakultäre Einrichtungen > Institut für Enterprise Systems (InES)
Bereits vorhandene Lizenz:	Creative Commons Namensnennung 4.0 International (CC BY 4.0)
Fachgebiet:	004 Informatik
Abstract:	Large language models demonstrate emergent modularity, where functionally specialized components and circuits arise to handle specific tasks or task formats. If similar modules arise in models trained on more cognitively plausible datasets, it could inform debates surrounding what kinds of would be learnable given more human-like language learning signals. In this paper, we describe a multimodal vision-language model submitted to the BabyLM Challenge. Our model achieves similar performance to the best-performing architectures from last year, though visual information does not improve performance on text-only tasks over text-only models (in accordance with prior findings). To better understand how the model processes the evaluation tasks of the BabyLM Challenge, we leverage causal interpretability methods to locate the neurons that contribute to the model`s final decisions. We find that the models we train are highly modular: distinct components arise to process related tasks. Furthermore, on text-and-image tasks, adding or removing visual inputs causes the model to use distinct components to process the same textual inputs. This suggests that modal and task-specific specialization is efficiently learned, and that a high degree of functional specialization arises in even small-scale language models.