Informing unsupervised pretraining with external linguistic knowledge


Lauscher, Anne ; Vulic, Ivan ; Ponti, Edoardo Maria ; Korhonen, Anna ; Glavaš, Goran



URL: https://arxiv.org/pdf/1909.02339.pdf
Additional URL: https://arxiv.org/abs/1909.02339
Document Type: Working paper
Year of publication: 2019
Place of publication: Ithaca, NY
Publishing house: Cornell University
Publication language: English
Institution: School of Business Informatics and Mathematics > Wirtschaftsinformatik III (Ponzetto 2016-)
School of Business Informatics and Mathematics > Text Analytics for Interdisciplinary Research (Juniorprofessur) (Glavaš 2017-)
Subject: 004 Computer science, internet
Keywords (English): Natural Language Processing , BERT
Abstract: Unsupervised pretraining models have been shown to facilitate a wide range of downstream applications. These models, however, still encode only the distributional knowledge, incorporated through language modeling objectives. In this work, we complement the encoded distributional knowledge with external lexical knowledge. We generalize the recently proposed (state-of-the-art) unsupervised pretraining model BERT to a multi-task learning setting: we couple BERT's masked language modeling and next sentence prediction objectives with the auxiliary binary word relation classification, through which we inject clean linguistic knowledge into the model. Our initial experiments suggest that our "linguistically-informed" BERT (LIBERT) yields performance gains over the linguistically-blind "vanilla" BERT on several language understanding tasks.

Dieser Eintrag ist Teil der Universitätsbibliographie.




+ Citation Example and Export

Lauscher, Anne ; Vulic, Ivan ; Ponti, Edoardo Maria ; Korhonen, Anna ; Glavaš, Goran (2019) Informing unsupervised pretraining with external linguistic knowledge. Ithaca, NY [Working paper]


+ Search Authors in

+ Page Views

Hits per month over past year

Detailed information



You have found an error? Please let us know about your desired correction here: E-Mail


Actions (login required)

Show item Show item