Informing unsupervised pretraining with external linguistic knowledge

Lauscher, Anne ; Vulić, Ivan ; Ponti, Edoardo Maria ; Korhonen, Anna ; Glavaš, Goran

Additional URL:
Document Type: Working paper
Year of publication: 2019
Place of publication: Ithaca, NY
Publishing house: Cornell University
Publication language: English
Institution: School of Business Informatics and Mathematics > Information Systems III: Enterprise Data Analysis (Ponzetto 2016-)
School of Business Informatics and Mathematics > Text Analytics for Interdisciplinary Research (Juniorprofessur) (Glavaš 2017-2021)
Subject: 004 Computer science, internet
Keywords (English): Natural Language Processing , BERT
Abstract: Unsupervised pretraining models have been shown to facilitate a wide range of downstream applications. These models, however, still encode only the distributional knowledge, incorporated through language modeling objectives. In this work, we complement the encoded distributional knowledge with external lexical knowledge. We generalize the recently proposed (state-of-the-art) unsupervised pretraining model BERT to a multi-task learning setting: we couple BERT's masked language modeling and next sentence prediction objectives with the auxiliary binary word relation classification, through which we inject clean linguistic knowledge into the model. Our initial experiments suggest that our "linguistically-informed" BERT (LIBERT) yields performance gains over the linguistically-blind "vanilla" BERT on several language understanding tasks.

Dieser Eintrag ist Teil der Universitätsbibliographie.

Metadata export


+ Search Authors in

+ Page Views

Hits per month over past year

Detailed information

You have found an error? Please let us know about your desired correction here: E-Mail

Actions (login required)

Show item Show item