Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairs

Reinig, Ines ; Markert, Katja

Document Type: Conference or workshop publication
Year of publication: 2023
Book title: The 15th International Conference on Computational Semantics : proceedings of the conference, June 21 - 23, 2023 Nancy, France
Page range: 1-15
Conference title: The 15th International Confrerence on Computational Semantics (IWCS) 2023
Location of the conference venue: Nancy, France
Date of the conference: 21.-23.06.2023
Publisher: Amblard, Maxime ; Breitholtz, Ellen
Place of publication: Nancy, France
Publishing house: Association for Computational Linguistics
Related URLs:
Publication language: English
Institution: School of Business Informatics and Mathematics > Information Systems III: Enterprise Data Analysis (Ponzetto 2016-)
Subject: 004 Computer science, internet
Abstract: Compared to English, German word order is freer and therefore poses additional challenges for natural language inference (NLI). We create WOGLI (Word Order in German Language Inference), the first adversarial NLI dataset for German word order that has the following properties: (i) each premise has an entailed and a non-entailed hypothesis; (ii) premise and hypotheses differ only in word order and necessary morphological changes to mark case and number. In particular, each premise and its two hypotheses contain exactly the same lemmata. Our adversarial examples require the model to use morphological markers in order to recognise or reject entailment. We show that current German autoencoding models fine-tuned on translated NLI data can struggle on this challenge set, reflecting the fact that translated NLI datasets will not mirror all necessary language phenomena in the target language. We also examine performance after data augmentation as well as on related word order phenomena derived from WOGLI. Our datasets are publically available at

Dieser Eintrag ist Teil der Universitätsbibliographie.

Metadata export


+ Search Authors in

+ Page Views

Hits per month over past year

Detailed information

You have found an error? Please let us know about your desired correction here: E-Mail

Actions (login required)

Show item Show item