|
Annotation sensitivity: Training data collection methods affect model performance
Kern, Christoph
;
Eckman, Stephanie
;
Beck, Jacob
;
Chew, Rob
;
Ma, Bolei
;
Kreuter, Frauke
|
DOI:
|
https://doi.org/10.18653/v1/2023.findings-emnlp.992
|
|
URL:
|
https://aclanthology.org/2023.findings-emnlp.992
|
|
Dokumenttyp:
|
Konferenzveröffentlichung
|
|
Erscheinungsjahr:
|
2023
|
|
Buchtitel:
|
Findings of the Association for Computational Linguistics: EMNLP 2023
|
|
Seitenbereich:
|
14874-14886
|
|
Veranstaltungsdatum:
|
6-10.12.23
|
|
Herausgeber:
|
Bouamor, Houda
;
Pino, Juan
;
Bali, Kalika
|
|
Ort der Veröffentlichung:
|
Singapore
|
|
Verlag:
|
Association for Computational Linguistics
|
|
Verwandte URLs:
|
|
|
Sprache der Veröffentlichung:
|
Englisch
|
|
Einrichtung:
|
Außerfakultäre Einrichtungen > MZES - Arbeitsbereich A
|
|
Fachgebiet:
|
004 Informatik
|
|
Abstract:
|
When training data are collected from human annotators, the design of the annotation instrument, the instructions given to annotators, the characteristics of the annotators, and their interactions can impact training data. This study demonstrates that design choices made when creating an annotation instrument also impact the models trained on the resulting annotations. We introduce the term annotation sensitivity to refer to the impact of annotation data collection methods on the annotations themselves and on downstream model performance and predictions. We collect annotations of hate speech and offensive language in five experimental conditions of an annotation instrument, randomly assigning annotators to conditions. We then fine-tune BERT models on each of the five resulting datasets and evaluate model performance on a holdout portion of each condition. We find considerable differences between the conditions for 1) the share of hate speech/offensive language annotations, 2) model performance, 3) model predictions, and 4) model learning curves. Our results emphasize the crucial role played by the annotation instrument which has received little attention in the machine learning literature. We call for additional research into how and why the instrument impacts the annotations to inform the development of best practices in instrument design.
|
 | Dieser Eintrag ist Teil der Universitätsbibliographie. |
 Suche Autoren in
BASE:
Kern, Christoph
;
Eckman, Stephanie
;
Beck, Jacob
;
Chew, Rob
;
Ma, Bolei
;
Kreuter, Frauke
Google Scholar:
Kern, Christoph
;
Eckman, Stephanie
;
Beck, Jacob
;
Chew, Rob
;
Ma, Bolei
;
Kreuter, Frauke
ORCID:
Kern, Christoph  ORCID: https://orcid.org/0000-0001-7363-4299, Eckman, Stephanie, Beck, Jacob, Chew, Rob, Ma, Bolei and Kreuter, Frauke ORCID: https://orcid.org/0000-0002-7339-2645
Sie haben einen Fehler gefunden? Teilen Sie uns Ihren Korrekturwunsch bitte hier mit: E-Mail
Actions (login required)
 |
Eintrag anzeigen |
|
