RUSSE: the first workshop on Russian semantic similarity

Panchenko, Alexander ; Loukachevitch, Natalia ; Ustalov, Dmitry ; Paperno, Denis ; Meyer, Christian ; Konstantinova, Natalia

URL:	http://www.dialog-21.ru/digests/dialog2015/materia...
Additional URL:	http://www.dialog-21.ru/media/2778/dialogue-2015_v...
Document Type:	Conference or workshop publication
Year of publication:	2015
Book title:	Computational Linguistics and Intellectual Technologies : Papers from the Annual conference "Dialogue 2015", Moscow, May 27 - 30, 2015
The title of a journal, publication series:	Dialogue
Volume:	14,2
Page range:	89-105
Conference title:	Computational Linguistics and Intellectual Technologies 2015
Location of the conference venue:	Moscow, Russia
Date of the conference:	May 27-30, 2015
Place of publication:	Moscow
Publishing house:	RSUH
ISSN:	2221-7932 , 2075-7182
Publication language:	English
Institution:	School of Business Informatics and Mathematics > Information Systems III: Enterprise Data Analysis (Ponzetto 2016-)
Subject:	004 Computer science, internet
Keywords (English):	computational linguistics , lexical semantics , semantic similarity measures , semantic relations , semantic relation extraction , semantic relatedness , synonyms , hypernyms , co-hyponyms
Abstract:	The paper gives an overview of the Russian Semantic Similarity Evaluation (RUSSE) shared task held in conjunction with the Dialogue 2015 conference. There exist a lot of comparative studies on semantic similarity, yet no analysis of such measures was ever performed for the Russian language. Exploring this problem for the Russian language is even more interesting, because this language has features, such as rich morphology and free word order, which make it signi cantly di erent from English, German, and other well-studied languages. We attempt to bridge this gap by proposing a shared task on the semantic similarity of Russian nouns. Our key contribution is an evaluation methodology based on four novel benchmark datasets for the Russian language. Our analysis of the 105 submissions from 19 teams reveals that successful approaches for English, such as distributional and skip-gram models, are directly applicable to Russian as well. On the one hand, the best results in the contest were obtained by sophisticated supervised models that combine evidence from di erent sources. On the other hand, completely unsupervised approaches, such as a skip-gram model estimated on a large-scale corpus, were able score among the top 5 systems.