Image with a message : towards detecting non-literal image usages by visual linking

Weiland, Lydia ; Dietz, Laura ; Ponzetto, Simone Paolo

Additional URL:
Document Type: Conference or workshop publication
Year of publication: 2015
Book title: The Workshop on Vision and Language 2015 (VL’15) : Vision and Language Meet Cognitive Systems ; Proceedings ; September 18, 2015 Lisbon, Portugal
Page range: 40-47
Date of the conference: 18.09.2015
Place of publication: Stroudsburg, Pa.
Publishing house: Assoc. for Computational Linguistics
ISBN: 978-1-941643-32-7
Publication language: English
Institution: School of Business Informatics and Mathematics > Semantic Web (Juniorprofessur) (Ponzetto 2013-2015)
Subject: 004 Computer science, internet
Abstract: A key task to understand an image and its corresponding caption is not only to find out what is shown on the picture and described in the text, but also what is the exact relationship between these two elements. The long-term objective of our work is to be able to distinguish different types of relationship, including literal vs. non-literal usages, as well as finegrained non-literal usages (i.e., symbolic vs. iconic). Here, we approach this challenging problem by answering the question: ‘How can we quantify the degrees of similarity between the literal meanings expressed within images and their captions?’. We formulate this problem as a ranking task, where links between entities and potential regions are created and ranked for relevance. Using a Ranking SVM allows us to leverage from the preference ordering of the links, which help us in the similarity calculation for the cases of visual or textual ambiguity, as well as misclassified data. Our experiments show that aggregating different features using a supervised ranker achieves better results than a baseline knowledge-base method. However, much work still lies ahead, and we accordingly conclude the paper with a detailed discussion of a short- and longterm outlook on how to push our work on relationship classification one step further.

Dieser Eintrag ist Teil der Universitätsbibliographie.

Metadata export


+ Search Authors in

+ Page Views

Hits per month over past year

Detailed information

You have found an error? Please let us know about your desired correction here: E-Mail

Actions (login required)

Show item Show item