Understanding the message of images with knowledge base traversals

Weiland, Lydia ; Hulpus, Ioana ; Ponzetto, Simone Paolo ; Dietz, Laura

DOI: https://doi.org/10.1145/2970398.2970414
URL: https://scholar.google.com/citations?view_op=view_...
Additional URL: http://www.bibsonomy.org/bibtex/d704a2e510a0497db6...
Document Type: Conference or workshop publication
Year of publication: 2016
Book title: Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval, ICTIR 2016, Newark, DE, USA, September 13-16, 2016
Page range: 199-208
Conference title: ICTIR'16
Location of the conference venue: Newark, DE
Date of the conference: 13.-16.09.2016
Publisher: Carterette, Ben
Place of publication: New York, NY
Publishing house: ACM
ISBN: 978-1-4503-4497-5
Publication language: English
Institution: School of Business Informatics and Mathematics > Information Systems III: Enterprise Data Analysis (Ponzetto 2016-)
Subject: 004 Computer science, internet
Abstract: The message of news articles is often supported by the pointed use of iconic images. These images together with their captions encourage emotional involvement of the reader. Current algorithms for understanding the semantics of news articles focus on its text, often ignoring the image. On the other side, works that target the semantics of images, mostly focus on recognizing and enumerating the objects that appear in the image. In this work, we explore the problem from another perspective: Can we devise algorithms to understand the message encoded by images and their captions? To answer this question, we study how well algorithms can describe an image-caption pair in terms of Wikipedia entities, thereby casting the problem as an entity-ranking task with an image-caption pair as query. Our proposed algorithm brings together aspects of entity linking, subgraph selection, entity clustering, relatedness measures, and learning-to-rank. In our experiments, we focus on media-iconic image-caption pairs which often reflect complex subjects such as sustainable energy and endangered species. Our test collection includes a gold standard of over 300 image-caption pairs about topics at different levels of abstraction. We show that with a MAP of 0.69, the best results are obtained when aggregating content-based and graph-based features in a Wikipedia-derived knowledge base.

Dieser Eintrag ist Teil der Universitätsbibliographie.

Metadata export


+ Search Authors in

+ Page Views

Hits per month over past year

Detailed information

You have found an error? Please let us know about your desired correction here: E-Mail

Actions (login required)

Show item Show item