Please use this identifier to cite or link to this item:
Files in This Item:
File SizeFormat 
p14-henning.pdf5,72 MBAdobe PDFView/Open
Title: Estimating the information gap between textual and visual representations
Authors: Henning, ChristianEwerth, Ralph
Publishers Version:
Issue Date: 2017
Published in: ICMR '17 Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, Bucharest, Romania — June 06 - 09, 2017 , Page 14-22
Publisher: New York City : Association for Computing Machinery
Abstract: Photos, drawings, figures, etc. supplement textual information in various kinds of media, for example, in web news or scientific pub- lications. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general il- lustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be de- scribed and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe cross- modal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach.
Keywords: Text-image relations; multimodal embeddings; deep learning
DDC: 020
License: This document may be downloaded, read, stored and printed for your own use within the limits of § 53 UrhG but it may not be distributed via the internet or passed on to external parties.
Dieses Dokument darf im Rahmen von § 53 UrhG zum eigenen Gebrauch kostenfrei heruntergeladen, gelesen, gespeichert und ausgedruckt, aber nicht im Internet bereitgestellt oder an Außenstehende weitergegeben werden.
Appears in Collections:Informationswissenschaften

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.