Please use this identifier to cite or link to this item:
|Title:||Estimating the information gap between textual and visual representations|
|Authors:||Henning, Christian; Ewerth, Ralph|
|Published in:||ICMR '17 Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, Bucharest, Romania — June 06 - 09, 2017 , Page 14-22|
|Publisher:||New York City : Association for Computing Machinery|
|Abstract:||Photos, drawings, figures, etc. supplement textual information in various kinds of media, for example, in web news or scientific pub- lications. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general il- lustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be de- scribed and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe cross- modal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach.|
|Keywords:||Text-image relations; multimodal embeddings; deep learning|
|License:||This document may be downloaded, read, stored and printed for your own use within the limits of § 53 UrhG but it may not be distributed via the internet or passed on to external parties.|
Dieses Dokument darf im Rahmen von § 53 UrhG zum eigenen Gebrauch kostenfrei heruntergeladen, gelesen, gespeichert und ausgedruckt, aber nicht im Internet bereitgestellt oder an Außenstehende weitergegeben werden.
|Appears in Collections:||Informationswissenschaften|
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.