Please use this identifier to cite or link to this item:
Title: Characterization and classification of semantic image-text relations
Authors: Otto, C.Springstein, M.Anand, A.Ewerth, R.
Publishers Version:
Issue Date: 2020
Published in: International Journal of Multimedia Information Retrieval Vol. 9 (2020), No. 1
Publisher: Berlin : Springer Nature
Abstract: The beneficial, complementary nature of visual and textual information to convey information is widely known, for example, in entertainment, news, advertisements, science, or education. While the complex interplay of image and text to form semantic meaning has been thoroughly studied in linguistics and communication sciences for several decades, computer vision and multimedia research remained on the surface of the problem more or less. An exception is previous work that introduced the two metrics Cross-Modal Mutual Information and Semantic Correlation in order to model complex image-text relations. In this paper, we motivate the necessity of an additional metric called Status in order to cover complex image-text relations more completely. This set of metrics enables us to derive a novel categorization of eight semantic image-text classes based on three dimensions. In addition, we demonstrate how to automatically gather and augment a dataset for these classes from the Web. Further, we present a deep learning system to automatically predict either of the three metrics, as well as a system to directly predict the eight image-text classes. Experimental results show the feasibility of the approach, whereby the predict-all approach outperforms the cascaded approach of the metric classifiers.
Keywords: Data augmentation; Image-text class; Multimodality; Semantic gap
DDC: 020
License: CC BY 4.0 Unported
Link to License:
Appears in Collections:Informationswissenschaften

This item is licensed under a Creative Commons License Creative Commons