Please use this identifier to cite or link to this item:
Files in This Item:
File SizeFormat 
10.1007_978-3-319-56608-5_15.pdf273,77 kBAdobe PDFView/Open
Title: “Are machines better than humans in image tagging?” - A user study adds to the puzzle
Authors: Ewerth, RalphSpringstein, MatthiasPhan-Vogtmann, Lo AnSchütze, Juliane
Publishers Version:
Issue Date: 2017
Published in: Advances in Information Retrieval - 39th European Conference on IR Research, ECIR 2017, Aberdeen, UK, April 8-13, 2017, Proceedings
Lecture Notes in Computer Science, Volume 10193, 978-3-319-56608-5
Publisher: Heidelberg : Springer
Abstract: “Do machines perform better than humans in visual recognition tasks?” Not so long ago, this question would have been considered even somewhat provoking and the answer would have been clear: “No”. In this paper, we present a comparison of human and machine performance with respect to annotation for multimedia retrieval tasks. Going beyond recent crowdsourcing studies in this respect, we also report results of two extensive user studies. In total, 23 participants were asked to annotate more than 1000 images of a benchmark dataset, which is the most comprehensive study in the field so far. Krippendorff’s α is used to measure inter-coder agreement among several coders and the results are compared with the best machine results. The study is preceded by a summary of studies which compared human and machine performance in different visual and auditory recognition tasks. We discuss the results and derive a methodology in order to compare machine performance in multimedia annotation tasks at human level. This allows us to formally answer the question whether a recognition problem can be considered as solved. Finally, we are going to answer the initial question.
DDC: 020
License: CC BY 4.0 Unported
Link to License:
Appears in Collections:Informationswissenschaften

This item is licensed under a Creative Commons License Creative Commons