Improving the Detection of Relations Between Objects in an Image Using Textual Semantics

Summary, in English

In this article, we describe a system that classifies relations between entities extracted from an image. We started from the idea that we could utilize lexical and semantic information from text associated with the image, such as captions or surrounding text, rather than just the geometric and visual characteristics of the entities found in the image. We collected a corpus of images from Wikipedia together with their corresponding articles. In our experimental setup, we extracted two kinds of entities from the images, human beings and horses, and we defined three relations that could exist between them: Ride, Lead,or None. We used geometric features as a baseline to identify the relations between the entities and we describe the improvements brought by the addition of bag-of-word features and predicate–argument structures that we extracted from the text. The best semantic model resulted in a relative error reduction of more than 18 % over the baseline

• Author: Dennis Medved; Fangyuan Jiang; Peter Exner; Magnus Oskarsson; Pierre Nugues; Karl Åström
• Editor: Ana Fred; Maria De Marsico; Antoine Tabbone
• Department/s: Department of Computer Science; Mathematics (Faculty of Engineering); ELLIIT: the Linköping-Lund initiative on IT and mobile communication; eSSENCE: The e-Science Collaboration; Robotics and Semantic Systems
• Publishing year: 2015
• Language: English
• Pages: 133-145
• Publication/Series: Pattern Recognition Applications and Methods /Lecture Notes in Computer Science
• Volume: 9443
• Links: http://dx.doi.org/10.1007/978-3-319-25530-9_9
• Document type: Conference paper
• Publisher: Springer
• Topic: Computer and Information Science
• Keywords: Semantic parsing; Relation extraction from images; Machine learning
• Conference name: 3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014)
• Conference date: 2014-03-06 - 2014-03-08
• Conference place: Angers, France
• Status: Published
• ISBN/ISSN/Other: ISBN: 978-3-319-25529-3; ISBN: 978-3-319-25530-9