Supporting multimedia retrieval in annotated content using hyperknowledge

Abstract

The meaning behind a multimedia content usually vary according to its fragments, composed by spatial coordinates or intervals of time. The description of these concepts and entities in function of these multimedia fragments (anchors) is what we argue that can fulfill the multimedia meaning (semantic) description. In this paper we propose a multimedia search and retrieval approach based on semantic annotations. More precisely, our approach has two main aspects: knowledge structuring and information retrieval. Users or information extraction services define semantic annotations over anchors of multimedia content, defining their abstract meaning. Under the hood, the system uses the hyperknowledge model for structuring these semantic annotations. For approaching the second aspect, we propose a search algorithm that processes hyperknowledge graphs and retrieves multimedia anchors whose semantics matches with a given query. As proof-of-concept, we present a multimedia annotation system that uses the hyperknowledge model for structuring annotations on multimedia content. We have assessed our approach by carrying out experiments with end users regarding the effectiveness of our search algorithm. The results of this paper indicate that the hyperknowledge model is able to properly represent knowledge information from annotations on multimedia content. And they also indicate that our algorithm works for retrieving multimedia content when using this model.