Semantics reinforcement and fusion learning for multimedia streams

Abstract

Fusion of multimedia streams for enhanced performance is a critical problem for retrieval. However, fusion performance tends to easily overt the hillclimb set used to learn fusion rules. In this paper, we perform fusion learning for multimedia streams using a greedy performance driven algorithm. In our fusion learning paradigm, fused output is a linear combination of multiple classifers or ranked streams. The algorithm is inspired from Ensemble Learning [2] but takes that idea further for improving generalization capability. A key application of our fusion learning algorithm, described in this work, is semantics reinforcement using an ensemble of classifers built using the same training dataset but groundtruth corresponding to different concepts. We expect that classifers built for semantically close concepts should reinforce each other's performance and fusion learning is an excellent post-classifcation way to reinforce semantics and performance. Fusion learning experiments have been performed on TRECVID 2005 test set. Experiments using the well established retrieval efectiveness measure of mean average precision reveal that our proposed algorithm improves over the best classifer (oracle) by 3.8%. We also present and discuss some interesting and intuitive semantic reinforcement trends observed during fusion learning. Copyright 2007 ACM.