Multimedia retrieval through spatio-temporal activity maps

Abstract

As multiple video cameras and other sensors generate very large quantities of multimedia data in media productions and surveillance applications, a key challenge is to identify the relevant content and to rapidly retreive the corresponding sensor data. A spatio-temporal activity map serves as an efficient and intuitive graphical user interface for multimedia retrieval, particularly when the media streams are derived from multiple sensors observing a physical environment. We formulate the media retrieval problem in this context, and develop an architecture for interactive media retrieval by combining spatio-temporal "activity maps" with domain specific event information. Activity maps are computed from trajectories of motion of objects in the environment, which in turn are derived automatically by analysis of sensor data. We present an activity map based video retrieval system for the sport of tennis and demonstrate that the activity map base scheme significantly helps the user in a) discovering the relevant portions of the data, and b) non-linearly retrieving the corresponding media streams.