AMREF: An Adaptive MapREduce Framework for real time applications

Abstract

This paper presents AMREF, an Adaptive MapREduce Framework designed for an effective use of computational resources in data center networks to deal with real time data intensive applications. AMREF entails its adaptivity from adaptive splitter, adaptive mappers and adaptive reducers in a stochastic control manner. We use three methods, feedback control, stochastic learning with smooth filter and kalman filter to implement the framwork. Comparison among the three methods suggests they can be effectively and efficiently used to reduce the makspan in three different real-world workload scenarios. © 2010 IEEE.