Scalability of the Nutch search engine
José E. Moreira, Maged M. Michael, et al.
ICS 2007
MapReduce is a scalable parallel computing framework forbig data processing. It exhibits multiple processing phases,and thus an efficient job scheduling mechanism is crucial forensuring efficient resource utilization. This work studies thescheduling challenge that results from the overlapping of the"map" and "shuffle" phases in MapReduce. We propose anew, general model for this scheduling problem. Further,we prove that scheduling to minimize average response timein this model is strongly NP-hard in the offline case andthat no online algorithm can be constant-competitive in theonline case. However, we provide two online algorithms thatmatch the performance of the offline optimal when given aslightly faster service rate.
José E. Moreira, Maged M. Michael, et al.
ICS 2007
Parijat Dube, Michael Tsao, et al.
ISPASS 2010
Shun-Zheng Yu, Zhen Liu, et al.
IPCCC 2002
Jian Tan, Hanhua Feng, et al.
ITC 2012