About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Paper
BigProvision: A provisioning framework for big data analytics
Abstract
In the past few years, big data has attracted significant attention, and many analytics platforms, such as Hadoop, have been developed to enable the analysis of massive data. Nevertheless, it is still very challenging to provision, let alone optimize, a comprehensive system that includes various aspects, from the computing infrastructure to the analytics programs. To tackle this challenge, in this article, we propose a novel provisioning framework, BigProvision, to provision big data analytics systems. The main idea of the framework is to first evaluate and model the performance of different big data analytics approaches, given a set of sample data and various analytics requirements, such as the expected results, budget, response time, and so on. Based on the evaluation and modeling results, BigProvision can generate a provisioning configuration that can be used to configure the whole system for big data analytics. To evaluate the performance of the proposed framework, we develop an experimental prototype that supports three analytics platforms, Hadoop, Spark, and GraphLab. Our experiments show that for the classic PageRank analysis, both GraphLab and Spark can outperform Hadoop under different requirements. Moreover, by modeling the results, our prototype can determine the expected settings, such as the number of machines and network capacity, for the system that shall handle the complete data set. The prototype and experiments demonstrate that the proposed framework has great potential to facilitate the provision and optimization of future big data analytics systems.
Related
Conference paper
Reconciling malware labeling discrepancy via consensus learning
Conference paper
Delay asymptotics for heavy-tailed MapReduce jobs
Conference paper