The Optimization of Resources Within the Implementation of a Big Data Solution

Abstract

Every day we notice and witness the creation of a significant volumes of data from user-data generated automatically on the internet. To control a large amount of data we need powerful tools that can give us the ability to store and process big data. Moreover, a problem will arise when we start the procedure on large amounts of data in parallel because clustered computing platforms are complex and difficult environments to manage. In this article, we propose a method based on machine learning algorithms that recommends a parameter of task parallelism in big data workloads. As a solution, we propose a method based on optimizing the parallelism of tasks in clustered applications and by borrowing methods of machine learning. In order to optimize the numbers of the clusters in the implementation of a solution Big Data.