HyperSpark: A Software Engineering Approach to Parallel Metaheuristics
Abstract
Metaheuristics are used to solve complex, untractable problems for which other approaches are unsuitable or unable to provide solutions in reasonable times. Although computing power has grown exponentially with the onset of Cloud Computing and Big Data, the domain of metaheuristics has not yet taken full advantage of this new potential. In this paper we address this gap by proposing HyperSpark, a metaheuristic optimization framework for the scalable execution of user-defined, computationally-intensive metaheuristics. HyperSpark provides a way to harness the benefits (e.g., scalability by design) and features (e.g., a simple programming model or ad-hoc infrastructure tuning) of state-of-the-art big data technology for the benefit of metaheuristic computation. We elaborate on HyperSpark and evaluate its efficiency and generality on several different metaheuristics for the Permutation Flow-Shop Problem (PFSP). We observe that HyperSpark results are comparable with the best solutions of the literature and this shows clearly the great potential behind the propsed approach.