Comparison of Derivative Free Algorithms in Optimization High Dimension Problems

Abstract

Gradient free optimization algorithms are often used when gradient information of a function is not available or it is difficult and costly to obtain or estimate. Many kinds of derivative free algorithms are used in the fields of statistics, and engineering. This paper compared five derivative-free optimization algorithms using a set of benchmark functions. Then the comparison results will be reported and discussed. The employed algorithms include Genetic algorithm, Particle Swarm Optimization, Nelder-Mead simplex and Nelder-Mead simplex Plus and quasi Gradient method. This master thesis (1) reviews how each optimization method works in detail; (2) tests the quality of each algorithm in optimizing high dimensional problems; (3) discuss different optimization method’s performance in optimization;(4) choose top performing optimization method solve particular artificial neural networks problems, which is intended to explore the potential of combining derivative free method with artificial neural networks.