Hyperparameter tuning methods in automated machine learning

Abstract

In this paper, we make a comprehensive review of the challenging task of hyperparameter optimization in automated machine learning. The commonly used hyperparameter optimization methods include single-shot sampling strategies, e.g., grid search, random search and sequential strategies where new trials are gradually augmented based on existing information, including Bayesian optimization, evolutionary algorithms, reinforcement learning-based methods. We find the sequential number-theoretic optimization (SNTO) algorithm proposed by Kai-Tai Fang and Yuan Wang in 1990 can also be applied in hyperparameter optimization, where sequential uniform designs are utilized to search the global optima in complex response surfaces. For illustration, various hyperparameter optimization methods are tested with two widely-used machine learning models including the support vector machine (SVM) and extreme gradient boosting (XGBoost), on two classical binary classification datasets. By analyzing the experimental results, we find a modified SNTO algorithm is quite promising in the hyperparameter optimization task.